US-VO Website

Building Registry Clients

As we have seen in several of the course presentations, applications may wish to use a more centralized discovery portal to find astronomical resources. While the registry portals have variability in their interfaces, the metadata standards, registry harvesting mechanisms, and emerging Registry Interface standard will provide valuable utilties for 'Mining the Sky'. These Registries, sometimes referred to as the "yellow books" for the VO are planned to go beyond course metadata descriptions to include fine level information which may be useful in building a science specific application.

In this part of the Registry session we will focus on the Registry Client interfaces and provide a basic exercise for you to build a client using one of the available public web services

Sample Applications with Registry Clients

With the registry web service interface there are a couple applications which you are already familiar with that use a Registry client component, namely Datascope and OpenSkyQuery (OSQ). These are used to search for standard VO services such as Cone, SIAP, and Skynode.

Special Case with Proxy class

With a client application which relies on a registry to find data content you may need to provide some failover protection. For example in the case of OpenSkyQuery, a RegistryAdmin proxy class is defined to set the url.

RegistryAdmin ra = new net.ivoa.Registry.RegistryAdmin();

ra.Url = url;

The client then is built for a registry web service which returns a DataSet class

string predicate = "resourcetype like 'skynode' and "+param+" order by shortname";
return (new RegistryAdmin()).DSQueryRegistry(predicate);

gets a dataset with all the skynodes.

Student Exercise: Adding registry support into our Cone Search Client.

In this session, we create a Cone Search client which uses the registry to get Cone Search base URLs.

Preparation

This section has no edifying content, but is necessary to prepare to run the exercise.

Type "ant prep"

Creating your registry client code

We start by building our client-side interface to the Registry's Web Service.

Type: "wsdl2java registry.wsdl"
registry.wsdl is the WSDL file for the NVO Registry at STScI. Running wsdl2java creates interface classes along with classes representing the XML types that are sent in the soap messages.

For example, briefly examine org/us_vo/www/RegistrySoap.java. You will see a method for each of the Web Service methods supported by the STScI Registry.
Compile the client code: "ant compileWSDLcode"
This compiles all the generated code along with our application code within the coneclient directory.

Cone Search Finder class

We will now create a class that will look specifically for Cone Search services in the Registry.

Open up nvoregistry/coneclient/FindConeSearch.java.
Notice the function called search(). This is a front end to the Web Service's queryRegistry() function. It takes an SQL "WHERE" clause and combines it with a constraint that matches only Cone Search services:
```
// Combine our query with a constraint to return only Cone Searches.
// The resulting query will look something like this:
// 
//    ServiceType like '%CONE%' and (Title like '%parallax%')
// 
query = "ServiceType like '%CONE%' and (" + query + ")";
          
```
Our method sends this modified query to the Web Service method, queryRegistry(). From the result, we extract an array of resource descriptions. Note that the descriptions are not VOResource records; rather they use a custom schema, SimpleResource (primarily for historical reasons).

Test the finder class (from the directory containing the build.xml file):

java coneclient.FindConeSearch "Description like '%parallax%'"

Feed one of the returned service URLs into the ConeCaller application:
```
java coneclient.ConeCaller 180.0 60.0 1.0 "http://chart.stsci.edu/GSCVO/HIPVO.jsp?"
          
```
This version of ConeCaller was modified to take a Cone Search base URL from the command line.

Now try the second service returned by FindConeSearch:
```
java coneclient.ConeCaller 180.0 60.0 1.0 "http://vizier.u-strasbg.fr/viz-bin/votable/-dtd/-A?-out.add=_RAJ2000,_DEJ2000&-source=I/280"
          
```
This produces an exception. Why?

Look at coneclient/ConeCaller.java within callConeService() to see how the URL is formed:
```
URL cone = new URL(coneUrl + "RA="+ra+"&DEC="+dec+"&SR="+sr);
          
```
This is compliant with the Cone Search specification. Unfortunately, this particular base URL from the registry needs a trailing &. Despite the standard, many of the URLs in the registry are incorrectly entered in this manner.

Fortunately, this is not hard to fix in our client application. FindConeSearch.java includes a correctBaseURL() function. We can incorporate this into our FindConeSearch application. Within its main() function, find this line:
```
System.out.println("Address: " + cs[i].getServiceURL());
          
```
and change it to:
```
System.out.println("Address: " + 
     FindConeSearch.correctBaseURL(cs[i].getServiceURL()));
          
```
Recompile this class by typing "ant", and the try our registry query again.
Exercise: Create a keywordSearch() method.
A keywordSearch() method is provided already by the STScI Registry. It simply compares all the input words to selected text fields in the resource description. Unfortunately, this implementation returns VOResource (v0.9) records, not SimpleResource.

Nevertheless, it would not be difficult to implement our own keyword search that does return SimpleResource records in addition to restricting the match to Cone Search services.

FindConeSearch.java contains a stubbed version of a keywordSearch() method (the last method in the source file); that is, you need to complete the implementation to make it work. If you need to cheat a little, just look at the FindConeSearch.java in the solutions subdirectory or review the answer here.
Exercise: Incorporate a keyword registry query into the ConeCaller application.
In our version of ConeCaller, if the fourth command-line argument is a base URL, it will use it to call the specific Cone Search service. If it is not a URL, any arguments after the third one are interpreted as keywords for a registry search, and a keyword query string is formed. In this exercise, pass this query to our FindConeSearch class to retrieve Cone Search base URLs. Then call one or more of the matched services.

For extra credit, call each cone search service and print the number of records it returns. Again, you can peek at ConeCaller.java in the solutions directory, or look here for an explanation.

Exercise: Create a keywordSearch() method

Our stub function looks like this:

public static SimpleResource[] keywordSearch(String keywords) 
     throws ServiceException, RemoteException 
{
    String query = "";    //
    String keyword;

    StringTokenizer st = new StringTokenizer(keywords);
    while (st.hasMoreTokens()) {
        keyword = st.nextToken();

        // add in our constraints to the query

    }

    System.err.println("Registry query: " + query);                 //

    // submit our query

    return null;
}

We can start by setting our base query. Change:

    String query = "";

to:

    String query = "ServiceType like '%CONE%'";

The while loop that is already in the stub parses out each word and sets it to keyword. For each keyword, we want it to match any of a selected set of text metadata. The constraint for each word then might look something like this, place just after keyword is set:

String constraint = "Title like '%" + keyword + "%' or " +
    "ShortName like '%" + keyword + "' or "

You could add any other attributes (e.g. Description, Subject, etc.).

In this implementation, we will require that all given keywords be matched somewhere. So we add each keyword constraint to the query with the following:

query += " and (" + constraint + ")";

Now we're ready to submit the query. This is done in exactly the same way we did it in the search() method, so we can just cut and paste the remaining code in (replacing the return null):

  // get a registry service object
  Registry regService = new RegistryLocator();

  // get an interface object that can accept our query.
  RegistrySoap regInterface = regService.getRegistrySoap();

  // Now submit the query
  ArrayOfSimpleResource results = regInterface.queryRegistry(query);

  // return all of the results
  return results.getSimpleResource();

You are done.

Exercise: Adding Registry Support to the Cone Search Client

The stubbed portion of ConeCaller.java that we need to add to is in the main() method; it looks like this:

    if (query != null) {
        // search the registry for ConeSearch services


    }

We can follow a bit of the pattern we used in FindConeSearch.java='s =main() to query the Registry. So to start, we can insert into the above if-block the following:

  SimpleResource[] cs = FindConeSearch.keywordSearch(query);
  if (cs == null || cs.length == 0) {
      System.err.println("No matching cone search services found");
  }
  else {
      System.out.println("==================================");

      for(int i=0; i < cs.length; i++) {
          // print out the title, URL, & number of records
          // returned by the cone search

      }
  }

The for loop steps through the matched service descriptions from which we need to extract the baseURL. We need to remember to fix any messy URLs, we can insert as the first step within the loop, the fix:

   String baseURL =  
      FindConeSearch.correctBaseURL(cs[i].getServiceURL());

Next we'll print out some some info from the resource description:

   System.out.println("Title: " + cs[i].getTitle());
   System.out.println("Address: " + baseURL);

As for submitting a Cone Search and counting the results, let's put that into a separate function.

   int nr = countHits(ra,dec,sr, baseURL);

   System.out.println("Matching catalog records found: " +
                       nr);
   System.out.println("==================================");

Our first implementation of countHits() might look pretty simple, borrowing from the readVot() function:

public static int countHits(double ra, double dec, double sr,
                            String coneUrl) 
    throws Exception
{
   URL cone = new URL(coneUrl + "RA="+ra+"&DEC="+dec+"&SR="+sr);

   VOTWrap.VOTable vot = VOTWrap.createVOTable(cone.openStream());
   VOTWrap.TableData tdata = 
       vot.getResource(0).getTable(0).getTableData();
   return tdata.getTRCount();
}

However, there are lots of things that can go wrong, so we better try to catch some of the errors with a try-catch enclosure:

public static int countHits(double ra, double dec, double sr,
                            String coneUrl) 
    throws Exception
{
   try {
      URL cone = new URL(coneUrl + "RA="+ra+"&DEC="+dec+"&SR="+sr);

      VOTWrap.VOTable vot = VOTWrap.createVOTable(cone.openStream());
      VOTWrap.TableData tdata = 
          vot.getResource(0).getTable(0).getTableData();
      return tdata.getTRCount();
   }
   catch (  ) {

   }
}

We'll catch three possible errors. First, despite our effort to clean up the URL, it may still be erroneous:

   catch (MalformedURLException ex) {
       System.err.println("Problem with URL: " + coneUrl);
       return 0;
   }

Next, the server may be down...

   catch (ConnectException ex) {
       System.err.println("Service not responding.");
       return 0;
   }

Finally, we'll catch any other possible problems:

   catch (Exception ex) {
       System.err.println("Unknown service error.");
       return 0;
   }

The most common reason for this last error condition is due to the VOTable not having a TABLE in it (causing a NullPointerException). Instead the VOTable only contains an error message. There is a standard for encoding errors, so in principle we could (and should) pull this out and display it. Unfortunately, very few Cone Search implementations follow the standard correctly.

Our implementation is mostly complete. To get our countHits() function to compile not that we have put in the catching of exceptions, we need to remove the throws clause in the signiture:

public static int countHits(double ra, double dec, double sr, 
                            String coneUrl) 
{
    try {
        URL cone = new URL(coneUrl + "RA="+ra+"&DEC="+dec+"&SR="+sr);

        VOTWrap.VOTable vot = VOTWrap.createVOTable(cone.openStream());
        VOTWrap.TableData tdata = 
            vot.getResource(0).getTable(0).getTableData();
        return tdata.getTRCount();
    }
    catch (MalformedURLException ex) {
        System.err.println("Problem with URL: " + coneUrl);
        return 0;
    }
    catch (ConnectException ex) {
        System.err.println("Service not responding.");
        return 0;
    }
    catch (Exception ex) {
        System.err.println("Unknown service error.");
        return 0;
    }
}