US-VO Website

SkyNodes

OpenSkyQuery(OSQ) portal allows cross matching between major catalogs in small areas of sky. It provides selection of parameters from catalogs using criteria based on comparisons between values contained in one or more catalogs. Catalogs are registered in the VO Registry as Skynodes. The OSQ portal queries the registry to find which Skynodes are available. The OSQ portal is built upon web services.

This session will demonstrate construction of a simple web service. The requirements and specifications of the Skynodes will be looked at. The specification includes a WSDL description of the Skynode service from which the server side stubs may be generated. This has already been performed and the entire server has been filled out in JAVA. We will look at configuring this JAVA Skynode server package for a particular catalog.

This is an outline of what will be covered:

SkyNodes
SOAP WebServices
Basic SkyNode Server
Full SkyNode Server

OpenSkyQuery

OpenSkyQuery is a web portal which provides several features for accessing Skynodes. If we visit the OSQ portal we can experiment a little and see the list of Skynodes. Some of these are locally maintained at JHU while others are available remotely. The cross-matching between these astronomical catalogs is possible through the use of XML and SOAP technology, i.e. web services. These Skynodes are 'discovered' and loaded into the portal by searching the NVO Registry at STScI/JHU (see the course section on Registry Client for more detail) We can go to the registry directly to see which Skynodes are available.

The registry metadata contains fields which indicate the type of Skynode, endpoint, primaryKey and table. These are all required fields for the OSQ to properly access the table fields and perform queries.

Currently the Info on Node is collected from tables in the Skynode build. Alternate suggestions (e.g. Astrogrid) have recommended that the table info from each node would be included in the registry metadata however with extensive database schemas, this has some affect on the curation and publication of 'service' specific information. We will leave this issue to a future date.

It's worth mentioning and quickly looking at some other importance features with OSQ.

Help
OSQ Architecture
Execution Plan
Build a Node (.NET) (old link on this to NVOSSI pages
Limitations
Let's look at some of the sample queries and the query builder with OSQ.

Skynode specification

The Skynode has been formally proposed as a Standard to the IVOA. We can look at these links below to see the related documents which should be used as a reference for development which complies with the current standard. NOTE: The Skynode is tightly coupled to the ongoing development of a VO Query language standard and as such can be found under this category on the IVOA Twiki site

IVOA Twiki for VOQL
- Skynode0_9.WSDL on Twiki
- Registry Schema for Skynode

Established ADQL (Newest proposal version 1.0)

VOQL Level 2
- VOQL 2 = ADQL 0.91 + Skynode + portal
Spec
Schema
Can have ADQL w/o a Skynode, but you can not have Skynode w/o ADQL (originally 1 spec, split)
JAVA parser in FullSkyNode package

Implementation

SOAP WebServices

At this point we have already heard a little about SOAP/XML and WebServices. The other examples for a Simple Server are not WebServices since they do not contain a machine readable description which allows a set of client stubs to be constructed. Generally when we speak of WebServices we mean a services accepting and transmitting SOAP messages and defined with a WSDL.

AXIS

We could write JSP or servlet which behaved as WebServices but the general approach is to use a Framework like :

SunOne
The Mind Electric GLUE
BEA WebLogic
IBM WebSphere
Microsoft .NET
Apache AXIS.

These frameworks serialize and de-serialize objects from the native language to/from XML. They also contain some tool which converts from WSDL to native classes.

We have selected AXIS for today although some of the services you have seen are from .NET. NOTE: In theory we would expect interoperability via SOAP standards to be seamless, there are variations between Axis versions in the serialization methods such that KNOWN deserialization errors will occur with OSQ. This is due to changes in the SOAP protocol and variations between Java and .NET (namely VOData deserialization with the .NET portal)

Ordinarily at this point we need to install axis but for your convenience the NVOSS software comes with Axis precopied into Tomcat. For some reason Axis does not provide a WAR file rather a directory still it is easy to install- for doing this manually you would simply need to copy the contents (axis) of the webapps directory(e.g. %NVOSS_HOME%\java\axis-1_2beta\webapps) from AXIS(in the java directory)into the webapps directory of our webserver (e.g. %CATALINA_HOME%\webapps).

Restarting Tomcat may be required during in this exercise since it only picks up on new WAR files not necessarily new directories.

If you already have Tomcat running

bouncetomcat

Or simple start Tomcat

starttomcat

And check http://localhost:8080/axis/happyaxis.jsp

A few additional jars have been added to axis which you otherwise have to download separately.

The WebService Server

So this may seem complex but actually making a web service is in many ways easier than the type if pages we have already constructed. This is because the framework does all the hard work. Lets look at a WebService called Nums in simplewebservice\server\src\sumsch\Nums.java

In AXIS one may take a file like this and rename it .jws place it in the axis folder and it works. That however is a little like our trivial JSP example from Simple Server - not very useful when we get even slightly more complex.

So we need to make a deployment descriptor - this is in the server directory called deploy.wsdd. To deploy this we need to put our classes into AXIS. The easy way to do this is make a jar (ant can do that ) and put in in the WEB-INF/lib of axis. We also need to run the adminclient with the descriptor.

Axis comes with some special tasked for ant so we can have a deploy target such as :

    <target name="deploy" depends="dist">
   <move file="${dist}/nums.jar" todir="${axis_deploy}/WEB-INF/lib"/>
   <axis-admin
       failonerror="true"
       xmlfile="deploy.wsdd"
   />
    </target>

So in the server directory we may do

ant deploy

Axis may not find the jar when it is deploy in which case you may need to start and stop tomcat do this with :

bouncetomcat

The WebService Client

This is where the real beauty of SOAP Services lie. You have seen some already but lets look at this one too. First we need to edit the web.xml file to have correct deployment description so edit in the client/conf dir the web.xml file to have the correct path.

<env-entry-value>/nvoss2005/java/data</env-entry-value>

Lets just look at NumsClient.java in the client directory.

The needed interface to the server is all generated by AXIS (another ant task def):

    <target name="axis" depends="">
   <axis-wsdl2java output="${generated}" url="http://localhost:8080/axis/services/Nums?wsdl"/>
    </target>

Hence we may simply type in the client directory

ant test

To show what you need to do to have a JSP page call such a service a deploy target has been added to the build.xml. This builds a war file with a webform to call the service. The jsps are in the web directory.

Still in the client directory

ant deploy

View this at:

http://localhost:8080/numsc/numform.jsp

WebService Server from WSDL

We can also generate a server from a WSDL not just a client. The ConeSearch for SDSS for example is a WebService at http://skyserver.sdss.org/vo/dr2cone/SDSSConeSearch.asmx.

The conewebservice directory contains a build.xml which will generate the server side java classes for this WSDL. Just type

ant axis

Now look at generated\edu\jhu\pha\skyservice\ConeSearchWSSoapImpl.java. Notice now we are not dealing with XML or web pages just pure java objects.

Deployment is also built in the ant file so :

ant deploy

should work.

Basic SkyNode Server

(Many thanks to Ramon Williamson NCSA for the Java SkyNode)

The OpenSkyQuery web site contains a download of the .NET SkyNode. This requires SQLServer and a special set of database routines to do the XMATCH and spatial searches. This is a Full SkyNode.

It was considered useful to also define a Basic SkyNode which could accept queries in ADQL. In the dev directory is an implementation of this in Java. Let's go to the directory where this is located and look at the available files. We will go through the BASIC version and then reuse our local skynode and look at a little more detail with the FullSkynode.

cd c:\nvoss2005\java\dev\basicskynode

One of the many key files in this directory is the README file.

The initial outline for this was generated from the WSDL contained in the specification of the SkyNode. In this case the WSDL provides a way to define an standard interface. Note the Skynode.wsdl is prestaged into the directory.

Client.

There is a client in the client sub dir of the basic skynode. It is possible to pass this a different end point to try other SkyNodes by passing the endpoint url on the command line.

Again most of this client is generated the only code that was written is in the file SkyNodeClient.java.

Let's change directories and go into the basicskynode/client directory. Again with Ant we generate all the client stubs and compile everything with :

ant compile


You may try this against SDSS or ADIL to see it works with .NET and Java.

ant testADIL
or
ant testSDSS

Server

The SkyNode specification is generally to expose a relational database through a standard interface. So we need a database. We could use MySQL or SQL Server (which is now free). In the later case though we should then use the .NET SkyNode also - this is the fastest and best way to make a SkyNode. For now lets concentrate on MySql. You can also for a student project try adapting this to other databases. See some of the sample config files.

The database access for a Skynode is managed via the configuration file. There is a README file of the BasicSkynode top directory (i.e. nvoss2005\java\dev\basicskynode) that explains the required configuration settings. Namely you need to specifiy element tags that are specific to your database. In our exercise since we are using MySql the file with the required settings has been named Configure.xml_mysql_abell, and contains the following elements. Notice the column tags have not been included, but they are also required and can be found in the configuration file.

	<MySQL  Configuration file  for abell Catalogue--> 
    <ShortName>ABELL</ShortName> 
    <XSLTPath>c:\nvoss\dev\basicskynode</XSLTPath> 
    <XSLTFile>adql2mysql_axis.xslv/XSLTFile> 
    <!--Database-Specific Parameters -->
	<JDBCDriver>com.mysql.jdbc.Driver</JDBCDriver> 
    <JDBCurl>jdbc:mysql://localhost/test</JDBCurl> 
    <Username>skynode</Username> 
    <Password>skynode</Password> 
    <ArchiveName>abell</ArchiveName>

Create Database

For most DBs we will need to load data. To build a full skynode in SQLServer there is extensive documentation on SkyServer.org.

The first thing we will need is a CSV file. If you have one great if not we can make one from the Abell catalogue using OpenSkyQuery.

NOTE: You have Abell.csv in your nvoss2005/java/data directory.

Now we need a table. MySql does not allow DEC as a column name ! Run MySql and create the table in the default test database (or equivalent schema if using another table)

mysql> use test  \g
mysql> create table Abell  ( unique_id int, name varchar(50), ra double, declination double , _count int, bmtype varchar(50), redshift real, rich varchar(50), dist varchar(50), vmag real, x real, y real, z real  ) \g

Now we import the data with :

mysqlimport  --verbose=TRUE --replace=TRUE  --fields-terminated-by=, test %NVOSS_HOME%\java\data\Abell.csv 
or on UNIX 
mysqlimport --local --verbose=TRUE --replace=TRUE --fields-terminated-by=, test $NVOSS_HOME/java/data/Abell.csv

If you have problems with the priviledges in performing the import, try appending to the mysqlimport

-u root -p

So We have a database, Normally we would tune this and create indexes etc. but we wont do that here.

To make this available to JDBC we have to grant some privs to our user:

>mysql  -u root

mysql> GRANT ALL PRIVILEGES ON test.* to skynode@127.0.0.1 identified by 'skynode';
:
:
mysql> FLUSH PRIVILEGES;

The Java SkyNode.

This prescription of SkyNode is sometimes referred to as the 'Open' Skynode (also in sync with the OSQ name) because it is not limited to a single framework (e.g. .NET) and Java provides a stable interoperable set of web service interfaces. If we are using this MySql database then the build.xml is already setup for that. If not we need to modify build.xml - near the top the config file is named. A bunch of config files are provided by Ramon for different DBMS. If you would like to try out another database, you can use one of the example files.

In the config file we ned to specify a full path to the Style Sheets this is important even if following this example especially if you are on unix look in the config file for a line like (NOTE you may need to edit this path) In any case lets look at Configure.xml_mysql_abell in the top level directory

  cd    c:\nvoss2005\java\dev\basicskynode

and then edit the Configure.xml_mysql_abell file to have the correct XSLT path to point to your basicskynode directory. Note for Linux users you should use the opposite slash between directories.

  <XSLTPath>c:\nvoss2005\java\dev\basicskynode</XSLTPath>

If you don't have the abell schema then the config file will need to be modified a little more. This config file also contains the username and password for conecting to the DBMS as well as the driver class name. The Driver for the dbms needs to be available in AXIS WEBINF/lib - mysql.jar has already been added if you used the standard setup. To compile and test this all we need to do is:

c:\nvoss2005\java\dev\basicskynode>ant dist

To deploy it if the test worked all we need to do is:

c:\nvoss2005\java\dev\basicskynode>ant deploy

We may check our deployment at http://localhost:8080/basicskynode/servlet/AxisServlet. This may require a stop and start of tomcat to see it show up. Once we see it listed we should be able to run our client which we looked at earlier. In the Client directory ..

ant testLocal

Registration

Once you have a node you need to register it. The only registry where this is currently possible is the STScI/JHU registry at http://nvo.stsci.edu/voregistry/index.aspx . In the case of a basic node the Compliance field needs to have the word BASIC in it.

The OpenSkyQuery portal uses the registry to locate nodes. You will notice the ADIL node on the portal - this is a Java node. You may also notice it does not have the + and - symbols since it can not be included in a XMATCH.

Full SkyNode Server

(Many thanks to Tom McGlynn for implementing the Java FullSkyNode)

The BasicSkynode implementation has been further extended to provide a mechanism for performing crossmatches. This is now in an early stage of developement but we will go over here an additional exercise to build and test the FullSkyNode implementation. If you remember from the OSQ portal architecture, the main difference between the Basic and Full Skynode is that we now are passing an ExecPlan to the Nodes. This allows nodes to serially pass queries.

Server

Since we have already created our MySQL database, we can proceed to look at a few additional capabilities now with the FullSkynode. As with the BasicSkynode there is a configuration file although it is now stored in the conf subdir. In addition to the element tags (settings) described in the BasicSkynode, we now need to designate the metafinderclass in the configuration file Configure.xml_mysql_abell

	<MetaFinderClass>net.ivoa.node.ConfigMetaFinder</MetaFinderClass>

This is because an additional class set was designed for handling other forms of Metadata (i.e. column specs) and we need to use the default setting for our exercise. If you are curious and would like to see another example you can reveiw the config.xml and see how the HEASARC archive Skynode is handled.

We also need to specify the SkyNodeClass tag.

    <SkyNodeClass>net.ivoa.node.FullNode</SkyNodeClass>

Database

We will use again the Abell MySQL database setup above. We can still use the same BasicSkynode stylesheet which converts the ADQL to MySQL. Again, you may need to edit this file. It is now located in a subdirectory fullskynode\conf. Again, edit the conf\Configure.xml_mysql_abell file for the xslt path. Note with UNIX you need to remove the c: and change the slash direction.

<XSLTPath>c:\nvoss2005\java\dev\basicskynode</XSLTPath>

Another IMPORTANT change to the configuration which is platform/MySQL dependency related is on the Unix systems (Mac/Linux) modify the IncludeTable and TableName tags in the Configure.xml_mysql_abell to have a capital 'A' for the Abell table. Note on Windows this is not required and in fact may cause a failure in the clients.

<!-- Tables to include in result --> <IncludeTable>Abell</IncludeTable> 
    <ExcludeColumn>abell.x</ExcludeColumn> 
    <ExcludeColumn>abell.y</ExcludeColumn> 
    <ExcludeColumn>abell.z</ExcludeColumn> 
    <!-- 
    Metadata for Tables --> <MetaTable> 
    <Table 
        ID= "Table1"> 
            <TableName>Abell</TableName>

The Java SkyNode.

With the BasicSkynode we did not go over the actual java implementation code but we will introduce it here because the FullSkynode contains the functionality of the BasicSkynode as well. Since we have used Axis to generate the java class files from the Skynode.wsdl, we are really only interested in the non-autogenerated code. This can be found in (and under) the directories:

net/ivoa/node and net/ivoa/client

and the file

net/ivoa/SkyNode/SkyNodeSoapImpl.java

is where the ExecPlan and PerformQuery functions are setup, although the actual lower level methods are in the fullnode.java class. You can look at these files if you are interested in the details of the cross-match and handling of the node exec plan. There are several features in these methods that perform translations between ADQL to SQL and again, the configuration settings provide the appropriate paths to the utilites needed.

So now we can build our FullSkynode server. As before with the BasicSkynode we need to generate our classes, compile the code, and create a war file. This includes the database connection.

ant dist

Deploy the war file to the Tomcat server, and then you can run unit tests to check that the server has proper connection to the database.

ant deploy
ant test

We may check our deployment at http://localhost:8080/fullskynode/servlet/AxisServlet. This may require a stop and start of tomcat to see it show up. Once we see our Axis service running we can run a few clients starting with a client to our local skynode service.

ant xtest0

We can also run a couple other client tests

ant QueryTest
ant testSDSS

Registration

Registering the FullSkynode is identical to the BasicSkynode except not you need to use the word FULL in the Compliance field. As before, the STScI/JHU registry at http://nvo.stsci.edu/voregistry/index.aspx contains the Skynode resources for the OpenSkyQuery. You can edit the BasicSkynode entry of as before perform and advanced query and edit the FullSkynode example by creating a new identifier, endpoint, etc. The advanced query we use...

identifier like '%ivo://sdss.jhu/services/openskynode/dr2%'

The new skynode service endpoint is:

http://localhost:8080/fullskynode/services/SkyNodeSoap

The OpenSkyQuery portal now should contain an Abell catalog FullSkynode entry. We can perform some basic queries and currently we are in the process of working out the portal cross-match capabilities. This is still a work in progress with the Open Skynode implementation.