Integrating Existing Tools Into The VO - Part I

Note: The presentation of this is only scheduled to last about 20 min. Obviously there is more here than can be discussed in that time and so this document is aimed at the proceedings, the presentation itself will be much quicker and cover only the bullet items.

IRAF Web Services Development

Introduction

Many of the tools and protocols beings used in the VO (and the Summer School) today are naturally new to astronomy (e.g. VOTable, SIA/Cone protocols, the VOPlot viewer), or were easily adapted to work in a VO environment given their existing purpose to work as a front-end for archives or data centers (e.g. Aladin already supported tables so adding VOTable wasn't difficult) or explore large multi-dimensional datasets (e.g. Mirage). Newly developed interfaces such as the STIL table library and the TOPCAT viewer built with it are developed in a "modern" language like Java. In the case of IRAF, however, we have an extensive collection of existing tasks that do everything from the strictly utilitarian processing of image data to highly specialized analysis tools. Additionally IRAF is a scripting/user environment already familiar to many astronomers that can be used in developing new science applications. Tasks, however, are implemented in a "custom" language (CL scripts or SPP) and lack the needed system infrastructure (e.g. native XML parsers or SOAP message handling) to easily handle VOTables or web-services interfaces, and resources needed for major system enhancements are scarce. Many other legacy systems share these same traits, but that doesn't make the systems or applications any less desirable in the VO era.

So, how can the VO make use of IRAF and the many other legacy software systems used by astronomers every day, and how can developers in environments that don't traditionally support web-services or network/web protocols make use of VO data? The answer, or at least one possible answer, is the tried-and-true method of using interfaces and wrappers.

Project Goals

The primary goals of the IRAF integration project are to

This aspect of the larger IRAF-VO development project focuses mainly on server-side integration of the system. We've seen from the mini-tutorial that it is possible to use host tools from within IRAF tasks to do much of the actual work required of a VO client application, however more complete and seamless integration of the system itself is an ongoing effort and is not discussed here.


Challenges

IRAF tasks may be compiled executables, script tasks, host commands, or some combination and so "standard" approaches to legacy code deployment as web-services are not always practical or appropriate. We can treat all tasks as a form of commandline tool (i.e. call the binary directly or use a host wrapper on CL scripts) but we then ignore our goal of limiting the number of changes required to each task and still need to handle the particulars of each task (parameters, binary location, etc). Additionally we must still deal with the serialization of VO data and protocols (VOTable, SOAP message handling, URLs, etc) in an environment where we may not be able to easily use existing interfaces for such things. Lastly, we must address the move from what is traditionally a desktop analysis environment operated by an intelligent user to the distributed data/computing model of the VO where interactive applications need to operate in batch mode easily, and it is now remote software invoking the task.

Approach

There are far too many tasks to consider integrating each into the VO individually, even if only trivial modifications are needed to each task -- a more systematic approach is needed. The strategy adopted then was to:

Keep it Natural and Build Bridges:
  • Java is a natural environment for implementing the web-service, dealing with SOAP and attachments, and URL handling.
  • IRAF is the natural environment for implementing new science functionality
  • So, figure out how Java can "run" IRAF the way a user would.
Separate the System from the Application:
  • Use core infrastructure where possible, i.e. stock IRAF/packages
  • Hand-code support routines only where necessary
  • Automate the generation of the web-service endpoints
Think BIG:
  • Interfacing individual tasks introduces too many complexities and too much hand-coding per task
  • Interface the entire system and we can separate the web-service from the legacy code entirely

System Overview

Figure 1.

The architecture for the current implementation is shown in Figure 1. At (1) the user initiates a request to the web-service using any form of client application (Java, Python, Perl, etc) and sends the standard SOAP message perhaps containing attachments of data to be processed. The web-service endpoint at (2) is auto-generated Java code to handle the message (unpack or convert data, validate input, etc) and construct a command string to be executed in the legacy system. Utility code (hand-written) in (3) sends the message as a simple text string over a socket to an "dapter process" that is already running on the system. This adapter at (4) is a Unix process that may be initiated from the xinetd super-daemon that spawns the entire IRAF system as a child process and opens an input socket to accept command input. Commands are passed through to the IRAF CL in (5) just as a user would have entered them at the keyboard, and output is returned along the same path to the user.

The WEBSVC external package is like any other IRAF package but was created to hold the various small wrapper scripts or new science tasks we wish to deploy. Wrapper scripts are often necessary for a number of reasons:

In addition, any other external package, core system task or host command is available just as it would be through the CL command prompt. The "IRAF adapter" process starts a CL session and initializes the child CL with the needed environment.
Figure 2. Sample Echo Service Config File and Generated Endpoint Code 
  <svc>
    <name>Test</name>
    <session>
        <port>8604</port>
        <logfile>Test.log</logfile>
    </session>
    <method>
        <name>echo</name>
        <ret_type>String</ret_type>
        <arg>
            <id>input</id>
            <type>String</type>
            <allow_null>no</allow_null>
        </arg>
        <iraf>
            <cmd>print $input</cmd>
            <return_type>String</return_type>
        </iraf>
    </method>
  </svc>

    public String[] myecho(String[] input, String session)
        throws RemoteException {
        	:
        baseaddr = baseaddr + session;
        try {
            input = Stringvalidate(no, input, null);

            // Get an IRAF client connection.
            if (session.trim().length() != 0)
                _clPort = Integer.parseInt(session);
            CLSocketClient cl = new CLSocketClient(_clPort);

            // Send the command.
            result[0] = cl.exec("cd " + baseaddr + ";" + "print " +
                    convert(input, "String") + "");
            return result;
        } catch (Exception e) {
            throw new RemoteException(e.toString());
        } finally { }
    }

Developing a new service is a matter of writing and XML configuration file that describes the web-service interface (name, argument lists, etc) and the associated IRAF functionality (i.e. the command string to be sent). XSLT strylesheets transform this XML config file into the Java endpoint code (much of which is redundant with other services), client test code and the user documentation via Javadocs. Build scripts tie this all together and will automatically deploy the service if needed. Abstract data types such as "VOTable" trigger code to be generated that automatically converts the VOTable IRAF doesn't understand to a FITS binary table it can deal with and passes this temporary table through for processing; output tables are likewise re-converted back to VOTable when returning from the service.

A sample (minimal) configuration file for an echo service is shown in Figure 2. The <svc> root element defines the service class, there may be any number of <method> blocks defining individual services. The <method> blocks define the calling argument list for the service as well as the associated IRAF command in the <iraf> block. In this case we simply call the print command of the CL with the input argument and return a string. The <session> block allows us to configure in part how the service will be deployed, e.g. running on a different inet port so that simple calculator services won't be blocked while a long-running image mosaicing service is processing another request. Any number of service classes may be running simultaneously (including private user sessions), each operating on a different socket. In the generated code fragment we see the use of the CLSocketClient class which is one of the hand-coded utilities need, here it simply sends the command string to the adapter process and returns the result to the service.

 


The Science Theme Services -- A Real World Example

The GALCAS/GALMORPH demo services developed for the science theme use both the above system to deploy the web-service and some of the techniques discussed in the IRAF mini-tutorial to implement the science functionality as a new IRAF task. In this case we used a 3-rd party external package (GIM2D from Luc Simard at CADC) for Bulge+Disk decomposition in GALMORPH and a modified version of SExtractor to compute the CAS (Concentration, Asymmetry and Surface brightness) parameters for both tasks. Unlike most IRAF packages, source code for GIM2D is not available from the author and documentation is sometimes anecdotal at best.

To achieve the science aim of the task, a number of individual tasks in the package needed to be combined to automate the process a user would normally execute as separate steps in the analysis. We also tuned parameters and the script itself to better handle the WESIX input catalog we are using to find galaxies and the SDSS image data we are expecting (since image-specific quantities such as seeing estimates, pixel-scale, detector gain etc are needed by GIM2D).

WESIX gave us a cross-matched catalog for galaxy identification, but the service doesn't provide the required localized background estimate as one of its options. This was an easy addition to SExtractor but meant we had to additionally run a localized SExtractor and merge the resulting table with the WESIX input to produce a suitable input table for the GIM2D tasks. The STILTS table utilities were used to convert the input tables to FITS tables for processing in the script, thereafter we used standard TABLES package tasks for manipulating tables.

With the proper input tables for our selected galaxies GIM2D ran much more reliably, but unfortunately it also ran more slowly (on the order of minutes/galaxy cutout to fit the model). As part of a research program this may be acceptable, but as a demonstration during a tutorial it wasn't nearly fast enough to process a sufficient number of galaxies to show a useful plot or provide students with a hands-on experience.

The next step was to create a faster service that could be used somewhat interactively. The script developed so far was already using a modified SExtractor with custom output columns so the decision was made to use a further modified version to compute the CAS parameters at the same time. For the GALCAS service we stop at this point and format a return table, for GALMORPH we continue on to the GIM2D processing stage and return all derived values in a merged table. This brought the total processing time down to a few tens of seconds for large clusters with only a slight overhead for development and processing


Lessons Learned

In developing both the system used to deploy IRAF tasks as web-services, and in looking at the different use-cases presented by existing applications of all types, of number of lessons were learned: