VOTables

Introduction and Background

VOTable is an XML-based format for representing astronomical catalogs (i.e. tabular data) in a uniform manner across all VO software and services. XML allows the use of industry standard tools and software introduced earlier, but more importantly allows us to capture the rich set of metadata associated with the data and/or columns of the table.

The immediate ancestors of the format are Astrores developed at CDS and the eXtensible Scientific Interchange Language (XSIL). several of the NVOSS faculty were heavily involved in the definition of the format and VOTable was one of the first standards developed by the VO and adopted by all IVOA partners.

The VOTable Document Format

The formal specification of VOTables is maintained at the IVOA site http://www.ivoa.net/Documents/latest/VOT.html and we'll only review the highlights of the format here. Key features of VOTables include:

• Full representation of metadata
• A hierarchy of RESOURCEs
• Separation of metadata and data within each RESOURCE
• Use of UCD (unified content descriptor) in FIELD definitions
• Can reference data contained w/in the VOTable or remote and/or binary streams. Tables may be:
• Pure XML
• Simple binary
• FITS Binary Tables

Sample VOTable

<?xml version="1.0"?>
<VOTABLE version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/VOTable/v1.1">
  <COOSYS ID="J2000" equinox="J2000." epoch="J2000." system="eq_FK5"/>
  <RESOURCE name="myFavouriteGalaxies">
    <TABLE name="results">
      <DESCRIPTION>Velocities and Distance estimations</DESCRIPTION>
      <PARAM name="Telescope" datatype="float" ucd="phys.size;instr.tel"
        unit="m" value="3.6"/>
      <FIELD name="RA" ucd="pos.eq.ra;meta.main" ref="J2000"
        datatype="float" width="6" precision="2" unit="deg"/>
      <FIELD name="Dec" ucd="pos.eq.dec;meta.main" ref="J2000"
        datatype="float" width="6" precision="2" unit="deg"/>
      <DATA>
        <TABLEDATA>
          <TR><TD>010.68</TD><TD>+41.27</TD></TR>
          <TR><TD>287.43</TD><TD>-63.85</TD></TR>
        </TABLEDATA>
      </DATA>
    </TABLE>
  </RESOURCE>
</VOTABLE>

Looking at the specification and the example above we find that the data model for a VOTable is composed of:

• VOTable = hierarchy of Metadata + associated TABLEDATA arranged as a set of TABLEs.
• Metadata = PARAMeters + INFOs + DESCRIPTIONs + LINKs + FIELDs + GROUPs.
• TABLE = list of FIELDs + TABLEDATA.
• TABLEDATA = stream of rows (TR tags)
• Row = list of cells (TD tags).
• Cell = Primitive (or variable-length list of Primitives, or multidimensional array of Primitives).
• Primitive = integer, character, float, ....

To help take some of the mystery out of a VOTable, let's break down the sample in places and explain each element in more detail.

VOTABLE

This is the root element of the XML document tree and there can therefore be only one occurance in a file. The children of a VOTABLE will typically include as part of the metadata:

• <DESCRIPTION>, <PARAM>, <INFO>
• <COOSYS> defines celestial coordinate system

Additionally, and the part that we're usually really interested in, one or more:

<RESOURCE>

Basically, a set of related tables. The RESOURCE is recursive (it can contain other RESOURCE elements), which means that the set of tables making up a RESOURCE may become a complex structure.

A RESOURCE may have one or both of the name or ID attributes; it may also be qualified by type="meta", meaning that the resource is descriptive only (does not contain any actual data in any of its sub-elements). Finally, the RESOURCE element may have a utype attribute to link the element to some external data model (introduced in version 1.1, see section 4.5 of the spec)

TABLE

A TABLE may contain descriptive or other metadata elements such as

• FIELD - Description of a table column
• PARAM - Constant value (i.e. a Table with one Cell)
• GROUP - Logically associates FIELDs or PARAMs
• LINK - Pointer to other documents or data

Data itself begins with the <DATA> tag in table cells in the same order as the <FIELD> definitions. All records MUST have the same format, empty cells are simply denoted with an empty tag.

FIELD

A FIELD is a description of a table column that may contain additional descriptive elements such as LINK, VALUES or DESCRIPTION. Attributes are used to specify

• a name/ID if the field is to be referenced later,
• the type/size/precision of the values
• units (using a controlled vocabulary),
• the ucd standardized classification of the quantity (details to be covered later),
• a utype that is used to reference an external datamodel (utype = "datamodel_identifier:role_identifier)

PARAM

A PARAM is like a FIELD but keeps a constant value. It has the same set of attributes as FIELD and can be thought of as a global definition that applies to the entire RESOURCE.

DATA

The DATA element is unique in a TABLE, however a RESOURCE may contain multiple TABLEs. There are three possible formats for the data:

• TABLEDATA - pure XML table composed of TR elements to define the rows and TD elements for each cell
• FITS - FITS binary table, may contain an extnum attribute to refer to a particular extension of a stream. Header keywords are typically encoded as PARAMs in the metadata.
• BINARY - for efficient transfer, data are encoded (e.g. base64 or gzip)

FITS and BINARY formats must contain a <STREAM> element, for instance

<TABLE> 
  <FIELD  ....>
  <DATA>
    <FITS extnum="2">
      <STREAM encoding="gzip" href="ftp://archive.nvo.org/myfile.fits.gz"/>
    </FITS>
  </DATA> 
</TABLE> 

GROUP

The GROUP element is a relatively new feature meant to allow the logical association of FIELD and PARAM metadata elements. As you can see from the example this form of association is not unusual in astronomical tables, however the GROUP element is not supported by all current parsers/writers (STIL java library, C++ Parsers from VO-India appear to be the only ones)

<TABLE name="Nutation and Abberation"">
  <FIELD name="Date">
  <GROUP name="Nutation">
    <FIELD name="Long">
    <FIELD name="Obl">
  </GROUP>
  <GROUP name="Abberation">
    <GROUP name="Equinox 1950">
       <FIELD name="C_1950">
       <FIELD name="D_1950">
    </GROUP>
    <GROUP name="Equinox 1955">
       <FIELD name="C_1955">
       <FIELD name="D_1955">
    </GROUP>
  </GROUP>
</TABLE>

Reading VOTable Example

In the $NVOSS_HOME/java/dev/readvotable directory you'll find a sample program using the VOTwrap interface for parsing the file. Below we show a slightly modified version of that same task to be used as a basis for discussion during the presentation.

package readvotable;
import edu.jhu.pha.ivoa.*;
import java.io.*;
import java.text.*;

class ReadVotable {
 public static void main(String[] args) throws Exception{
   int ind =0; 
   if (args.length == 0 ) {
	System.out.println("You need to suply the filename of a votable ");
	System.exit(1);
   }
   readVot(args[0]); 
 }

 public static void readVot(String fname) throws Exception{
	InputStream is = new FileInputStream(fname);	
	VOTWrap.VOTable vot = VOTWrap.createVOTable(is);

	VOTWrap.Resource res = vot.getResource(0);
	VOTWrap.Table tab =  res.getTable(0);
	int fcount = tab.getFieldCount();
	int rcount = tab.getTableData().getTRCount();

 	for (int f=0; f < fcount; f++) {
	   VOTWrap.Field field = tab.getField(f);
	   System.out.print (field.getName()+":"+field.getUCD()+" ");
	}

	System.out.println();
	System.out.println (" There are "+ fcount +" fields on " + rcount + " rows:");

 	for (int r=0; r < rcount; r++) {
	  VOTWrap.TR row = tab.getTableData().getTR(r);
 	  for (int f=0; f < fcount; f++) {
	     VOTWrap.TD td = row.getTD(f);
	     System.out.print (td.getPCDATA()+"   ");
	  }
	  System.out.println();
	}
 }
}

Exercise

Beginning with the above program or the original code, modify the task to read a VOTable and output one or more HTML tables (i.e. only the data or separate tables for the metadata and tabular data). The java print() or println() would be used to write out the additional markup.

XSLT Transformation Example

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" 
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/VOTABLE">

<html><body>
 <xsl:for-each select="RESOURCE/TABLE">
  <table border="1">
  <tr> <xsl:for-each select="FIELD">
     <td><b><xsl:value-of select="@name" /> </b></td>
  </xsl:for-each> </tr>

  <xsl:for-each select="DATA/TABLEDATA/TR">
       <tr>
        <xsl:for-each select="TD">
          <td width="120"><xsl:value-of select="." /></td>
        </xsl:for-each>
       </tr>
  </xsl:for-each>
  </table>
 </xsl:for-each>
</body>
</html>
</xsl:template>
</xsl:stylesheet>

Examining VOTables

The earlier introduction to VO applications mentioned a number of tools that make use of VOTables for image overlays (Aladin), plotting (VOPlot) or interaction (TOPCAT). These tools are all included in the NVOSS software distribution and you should experiment with these.

Command-line VOTable Tools

In developing your own applications or services you may or may not need to ever actually parse or create a VOTable yourself. In doing science with VO data and your own legacy code, it may be more convenient to simply convert or manipulate a VOTable into some form that is easier to use. An excellent set of tools to do this is the STILTS command-line tools available from Starlink.

STILTS currently consists of the following commands:

tcopy - Table format converter

Converts tables between formats.

tpipe - Generic table pipeline processing utility

Powerful command providing row selection, sorting, column rearrangement, algebraic data manipulation, statistical calculations, metadata display, format conversion, etc.

votcopy - VOTable encoding translator

Copies VOTable data leaving the structure intact but changing the data encoding between TABLEDATA, BINARY and FITS.

votlint - VOTable validity checker

Checks whether a VOTable document conforms to the standard, reporting on many aspects beyond simple conformance to the schema/DTD.

The NVOSS software distribution contains the jar file containing a programatic interface to each of these procedures, the command-line tools may be installed by downloading from the STILTS link above

VOTable Resources

Parsers

Applications