Main Page

From autoplot.org

Jump to: navigation, search
Start Autoplot
Start Autoplot

Contents

  1. Introduction
    1. Getting Started
    2. Summary
    3. Key Features
    4. Who's Using Autoplot
  2. Formats Read
    1. FITS
    2. NCML file
    3. SPASE file
    4. VAP file
    5. Das2Streams and QStreams
    6. ASCII table reader
    7. Binary Table Reader
    8. CDF file reader
    9. CEF File Reader
    10. netCDF file reader
    11. OPeNDAP
    12. Excel Spreadsheet
    13. TSDS
    14. Images
    15. Wav Files
    16. Jython Files
  3. Servlet, Applet, WebStart, etc.
    1. Servlet
    2. Applet
    3. WebStart
    4. Command line
      1. BASH without webstart
      2. BASH with webstart
      3. Using Nailgun (Advanced)
    5. IDL/MATLAB
    6. Jython
  4. Development
    1. Feature requests
    2. List of URIs for Testing
    3. Demo Bookmarks
    4. Report a bug
    5. Email List
    6. Development Versions
    7. Source Code
    8. Design Documents
  5. Acknowledgments

1 Introduction

1.1 Getting Started

Click the "Start Autoplot" link and then select any of the bookmarks. For help on interacting with the data, see help.

1.2 Summary

Autoplot is an interactive browser for data on the web. Give Autoplot a URL or a name of a file on your computer and it tries to create a sensible plot of the contents in the file. It was developed to allow quick and interactive browsing of data and metadata files that are often encountered on the web. In the same way that you browse content stored in .html, .jpeg, and .png files on the web, Autoplot allows you to interactively browse data stored in .cdf, .nc, and many more file formats.

Autoplot was developed under the NASA Virtual Observatories for Heliophysics program in a collaborative effort among several institutions.

1.3 Key Features

  • Reads CDF, netCDF, fits, Excel, OPeNDAP, multiple ASCII formats, and many other file formats.
  • Wildcards can be used to allow browsing data stored in multiple files.
  • Any displayed data may be saved to disk in ACSII. Plots to PNG, PDF, and SVG.
  • State may be saved and restored.
  • Reads CDF files and will display its metadata. If no variable is specified, a list of parameters is presented.
  • If given a SPASE XML file, it plots the URL in the Numerical Data Access URL section and shows the SPASE record in the metadata tab.

1.4 Who's Using Autoplot

2 Formats Read

The data source type is determined using the mime type in the HTTP headers; if the mime type is not specified, then the file extension is used. The data source type can be explicitly specified by prefixing the URI with an extension followed by a semi-colong. For example: nc:http://www.autoplot.org/data/autoplot.dat tells Autoplot to treat the data returned by this URI as netCDF-formatted.

Once the data source type (most often the file format type) is identified, the URI's parameter string is interpreted. Different URI types may have different parameter arguments. Additional formats may be added - see the tutorial for adding a data source to Autoplot: Adding_Data_Sources

Note that if a parameters string is not given, the parameters in the file or data stream can often be listed by clicking the folder icon to the right of the address bar, or by pressing CTRL-SPACE.

2.1 FITS

http://fits.gsfc.nasa.gov/fits_intro.html

2.2 NCML file

http://www.unidata.ucar.edu/software/netcdf/ncml

2.3 SPASE file

If the URL or filename corresponds to a SPASE XML file with a NumericalData element, an attempt is made to plot the first AccessURL in the AccessInformation section. (We will soon change this so that the default behavior is to look for a URL in the Granule element and to use the information in the NumericalData node to populate the metadata tree.)

2.4 VAP file

A VAP file is a VxOware Autoplot save file that includes information about style and formatting, along with (optionally) the data. Any plot state may be saved, and if desired, the actual data will be stored in the .vap file. If you want to view the data when the original data file is no longer available, select "include data in file" when saving as a .vap file. If this option is not checked, then only the information about where the data should be obtained is stored in the .vap file.

2.5 Das2Streams and QStreams

Das2Streams are self-describing, streaming format developed and used by the Plasma Wave Group at the University of Iowa. "Streaming" is the requirement that at any point along the stream, you have a all the data you need have a valid and complete stream. They are generally useful for serializing and deserializing data in das2's internal data model, and Autoplot's QDataSet model. The design goal is any dataset that can be represented in Autoplot can be serialized into a das2Stream. "QStream" refers specifically to a das2Stream with a QDataSet on it, and has the extension .qds. Legacy das2Streams have a .d2s extension.

  • mime type: application/x-das2stream
  • extension: .d2s .das2stream .qds
  • example URLs: 
http://autoplot.org/data/proton_density.qds
http://autoplot.org/data/proton_velocity_rtn.qds
  • Supports Formatting: yes, rank 1, 2, 3 to QStream
  • parameters
    • arg_0 is the named parameter within the stream, if not specified then default parameter is used.
  • output parameters
    • type
      • binary means format to binary stream

2.6 ASCII table reader

This reads in a flat ascii file with one record per line.

  • mime type: text/plain
  • extension: .dat, .txt, .csv
  • example URLs: 
ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_1963.dat?column=field17
ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_$Y.dat?time=field0&timeFormat=$Y+$d+$H&column=field17
http://goes.ngdc.noaa.gov/data/avg/2004/A1050412.TXT?skip=23&timeFormat=$y$m$d+$H$M&column=E1&time=YYMMDD
ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_$Y.dat?column=field17&timerange=1963&timeFormat=$Y+$j+$H&time=field0&validMax=999
  • Supports Formatting: yes, rank 1 and 2.
  • parameters
    • fixedColumns an optimized parser should be used since each row of the file has a fixed column width. By default the row are split into columns using a delimiter. The value may take several forms:
      • value contains ",": specify column locations as in "0-10,20-34"
      • value is int: specify the number of columns in each row. Actual may be less.
      • value is unspecified: the first row is split to determine the column locations.
    • columnCount override the number of columns.
    • column identifies the field in each record to treat as the dependent variable. By default the columns are named field0, field1, field2, etc.
    • depend0 identifies a field as the independent variable.
    • rank2 when in the URL indicates that the dataset produced should be a rank 2 dataset, and the number of fixed columns indicates the number of elements per row. When the value contains a colon (:), this indicates a range. The range is <firstColumn>:<lastColumn exclusive> or <firstColumn>-<lastColumn inclusive>.
      • value may be empty, meaning use all columns
      • 1: means the second row and up.
      • -5: means the last five rows.
      • 20:24 means the four rows starting at the 21st row.
      • field3:field6 three columns
      • B_x_gsm-B_z_gsm three columns.
    • depend1Labels indicates the first record contains the labels for the rank 2 dataset. (<firstColumn>:<lastColumn exclusive>)
    • depend1Values indicates the first record contains the values for the rank 2 dataset (so it is displayed as a spectrogram). (<firstColumn>:<lastColumn exclusive>)
    • skip is an integer indicating the number of lines that should be skipped before processing begins.
    • fill is the number that indicates missing or fill data.
    • timeFormat specifies the time format, based on the Unix date command. "ISO8601" means the times are ISO8601 conforment, or use template with fields from [1].
    • time specifies the field that is the time record. This also sets the independent variable.
    • delim identifies the delimiter character. By default, the first record is inspected for commas, tabs and then whitespace.
    • comment prefix string that indicates records to ignore.
    • fill values to be treated as fill data.
  • Wildcards and Aggregation

For ASCII files, you may specify wildcards in the URL passed to Autoplot in order to tell it how the files are structured and how to extract time from the file. The following shows a basic example of this functionality.

Load first column of the OMNIWeb2 text files 

ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_1963.dat

You should see 24*365 = 8760 values of 1963 plotted.

In the directory, the files are all named omni2_YYYY.dat. We can specify a time range by using 

ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_%Y.dat?timerange=1963-1965

If you zoom in on the line, you will see that there are now 24*365 = 8760 values of 1963 and 24*366 = 8784 values of 1964.

Now plot the 17th column, which is listed as Bz (GSM) in OMNIWeb2 metadata file. 

ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_%Y.dat?column=field17&timerange=1963

(This feature is not implemented in the current release of Autoplot.) In order have time represented on the x-axis, you must tell autoplot which columns to use. In this example, we plot the 17th column in the file. 

ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_%Y.dat?time=field0&timeFormat=%Y %D %H&column=field17

Note all data sources can be aggregated using time format specifiers in the file part of the URL. Time format specifiers in the query part of the URL (after the ?) do not trigger aggregation.

2.7 Binary Table Reader

BinaryTableReader reads in datasets from binary files.

  • mime type:
  • extension: .bin
  • example URLs: 
file:///media/mini/data.backup/examples/bin/fromidl.bin?type=float&fieldCount=2&column=1
  • supports formatting: yes, rank 1 and 2, written out as doubles.
  • parameters
    • byteOffset int, number of bytes skip before reading the data. default is 0.
    • byteLength int, total number of bytes to read. default is the content length.
    • recLength int, the number of bytes per record.
    • recOffset int, the byte offset into the record.
    • fieldCount int, number of fields per record, when recLength is not specified. default is 1.
    • column int, field number for the dependent parameter, then recOffset is not specified.
    • type data type for the dependent parameter. double, float, long, int, short, byte, ubyte. Default is ubyte.
    • depend0Offset int, byte offset into each record for the independent parameter.
    • depend0 int, field number for the independent parameter when depend0Offset is not specified. Default is no independent parameter and the byte offset is the independent parameter.
    • depend0Type data type for the independent parameter. double, float, long, int, short, byte.
    • byteOrder data type byte order (endianness). little or big. Default is little.
    • validMin, double minimum valid value
    • validMax, double, maximum valid value

How to reverse engineer a binary file with the binary reader:

  • plot the data using the default 1-byte data.
  • Look for a repeating pattern. This implies a data type. For example, if every fourth point is a peak, then try type=float. If the period is 15 bytes long, then try to identify the data type of the first field by using recLength=15&type=float, recLength=15&type=double, etc. Once the first field's data type is appearent, it's easy to see if the recLength is correct. Use recOffset to look at other bytes within each record.
  • If you're getting repeating data with large exponents, then try byteOrder=big
  • Once the byteOrder is determined, then it's much easier to identify fields.

2.8 CDF file reader

Reads in a variable from a CDF file.

  • mime type: application/x-cdf-file (Note some servers advertise a content type of application/x-netcdf.)
  • extension: .cdf
  • parameters
    • The parameter is the name of the cdf variable.
    • If not specified, a list of possible variables is given.
  • Wildcards and aggregation

In the following, we tell autoplot that the file name has a four-digit year, a two-digit month, and a two-digit day. Then we ask it to plot data on the day 20000109 using the time wildcards described in [2].

The first part of the url is 

http://cdaweb.gsfc.nasa.gov/istp_public/data/polar/hyd_h0/

the file part of the url is 

2000/po_h0_hyd_%Y%m%d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20000101

to specify more than one day, use 

2000/po_h0_hyd_%Y%m%d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20000101 - 20000103 (end is not inclusive)

to allow access to data that crosses a year boundary, use 

%Y/po_h0_hyd_%Y%m%d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20001231 through 20010101

2.9 CEF File Reader

Reads data in CEF for Cluster.

2.10 netCDF file reader

Reads in a variable from a netCDF file.

  • mime type: application/x-netcdf
  • extension: .nc
  • example URL (described on the unidata web page html): 
http://www.unidata.ucar.edu/software/netcdf/examples/tos_O1_2001-2002.nc?tos
  • parameters
    • The parameter is the name of the netcdf variable.
    • If not specified, a list of possible variables is given.

2.11 OPeNDAP

Reads in data from an OpenDAP server. When data files are served from an OPeNDAP server, subsets of a variable in a file may be requested.

2.12 Excel Spreadsheet

Data is read from a Microsoft Excel spreadsheet using Jakarta POI.

  • mime type: application/vnd.ms-excel
  • extension: .xls
  • example URL: http://autoplot.org/data/autoplot.xls?column=A
  • supports formatting: yes, rank 1 and 2
  • parameters
    • column is the name of the column (e.g. "C"), and in brackets the starting and ending rows.
    • depend0 is the name of the column identifying the X values for each value.
    • firstRow is the first row to read (1 is the first row)
    • sheet is the name of the sheet within the workbook to read.

2.13 TSDS

Data is read from TSDS server.

  • mime type:
  • extension: .tsds (use vap+tsds:http://...)
  • dependencies
    • BinaryDataSource

2.14 Images

Introduced for fun, this data source reads images using java's ImageIO library.

  • mime type:
  • extension: .gif .jpg .png
  • parameters
    • channel specifies how components should be combined to produce the rank 2 dataset.
      • If channel is not specified, then a rank 3 dataset with original image channels results.
      • Example channels include:
      • red, green, blue, alpha
      • hue, saturation, value (brightness)
      • greyscale is faster to calculate than value

2.15 Wav Files

The result of a tutorial on writing data source plugins, this plots the waveform within wav files.

  • extension: .wav

Example URL:

2.16 Jython Files

Jython scripts can be used to create new datasets by combining other datasets.

Example Script (/home/user/script.jy): 

 amplitude= params['amp']
 result= amp * sin( linspace( 0, PI*2, 100 ) )

This is called like so: 

 /home/user/script.jy?amp=2

A list of commands available to the scripts: jython_data_source_commands

3 Servlet, Applet, WebStart, etc.

3.1 Servlet

Documentation for using the servlet is in the servlet_guide

Autoplot can be run in a server mode in which case the user specifies a URL in a web browser such as 

  http://autoplot.org/AutoplotServlet?url=http://autoplot.org/demos/autoplot.cdf?B_GSM

and a PNG image is returned.

3.2 Applet

Documentation for using the applet is in the applet_guide

3.3 WebStart

Documentation for starting Autoplot with WebStart is in the webstart_guide

3.4 Command line

3.4.1 BASH without webstart

This does not work at present because we do not bundle a versioned zip file 

#!/bin/sh
 
wget http://www.cottagesystems.com/virbo/apps/autoplot/autoplot.1.0.0.zip
unzip autoplot.1.0.0.zip
 
JAVA=/opt/jdk1.5.0_10/bin/java
 
CP=`ls lib/* | xargs | tr ' ' :`:VirboAutoplot.jar
 
echo $JAVA -cp $CP org.virbo.autoplot.AutoPlotUI
$JAVA -cp $CP org.virbo.autoplot.AutoPlotUI $1

3.4.2 BASH with webstart

Call autoplot from the command line. javaws and perl must be in your path. 

!/bin/bash
# Usage: autoplot.sh
#        autoplot.sh URL
#        autoplot.sh FILE
#
# FILE may be an absolute or relative path, e.g.,
# /tmp/autoplot.cdf or file:///autoplot.cdf, or ./test/autoplot.cdf
#
# Downloads autoplot.org/autoplot.jnlp, modifies the input argument in
# autoplot.jnlp to be URL or FILE, and calls javaws on the modified autoplot.jnlp
#
# TODO:
#   check $JAVA_HOME 
 
if [ -z "$(which javaws 2>/dev/null)" ]; then
    echo "autoplot: error: Java Web Start must be installed and on your PATH to run Autoplot."
    exit 1
fi
if [ -z "$(which perl 2>/dev/null)" ]; then
    echo "autoplot: error: Perl must be installed to run this script."
    exit 1
fi
if [ -z "$(which wget 2>/dev/null)" ]; then
    echo "autoplot: error: wget must be installed to run this script."
    exit 1
fi
 
#wget -O - "http://www.cottagesystems.com/virbo/apps/autoplotDev/autoplot.jnlp" > /tmp/autoplot.jnlp
wget -O - "http://www.cottagesystems.com/virbo/apps/autoplot/autoplot.jnlp" > /tmp/autoplot.jnlp
#wget -O - "http://autoplot.org/autoplot.jnlp" > /tmp/autoplot.jnlp
 
#myfp=`which $0`
#mydir=`dirname $myfp`
mydir=`pwd`
infile=$1
 
if [ "http://" != "${infile:0:7}" ]; then
    # If leading ./, remove it
    if [ "./" == "${infile:0:2}" ]; then
        infile=${infile:2:${#infile}}
    fi
    # If no leading /, add absolute path
    if [ "/" == "${mydir:0:1}" ]; then
        infile="$mydir/$infile"
    fi
    # If no leading file:/, add it
    if [ "file:/" != "${infile:0:6}" ]; then
        infile="file:/$infile"
    fi
fi
 
echo "autoplot: Input file is $infile"
 
perl -pi -e "s#<!--<argument>.*</argument>-->#<argument>$infile</argument>#;" /tmp/autoplot.jnlp
# Omit the href in the jnlp element so that webstart does not load the rest of the jnlp from the web.
perl -pi -e "s/href=\"autoplot.jnlp\">/>/;" /tmp/autoplot.jnlp
 
javaws /tmp/autoplot.jnlp

3.4.3 Using Nailgun (Advanced)

If would would like to call autplot repeatedly from the command line as shown below, you may use Nailgun to avoid the overhead of starting a JVM each time autoplot is called. This program may be used to compare the speed of nailgun for a trivial "hello world" application writtenin java and c: http://aurora.gmu.edu/svn/java/nailgun

This zip package contains an example of calling autoplot from the command line using nailgun. It has only been tested on a few 32-bit machines. On a 64-bit Ubuntu 7.10 machine, I get the error 

Exception in thread "AWT-EventQueue-0" java.lang.AbstractMethodError:  
org.apache.xerces.dom.DocumentImpl.getXmlStandalone()Z
       at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.setDocumentInfo(DOM2TO.java:373)
       at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:127)
       at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:94)

3.5 IDL/MATLAB

We are developing code that uses Autoplot to pull data into IDL and MATLAB. Our notes on the development is here.

Some older scripts show how to pass data to Autoplot from IDL and MATLAB through a pipe: http://vxoware.svn.sourceforge.net/viewvc/vxoware/autoplot/trunk/VirboAutoplot/src/external/ Note that these are proof-of-concept scripts that require knowledge of undocumented internals of Autoplot.


3.6 Jython

Autoplot uses Jython (Python in Java) for scripting data sources and the application itself. The JythonDataSource introduces URLs like "http://www.autoplot.org/data/imageDiff.jy", and it evaluates a script (local or remote) then plots the variable "result". The script looks like this: 

  ds1= getDataSet( 'http://www.autoplot.org/data/image/Capture_00158.jpg?channel=greyscale' )
  ds2= getDataSet( 'http://www.autoplot.org/data/image/Capture_00159.jpg?channel=greyscale' )
  result= abs( ds2- ds1 )

This script shows the differences between two greyscale images.

Additionally, the scripting engine is available to control the application. When the feature "server" is enabled, a socket is opened and commands like "writeToPng('foo.png')" can be used to control the application. this javadoc describes the commands available in this context. Examples are available here: https://vxoware.svn.sourceforge.net/svnroot/vxoware/autoplot/trunk/JythonDataSource/src. More information on QDataSet http://www.das2.org/wiki/index.php/Das2_DataSet_Abstraction_3.0 and http://autoplot.org/jython_data_source_commands

4 Development

4.1 Feature requests

Autoplot was initially developed as part of a NASA VxO grant for the ViRBO project. Development will continue and we welcome requests for new features. Submit requests to http://sourceforge.net/projects/vxoware/autoplot or send an email to the list given at http://groups.google.com/group/autoplot.

4.2 List of URIs for Testing

Here is a List Of URIs for testing. These include links that demonstrate that things work, and may quickly demonstrate a sources that don't work. Feel free to add to the list, adding a comment if it doesn't work properly.

4.3 Demo Bookmarks

Here is a set of bookmarks that demonstrate Autoplot's capabilities. These are the demo bookmarks that are loaded when Autoplot is first run. They are numbered sequentially, and these numbers should not change so each demo has an identity.

4.4 Report a bug

The preferred method of reporting a bug is through the sourceforge bug tracker system. The feature "Log Console" records internal logging messages and catches stdout and stderr. This information is useful for debugging, and the console's Save button saves these messages to a file.

4.5 Email List

For discussion about Autoplot, send email to autoplot@googlegroups.com. You may browse the email archive at http://groups.google.com/group/autoplot

4.6 Development Versions

  • Develop versions are released early and often to deliver bugfixes and new features, generate feedback, and to engage the community without the costly rigor of a production release.
  • Development release list
  • Change Log describes the changes in each version.
  • List Of URLs for testing

4.7 Source Code

4.8 Design Documents

The Design Documents area contains documents planning and describing Autoplot's design.

5 Acknowledgments

Autoplot was developed with support or code contributions from ViRBO, VMO, RBSP-ECT, and the Radio and Plasma Wave Group at The University of Iowa.

Retrieved from "http://autoplot.org/Main_Page"
Personal tools