developer

From autoplot.org

Jump to: navigation, search

Purpose: Links for miscellaneous developer pages.

Contents

  1. Development Versions
  2. Use methods
    1. Servlet
    2. Applet
    3. WebStart
    4. Jython
    5. IDL/MATLAB
    6. Command line
      1. WebStart
      2. Non-Webstart
        1. Command-Line Arguments
        2. Notes on data server
        3. Linux
        4. Mac
      3. Non-WebStart + Nailgun (faster)
  3. Development
    1. Development Versions
    2. Feature requests
    3. Report a bug
    4. Automated Testing
    5. Manual Testing
    6. Email List
    7. Presentations
    8. Source Code
    9. List of URIs for Testing
    10. Notes on Debugging
  4. Who's Using Autoplot
  5. Design Documents
    1. Misc.
    2. Autoplot 2012b
      1. URIs for Renderers
      2. Axis Position / Overlay Plots, not just plotElements
      3. Fill Above UpperReference, Fill Below LowerReference
      4. Slicer Type
      5. Annotation Objects
      6. Colors in Vap Files
      7. org.das2 in Vap Files
      8. Override Units
      9. Physical Units system
      10. allow XSLT modifiers to create DOMs
      11. Declare TimeSeriesBrowse connections
    3. Autoplot 2011
      1. TimeSeriesBrowse
      2. Data Sources
      3. Graphics
      4. GUI
      5. Layout
      6. DOM version 1.8
      7. User Options
    4. Autoplot 2010
      1. Major Limitations of Current Autoplot
      2. Scalability from Delegation
      3. DOM Tree for Autoplot
      4. First Implementation
      5. Controller for Autoplot
        1. Methods
      6. Use Cases
      7. Layout Specification
  6. Autoplot Objects
    1. DOM Controllers
    2. DOM Nodes
      1. syncTo
      2. diffs
      3. copy

1. Development Versions

The development versions are released "early and often" to accelerate maturing of code. They are located here: Autoplot_Change_Log

2. Use methods

Autoplot is typically run as a "thick client" on the scientist's workstation. The ~25MB application is downloaded and run on top of the Java platform. This is not the only way to use the application, however, and this section covers other modes.

2.1. Servlet

See servlet_guide. Autoplot can be run in a server mode in which case the user specifies a URL to file in a web browser and a PNG image is returned. Notes for developers of the servlet are here: developer.servlet

2.2. Applet

See applet_guide.

2.3. WebStart

See webstart_guide.

2.4. Jython

Autoplot uses Jython (Python in Java) for scripting data sources and the application itself. Selecting Options->Enable Feature->Script Panel will reveal a tab named "script". See developer.scripting.

2.5. IDL/MATLAB

See developer.idlMatlab.

2.6. Command line

2.6.1. WebStart

To start Autoplot from the command line, use

javaws http://autoplot.org/autoplot.jnlp

To start Autoplot with an initial URI in the address bar, use the following script.

!/bin/bash
# Usage: autoplot.sh
#        autoplot.sh URL
#        autoplot.sh FILE
#
# FILE may be an absolute or relative path, e.g.,
# /tmp/autoplot.cdf or file:///autoplot.cdf, or ./test/autoplot.cdf
#
# Downloads autoplot.org/autoplot.jnlp, modifies the input argument in
# autoplot.jnlp to be URL or FILE, and calls javaws on the modified autoplot.jnlp
#
# TODO:
#   check $JAVA_HOME 
if [ -z "$(which javaws 2>/dev/null)" ]; then
    echo "autoplot: error: Java Web Start must be installed and on your PATH to run Autoplot."
    exit 1
fi
if [ -z "$(which wget 2>/dev/null)" ]; then
    echo "autoplot: error: wget must be installed to run this script."
    exit 1
fi
 
mydir=`pwd`
infile=$1
 
if [ "http://" != "${infile:0:7}" ]; then
    # If leading ./, remove it
    if [ "./" == "${infile:0:2}" ]; then
        infile=${infile:2:${#infile}}
    fi
    # If no leading /, add absolute path
    if [ "/" == "${mydir:0:1}" ]; then
        infile="$mydir/$infile"
    fi
    # If no leading file://, add it
    if [ "file://" != "${infile:0:7}" ]; then
        infile="file://$infile"
    fi
fi
 
# See http://autoplot.org/webstart_guide for the syntax of jnlp.cgi
 
cmd="wget -q -O - \"http://autoplot.org/autoplot.jnlp?open="$infile"\" > /tmp/autoplot.jnlp"
echo $cmd
eval $cmd
# non-Sun Java WebStart is not full implementation of Sun WebStart
/usr/lib/jvm/java-6-sun/bin/javaws /tmp/autoplot.jnlp

2.6.2. Non-Webstart

Note: documentation needs URLs for non-64-bit libcdfNativeLibrary binary files. Note on note: in the current version, vap+cdfj will resolve vap+cdf if the native libraries aren't available.

wget -O /tmp/ap/autoplot.jar  http://autoplot.org/jnlp/latest/autoplot.jar
java -jar /tmp/ap/autoplot.jar  # see that Autoplot works

If you download the native libraries (.so, .dll) for your platform, you can use:

cd /tmp/ap/
wget  -O /tmp/ap/nativeCdfLibs.jar http://autoplot.org/autoplot/autoplot/cdf-jnlp-3.4.1/lib/3.4.1/linux/amd64/nativeCdfLibs.jar
unzip nativeCdfLibs.jar  # you should see libcdfNativeLibrary.so
java -D java.library.path=/tmp/ap/ -jar /tmp/ap/autoplot.jar

2.6.2.1. Command-Line Arguments

You can launch with command line arguments as well, like so:

java -cp autoplot.jar org.virbo.autoplot.AutoplotUI http://autoplot.org/data/autoplot.cdf?Magnitude
java -cp autoplot.jar org.virbo.autoplot.AutoplotUI --help

2.6.2.2. Notes on data server

The following documentation refers to the application used to serve data. This needs to be cleaned up.

To see command line options, type

java -Djava.awt.headless=true -Djava.library.path=. -cp autoplot.jar org.virbo.autoplot.AutoplotServer

2.6.2.3. Linux

On Linux systems the default install of Java is not complete. On a Debian-based system, use

apt-get install openjdk-6-jre

and call java using

/usr/lib/jvm/java-6-openjdk/bin/java
wget "http://autoplot.org/hudson/job/autoplot-release/lastSuccessfulBuild/artifact/autoplot/VirboAutoplot/dist/autoplot.jar"
 
# 64-bit Linux
wget http://autoplot.svn.sourceforge.net/viewvc/autoplot/autoplot/branches/agu2009/VirboAutoplot/libcdfNativeLibrary.so
 
java -Djava.awt.headless=true -Djava.library.path=. -cp autoplot.jar org.virbo.autoplot.AutoplotServer -u "http://autoplot.org/data/autoplot.cdf?BGSM" -f png -o BGSM.png

2.6.2.4. Mac

curl http://autoplot.org/hudson/job/autoplot-release/lastSuccessfulBuild/artifact/autoplot/VirboAutoplot/dist/autoplot.jar > autoplot.jar
 
# 64-bit mac
curl "http://sscweb.gsfc.nasa.gov/skteditor/cdf/lib/3.3.1/mac/x86_64/nativeCdfLibs.jar" > nativeCdfLibs.jar   
# 32-bit mac
#curl "http://sscweb.gsfc.nasa.gov/skteditor/cdf/lib/3.3.1/mac/x86/nativeCdfLibs.jar" > nativeCdfLibs.jar 
 
mv nativeCdfLibs.jar nativeCdfLibs.zip
unzip nativeCdfLibs.zip
 
# Test operation:
java -Djava.awt.headless=true -Djava.library.path=. -cp autoplot.jar org.virbo.autoplot.AutoplotServer -u "http://autoplot.org/data/autoplot.cdf?BGSM" -f png -o BGSM.png

will create Image:BGSM.png

2.6.3. Non-WebStart + Nailgun (faster)

If you would like to call Autoplot repeatedly from the command line, you may use Nailgun to avoid the overhead of starting a JVM each time Autoplot is called (but setting up an Autoplot server is probably a better solution).

This [1] program may be used to compare the speed of nailgun for a trivial "hello world" application written in java and c.

First follow the instructions for running Autoplot in #Basic. Then

wget "http://sourceforge.net/projects/nailgun/files/nailgun/0.7.1/nailgun-0.7.1.zip/download"
unzip nailgun-0.7.1.zip
 
java  -Djava.awt.headless=true -Djava.library.path=. -cp autoplot.jar -cp ./autoplot.jar:nailgun-0.7.1/nailgun-0.7.1.jar:. com.martiansoftware.nailgun.NGServer &
 
# Give the server some time to start
sleep 1
 
nailgun-0.7.1/ng org.virbo.autoplot.AutoplotServer -u "http://autoplot.org/data/autoplot.cdf?BGSM" -f png -o BGSM1.png 
nailgun-0.7.1/ng ng-stats
 
nailgun-0.7.1/ng org.virbo.autoplot.AutoplotServer -u "http://autoplot.org/data/autoplot.cdf?BGSM" -f png -o BGSM2.png
nailgun-0.7.1/ng ng-stats

3. Development

3.1. Development Versions

3.2. Feature requests

Autoplot was initially developed as part of a NASA VxO grant for the ViRBO project. Development will continue and we welcome requests for new features. Submit requests to http://sourceforge.net/projects/autoplot or send an email to the list given at http://groups.google.com/group/autoplot.

3.3. Report a bug

The preferred method of reporting a bug is through the sourceforge bug tracker system. The feature "Log Console" records internal logging messages and catches stdout and stderr. This information is useful for debugging, and the console's Save button saves these messages to a file.

3.4. Automated Testing

Autoplot uses Hudson (now named Jenkins) for managing automated testing and reporting.

  • Once each night and shortly after any commit of source code, hundreds of tests are performed to detect if a change has been made that breaks some aspect of the system. These are currently broken up in to roughly 40 functional areas in these the tests here: http://papco.org:8080/hudson/. These make static images of vap products, test layout, test the IDL/Matlab interface, do PNGWalk generation, etc. This is not a thorough test, but it often catches errors that before would have been reported in a desperate email the week before a meeting.
  • http://apps-pw.physics.uiowa.edu/hudson/ also does FindBugs analysis that analyses the code for suspicious and oddly-worded code, and has identified a number of mistakes as well.
  • http://autoplot.org:8080/hudson/job/autoplot-release/ provides a convenient means for making one-of releases and greatly simplifies the release process.

3.5. Manual Testing

We have a script for human-driven testing here: developer.tests. These are procedures for tests that don't translate into automatic tests, such as GUI operations and multiple-platform testing.

3.6. Email List

For discussion about Autoplot, send email to autoplot@googlegroups.com. You may browse the email archive at http://groups.google.com/group/autoplot

3.7. Presentations

  • AGU 2010 poster: ppt
  • QDataSet Poster talks about how data is represented: ppt

3.8. Source Code

3.9. List of URIs for Testing

Here is a List Of URIs for testing. These include links that demonstrate that things work, and may quickly demonstrate a sources that don't work. Feel free to add to the list, adding a comment if it doesn't work properly.

3.10. Notes on Debugging

Here is a place for thoughts on techniques for debugging, for the senior developer's reference, and for those who would like to get into the code.

4. Who's Using Autoplot

5. Design Documents

5.1. Misc.

5.2. Autoplot 2012b

Autoplot 2012a has had a feature set that hasn't required a new vap version. This is in part because I was being lazy, because I can't guarantee that new versions are pushed out. It's final version was 1.07. There have been a number of feature requests that affect the vap, and I should do these all at once.

5.2.1. URIs for Renderers

For a while it's been clear that a more generic way of controlling plot rendering is needed. Currently, properties are added to the vap, readers and writers are made to handle those properties, and bindings made to implement the properties in das2. This works fine when a small set of properties is controlled, but as the number of degrees freedom increases, so must the complexity. Take for example a contour plot. This is clearly going to need a string to control contour positions. Then take the request to have separate fill-to-reference lines for above and below (grey if below 0.0V, red if above 5.0V). Last consider that we want to allow custom renderers that need some generic control. For all the reasons URIs have been effective for loading data, they will be effective for painting data. Note too, that ideally users would never see these, they are simply there to provide flexible control that allows Autoplot to delegate responsibility for implementing graphics to third parties.

So for example:

rend+contour:levels=10,20,30,40&fillBelow=lightGrey,grey,darkGrey,red
rend+series:fillAboveColor=red&fillAboveReference=5.0V&fillBelowColor=grey&fillBelowReference=0.0V

I'd like to get this in place for people that want to play with it, but it will be limited until there are GUIs, etc, and that's a big job. So I'll be adding things to the style node as well for now.

5.2.2. Axis Position / Overlay Plots, not just plotElements

We will support two plots being drawn on top of one another, so that two Y axis can control plotElements. das2 already supports this, but it is not saved in the vap.

5.2.3. Fill Above UpperReference, Fill Below LowerReference

To implement warning levels such as grey if below 0.0V, red if above 5.0V. The legacy fill-to-reference will still be supported as a third property that is drawn first.

5.2.4. Slicer Type

Some time I just need to put this squirrelly feature in, and see what problems it creates

5.2.5. Annotation Objects

To support titles and plot annotations, a set of annotations for decorating a plot. These would include:

  • canvas title (text)
  • callout for features (arrows, lines, text)

5.2.6. Colors in Vap Files

I just noticed that blue is value="#ff" in the vap file. ColorSerializeDelegate should provide the names of common colors.

5.2.7. org.das2 in Vap Files

Also, there are things like

<property name="colortable" type="enum:org.das2.graph.DasColorBar$Type" value="color_wedge"/>

We should get rid org.das2 references from the vap file.

5.2.8. Override Units

Reiner would like to be able to override units in the Data tab, the same way you can override valid range and fill. This would help out ascii files, which have limited support for units.

5.2.9. Physical Units system

It would be nice to finally implement a physical units system, so that cm divided by sec would be cm/s... We've talked about doing this for years.

5.2.10. allow XSLT modifiers to create DOMs

Bob would like to use XLST to specify DOMs, so you could say foo.xslt&uri1=...&uri2=...&uri3=...

5.2.11. Declare TimeSeriesBrowse connections

Right now these are calculated automatically at runtime. They should probably be explicit in the file. For example, Reiner has a products that would have no time axes, but need to run on pngwalks.

5.3. Autoplot 2011

It would be useful to identify the feature set of the Autoplot2011 branch.

5.3.1. TimeSeriesBrowse

  • TimeSeriesBrowse will reconnect itself to application timerange if the result is not a timeseries.
    • (TODO: this doesn't work when you reload the file.)
  • Automatic aggregation detection. Plotting "20100101.dat" will look for "$Y$m$d" and use the aggregation automatically. [NOTYET]

5.3.2. Data Sources

  • pure-java cdf reader handles ".cdf" (NOT YET)
  • inline specification of filters, such as
    • vap+bin:file:///tmp/foo.wav?column=1&recLength=3|histogram (NOT YET)
  • introduce the general way to specify depend datasets, such as: (NO, 2012)
    • vap+bin:file:///tmp/foo.dat?recLength=45&depend0.column=0&column=2 , instead of
    • vap+bin:file:///tmp/foo.dat?recLength=45&depend0offset=0&column=2
  • introduce other standard parameters for easily modifying the dataset, like title, label, rendertype, etc.
  • csv is handled by 3rd party and correct csv reader. (YES, DONE)

5.3.3. Graphics

  • legend position, and legend outside plot (DONE, though not yet in vap)
  • plots.xaxis.ticklen (DONE)
  • improve cadence logic, there's a "corner case" of just four corners that I don't handle, where the histogram resolution is too fine so the non-zero peaks are spaced regularly.
  • new annotations in labels
    •  %{VERSION} from QDataSet
    •  %{CONTEXT} shows any context, e.g. slice position, associated with a dataset.
    •  %{COMPONENT} shows slice positions, components, or additional processing.
  • Time Axis annotations (ephemeris, TCA. It goes by a number of names).
  • RGB image renderer (DONE)
  • Events bar renderer (DONE)
  • rank 0 renderer that allows data to be reported. (DONE, digital supports this)
  • digital renderer used for enumeration data.
  • Improve list of available renderers to only those that handle dataset scheme, checkmark next to active render type.

5.3.4. GUI

  • Optional x-axis (time) range controller on canvas tab. (done)
  • Option to use $Y-$j day of year for ticks instead of $Y-$m-$d (done)
  • Links over to help documentation (done)
  • Flip y axis (Garcia) (NOTYET)
  • Labels option to only face two directions, many publishers require this. (Bill K.) (NOTYET)

5.3.5. Layout

  • Automatic layout inserts extra spaces for multiline plots and axes. (Need to specify how this is going to work.) developer.autolayout
  • Handle stack plots that don't have x ticks.
  • plot above/plot below, insert plot below, insert plot above. (done)
  • convert current plot to M by N
  • easy way reduce/enlarge height

5.3.6. DOM version 1.8

Everyone has figured out there there are lots of accessible Das2 features that don't make it into the DOM. Bob has requested that "everything" be in the DOM, and to the extent that this is possible, I'll do this. These features include:

  • legend position (NOTYET)
  • plots.xaxis.ticklen (NOTYET)
  • overriding cadence (NOTYET)
  • Flip y axis (Garcia)
  • Labels option to only face two directions (THIS MISSED the 2011 branch, but is a trivial change. I'll probably introduce it for 2011a_8 as a user preference.

5.3.7. User Options

  • basic/expert mode hides large sections of the DOM to help to get started. [DONE]
  • basic mode will:
    • hide the controller nodes
    • show [?] in section labels to link to documentation
    • hide the "data" "script" "console" tabs
    • show "add plot from...cdaweb..." in basic mode
  • expert mode will:
    • show the "data" "script" "console" tabs
    • show [?] in section labels to link to documentation
  • see developer.novice

5.4. Autoplot 2010

5.4.1. Major Limitations of Current Autoplot

A couple of easily described limitations of autoplot present a nice use case and hopefully help to specify the beefy version.

  • Only one dataset can be viewed at a time.
    • no stack plots
    • no overplots
    • no correlation plots
  • Some properties can be changed, but changes are not saved.
    • row, column layout
    • contour numbers

The more configurable the application becomes, the more pressing the need for a beefy system for storing application state. This has been a lingering problem with das2 for years.

5.4.2. Scalability from Delegation

The way complex state is achieved is to delete object state storage out to the objects. The application knows how to store itself by delegating to the objects that implement it.

5.4.3. DOM Tree for Autoplot

Upper-case nodes represent nodes with children, lower case nodes are leaf nodes that can be set. 'Italics' are simply comments about the node. An example node may exist below a parent to show an example child, and an ID will show how it might be accessed.

Style is a node below many nodes, which contain properties that affect the application view but not the model. DOMs may be deserialized while ignoring these, leaving user preferences in place. Further "Style Sheets" may be applied to a DOM with affecting the interpretation the application state.

EmSpec is a string like 50%+5em, which are to be interpreted relative to a column or row. 20% means 20% of the way from min to max. 5em means 5 ems, the height of the font. 20px or 20pt means 20 pixels.

Propose/Specify DOM for autoplot application.

5.4.4. First Implementation

Here's how the DOM is shaping up:

Some differences to note:

  • less hierarchical than first conceived. It's easier to manage when the model's structure doesn't change, so have plots contain panels by reference rather than object hierarchy. panel.plotId contains the plot that contains it. This also means moving a panel is just resetting the plotId.
  • many controllers. Back in the Spring I expected there would be one massive controller, but it works nicely to have a controller for each node.
  • controllers in the tree. Right now, the controllers are accessible in the tree with each node's getController method. This may change, since it pollutes the model with the controller. But it does make it very easy to find things.
  • application.options rather than application.style. This is closer to the legacy structure and this may change as the idea of "style sheets" is introduced. Options are per-user inter-session persistent.

5.4.5. Controller for Autoplot

The controller contains methods for controlling the DOM.

5.4.5.1. Methods

  • bind( 'Canvas[0].Plot[0].Xaxis.range', property( DOM.getElementById('annotation_1'), 'text' )
  • property( Object o, String property )
  • insertRow( ) adds below the current focus
  • insertColumn( )
  • focusElement( )
  • addDataSource( DataSource )

5.4.6. Use Cases

  • plot with two datasets overlayed.
  • correlation between two datasets.
  • add ContoursRenderer
  • add TimeRangeSelector
  • add page for collecting digitize times
  • run daily plots for year
  • style sheets demo similar to CSS Zen Garden

5.4.7. Layout Specification

"Rows" and "columns" are used to control the position of plots. Autoplot uses strings "5em,100%-10em" to control the position of these objects. The comma delineates min and max specifications, which correspond to top,left and bottom, right for rows and columns respectively. The text "100%-10em" means 100% of the way from the way left (min) to right (max), and the -10em means come back 10 em heights.

6. Autoplot Objects

6.1. DOM Controllers

The DOM Controllers are objects that manage nodes of the DOM. For example, when the slice index of a DataSourceFilter node is changed, the controller slices the original dataset to create a new dataset, and fires off an event notification of the new dataset.

DataSourceController: Converts the DataSourceFilter node URI string into a dataset. (Filter comes from that this once would slice as well, but that function may be removed. We may allow one node to simply filter another node.)

PlotElementController: A PlotElement has references to a DataSourceFilter,a Plot node, and render type string identifying the rendering method. This takes the QDataSet produced by the DataSourceController and renders it onto the plot. Note that this may also filter the data. When vectors are displayed, several PlotElements are added for each component. Also, each PlotElement contains default plot ranges and labels.

PlotController: The Plot node has range and label settings. This controller will detect when autoranging should occur and combines the plot ranges into a default plot range.

TimeSeriesBrowseController: When a DataSource indicates it has the TimeSeriesBrowse capability, this controller is created to manage loading new data when new ranges are displayed.

ApplicationController: Its job is to simply add and remove listeners as the dom is manipulated, to keep track of the application's history, and to implement load and save.

Note a number of things are handled as part of the Das2 graphics library or Java:

  • Updates to the plot when axes are moved or labels changed via property editor.
  • Connectors between plots are painted and managed by das2.
  • Bindings are simply Java Beans Bindings between property nodes.

6.2. DOM Nodes

6.2.1. syncTo

syncTo method should set all the properties of a node to those of another node. This should include the id. This should include references to id's.

6.2.2. diffs

diffs returns an array of diff descriptor objects that can do and undo the diff. The returned diffs should include the id. The returned diffs should include changes in references to other

6.2.3. copy

copy is a deep copy of the node, including id and references. However, this is often implemented with clone() it's extremely important that the controller not be copied. Most copy implementations will explicitly set the controller to null.

Personal tools