developer
From autoplot.org
Purpose: Links for miscellaneous developer pages.
1. Development Versions
The development versions are released "early and often" to accelerate maturing of code. They are located here: Autoplot_Change_Log
- http://autoplot.org/jnlp/latest/autoplot.jar is the "production version" which is done roughly monthly, has had extensive testing, and passes all Jenkins tests. If a new problem is discovered by someone using this, I stop what I'm doing until this is fixed.
- http://autoplot.org/jnlp/devel/autoplot.jar is the "development version" which is made roughly weekly, or if a change is requested by someone. Typically all the Jenkins tests are passing, but new problems may be encountered and testing is not as thorough. This is the version that I typically use as I work on campus.
- https://ci-pw.physics.uiowa.edu/job/autoplot-release/lastSuccessfulBuild/artifact/autoplot/Autoplot/dist/autoplot.jar is the automatic "nightly" build (often more often) which is used to get a new feature out for immediate use or fix a problem, but has typically no human testing. Jenkins tests are run against this, but a test can fail and this release will be available regardless.
The Jenkins interface is at https://ci-pw.physics.uiowa.edu/job/autoplot-release/
2. Use methods
Autoplot is typically run as a "thick client" on the scientist's workstation. The ~25MB application is downloaded and run on top of the Java platform. This is not the only way to use the application, however, and this section covers other modes.
2.1. Servlet
See servlet_guide. Autoplot can be run in a server mode in which case the user specifies a URL to file in a web browser and a PNG image is returned. Notes for developers of the servlet are here: developer.servlet
2.2. WebStart
See webstart_guide.
2.3. Jython
Autoplot uses Jython (Python in Java) for scripting data sources and the application itself. Selecting Options->Enable Feature->Script Panel will reveal a tab named "script". See developer.scripting.
2.4. IDL/MATLAB
See developer.idlMatlab.
2.5. Command line
2.5.1. WebStart
To start Autoplot from the command line, use
javaws http://autoplot.org/autoplot.jnlp
To start Autoplot with an initial URI in the address bar, use the CGI script on Autoplot.org which resolves to a jnlp containing the URI:
javaws http://autoplot.org/autoplot.jnlp?http://autoplot.org/data/autoplot.cdf?Magnitude
Note, however, that as of 2018 it looks like Webstart is no longer going to be supported with Java 11 and up. Autoplot currently supports Java 7 and will soon require Java 8, but webstart support will be removed at some time in the next five years.
2.5.2. Non-Webstart
Note: documentation needs URLs for non-64-bit libcdfNativeLibrary binary files. Note on note: in the current version, vap+cdfj will resolve vap+cdf if the native libraries aren't available.
wget -O /tmp/ap/autoplot.jar http://autoplot.org/jnlp/latest/autoplot.jar java -jar /tmp/ap/autoplot.jar # see that Autoplot works
2.5.2.1. Command-Line Arguments
You can launch with command line arguments as well, like so:
java -cp autoplot.jar org.autoplot.AutoplotUI http://autoplot.org/data/autoplot.cdf?Magnitude java -cp autoplot.jar org.autoplot.AutoplotUI --help
2.5.2.2. Notes on data server
The following documentation refers to the application used to serve data. This needs to be cleaned up.
To see command line options, type
java -Djava.awt.headless=true -cp autoplot.jar org.autoplot.AutoplotServer
2.5.2.3. Linux
On Linux systems the default install of Java is not complete. On a Debian-based system, use
apt-get install openjdk-8-jre
and call java using
/usr/lib/jvm/java-8-openjdk/bin/java
wget "http://autoplot.org/jnlp/latest/autoplot.jar" java -Djava.awt.headless=true -cp autoplot.jar org.autoplot.AutoplotServer -u "http://autoplot.org/data/autoplot.cdf?BGSM" -f png -o BGSM.png
2.5.2.4. Mac
curl http://autoplot.org/jnlp/latest/autoplot.jar > autoplot.jar # Test operation: java -Djava.awt.headless=true -cp autoplot.jar org.autoplot.AutoplotServer -u "http://autoplot.org/data/autoplot.cdf?BGSM" -f png -o BGSM.png
will create Image:BGSM.png
3. Development
3.1. Feature requests
Autoplot was initially developed as part of a NASA VxO grant for the ViRBO project. Development will continue and we welcome requests for new features. Submit requests to http://sourceforge.net/projects/autoplot or send an email to the list given at http://groups.google.com/group/autoplot.
3.2. Report a bug
The preferred method of reporting a bug is through the sourceforge bug tracker system. The feature "Log Console" records internal logging messages and catches stdout and stderr. This information is useful for debugging, and the console's Save button saves these messages to a file.
3.3. Automated Testing
Autoplot uses Jenkins for managing automated testing and reporting.
- Once each night and shortly after any commit of source code, hundreds of tests are performed to detect if a change has been made that breaks some aspect of the system. These are currently broken up into roughly 50 functional areas in these the tests here: https://papco.org/jenkins/. These make static images of vap products, test layout, test the IDL/Matlab interface, do PNGWalk generation, etc. This is not a thorough test, but it often catches errors that before would have been reported in a desperate email the week before a meeting.
- https://ci-pw.physics.uiowa.edu/ also does FindBugs analysis that analyses the code for suspicious and oddly-worded code, and has identified a number of mistakes as well.
- https://ci-pw.physics.uiowa.edu/job/autoplot-release/ provides a convenient means for making one-of releases and greatly simplifies the release process.
3.4. Manual Testing
We have a script for human-driven testing here: developer.tests. These are procedures for tests that don't translate into automatic tests, such as GUI operations and multiple-platform testing.
3.5. Email List
For discussion about Autoplot, send email to autoplot@googlegroups.com. You may browse the email archive at http://groups.google.com/group/autoplot
3.6. Presentations
3.7. Source Code
- The source code for Autoplot is located at http://autoplot.svn.sourceforge.net/viewvc/autoplot/autoplot/ (GPL license)
- Instructions for building are located here: Build from Source (with notes for Eclipse, NetBeans, and Ant)
- svn at https://autoplot.svn.sourceforge.net/svnroot/autoplot/autoplot/ (and add "trunk" for the main trunk, or you'll get all the tagged versions too.)
- The source code for das2 library is located at http://das2.org. (GPL license)
- Details about the internal workings of das2 may be found on their wiki
- Details about the QDataSet data model may be found at QDataSet
3.8. List of URIs for Testing
Here is a List Of URIs for testing. These include links that demonstrate that things work, and may quickly demonstrate sources that don't work. Feel free to add to the list, adding a comment if it doesn't work properly.
3.9. Notes on Debugging
Here is a place for thoughts on techniques for debugging, for the senior developer's reference, and for those who would like to get into the code.
4. Who's Using Autoplot
- ViRBO to view data files and data streamed from its time series server
- VMO to view granules of data using webstart mode
- RBSP Emfisis instrument team to look at data packets coming from GSEOS, and eventually science analysis
- Juno Wave instrument team look at level zero data, science analysis
- Scientists at U. Iowa working on Cassini use Autoplot to read data into Matlab for analysis
- RBSP ECT instrument team as a part quality control workflow, and eventually science analysis
- Space Physics Data Facility to view data using autoplot in webstart mode
- Radio and Plasma Wave Group at The University of Iowa For analysis, publication graphics, and custom applications for verifying and working with data
- The ACE Science Center to generate server-side graphics
- Center for Space Physics at Boston University for browsing CAMMICE/MICS data
- Virtual Model Repository to generate server-side graphics using command-line server
- APL is using Autoplot internals to develop applications
- EMWRAP at U. Iowa is using Autoplot for display and testing of science products.
5. Design Documents
5.1. Misc.
- developer.idlMatlab IDL/MATLAB interface
- developer.novice discussion of a basic/expert user mode.
- See Special:Allpages for a list of all pages on this wiki.
- QDataSet beginnings of a QDataSet tutorial and QDataSet technical description. QDataSet is the internal data model.
- developer.scripting Help with scripting
- developer.guiConventions GUI Conventions
- developer.URI_syntax Describes Autoplot data set URI construction, rules, and conventions.
- developer.URI_completion Description of the completion engine used to create valid URIs, to assist humans and perhaps allow for data mining...
- developer.Title_and_Label_Templates proposal for how titles and labels can use templates to insert metadata.
- developer.inlineData proposal for URIs that contain data for quick plotting and tags.
- developer.axis_auto_property proposal for adding "auto" property to axis so autoranging is finely controlled.
- developer.DataSource_API_Change proposed for DataSources that would allow for database and other non-file-based DataSources.
- developer.panel_rank_reduction spec for how panels can reduce data that's handed to them, and implementation notes.
- developer.userDirectory layout of the user data directory. Right now this is just data, but I want to put other things here too.
- developer.bindingTypes initial thoughts about the need for more binding types.
- developer.helpsets conventions for developing and maintaining help documents.
- developer.datasourceURI.extensions propose new features for data sources
- developer.dataset.filters work out spec for user-defined filters for QDataSet, Autoplot and TSDS.
- developer.python.indexing document the logic for PyQDataSet.__getItem__ index parameter.
- developer.addingHelpFiles adding help documents
- developer.respondingToEvents need to establish conventions for how to respond to events (valueIsAdjusting, etc).
- developer.multiDataSetConventions thoughts about conventions for data readers to read in Z(X,Y) from any data source.
- developer.bugSequences Sequences which have been known to demonstrate bugs. These are probably good to check periodically.
- developer.autolayout New version of the autolayout that removes gaps and overlaps between panels.
- developer.logging clean-up of the under-used and inconsistent logging system.
5.2. Autoplot 2012b
Autoplot 2012a has had a feature set that hasn't required a new vap version. This is in part because I was being lazy, because I can't guarantee that new versions are pushed out. It's final version was 1.07. There have been a number of feature requests that affect the vap, and I should do these all at once.
5.2.1. URIs for Renderers
For a while it's been clear that a more generic way of controlling plot rendering is needed. Currently, properties are added to the vap, readers and writers are made to handle those properties, and bindings made to implement the properties in das2. This works fine when a small set of properties is controlled, but as the number of degrees freedom increases, so must the complexity. Take for example a contour plot. This is clearly going to need a string to control contour positions. Then take the request to have separate fill-to-reference lines for above and below (grey if below 0.0V, red if above 5.0V). Last consider that we want to allow custom renderers that need some generic control. For all the reasons URIs have been effective for loading data, they will be effective for painting data. Note too, that ideally users would never see these, they are simply there to provide flexible control that allows Autoplot to delegate responsibility for implementing graphics to third parties.
So for example:
rend+contour:levels=10,20,30,40&fillBelow=lightGrey,grey,darkGrey,red rend+series:fillAboveColor=red&fillAboveReference=5.0V&fillBelowColor=grey&fillBelowReference=0.0V
I'd like to get this in place for people that want to play with it, but it will be limited until there are GUIs, etc, and that's a big job. So I'll be adding things to the style node as well for now.
5.2.2. Axis Position / Overlay Plots, not just plotElements
We will support two plots being drawn on top of one another, so that two Y axis can control plotElements. das2 already supports this, but it is not saved in the vap.
5.2.3. Fill Above UpperReference, Fill Below LowerReference
To implement warning levels such as grey if below 0.0V, red if above 5.0V. The legacy fill-to-reference will still be supported as a third property that is drawn first.
5.2.4. Slicer Type
Some time I just need to put this squirrelly feature in, and see what problems it creates
5.2.5. Annotation Objects
To support titles and plot annotations, a set of annotations for decorating a plot. These would include:
- canvas title (text)
- callout for features (arrows, lines, text)
5.2.6. Colors in Vap Files
I just noticed that blue is value="#ff" in the vap file. ColorSerializeDelegate should provide the names of common colors.
5.2.7. org.das2 in Vap Files
Also, there are things like
<property name="colortable" type="enum:org.das2.graph.DasColorBar$Type" value="color_wedge"/>
We should get rid org.das2 references from the vap file.
5.2.8. Override Units
Reiner would like to be able to override units in the Data tab, the same way you can override valid range and fill. This would help out ascii files, which have limited support for units.
5.2.9. Physical Units system
It would be nice to finally implement a physical units system, so that cm divided by sec would be cm/s... We've talked about doing this for years.
5.2.10. allow XSLT modifiers to create DOMs
Bob would like to use XLST to specify DOMs, so you could say foo.xslt&uri1=...&uri2=...&uri3=...
5.2.11. Declare TimeSeriesBrowse connections
Right now these are calculated automatically at runtime. They should probably be explicit in the file. For example, Reiner has products that would have no time axes, but need to run on pngwalks.
5.3. Autoplot 2011
It would be useful to identify the feature set of the Autoplot2011 branch.
5.3.1. TimeSeriesBrowse
- TimeSeriesBrowse will reconnect itself to application timerange if the result is not a timeseries.
- (TODO: this doesn't work when you reload the file.)
- Automatic aggregation detection. Plotting "20100101.dat" will look for "$Y$m$d" and use the aggregation automatically. [NOTYET]
5.3.2. Data Sources
- pure-java cdf reader handles ".cdf" (NOT YET)
- inline specification of filters, such as
- vap+bin:file:///tmp/foo.wav?column=1&recLength=3|histogram (NOT YET)
- introduce the general way to specify depend datasets, such as: (NO, 2012)
- vap+bin:file:///tmp/foo.dat?recLength=45&depend0.column=0&column=2 , instead of
- vap+bin:file:///tmp/foo.dat?recLength=45&depend0offset=0&column=2
- introduce other standard parameters for easily modifying the dataset, like title, label, rendertype, etc.
- csv is handled by 3rd party and correct csv reader. (YES, DONE)
5.3.3. Graphics
- legend position, and legend outside plot (DONE, though not yet in vap)
- plots.xaxis.ticklen (DONE)
- improve cadence logic, there's a "corner case" of just four corners that I don't handle, where the histogram resolution is too fine so the non-zero peaks are spaced regularly.
- new annotations in labels
- %{VERSION} from QDataSet
- %{CONTEXT} shows any context, e.g. slice position, associated with a dataset.
- %{COMPONENT} shows slice positions, components, or additional processing.
- Time Axis annotations (ephemeris, TCA. It goes by a number of names).
- RGB image renderer (DONE)
- Events bar renderer (DONE)
- rank 0 renderer that allows data to be reported. (DONE, digital supports this)
- digital renderer used for enumeration data.
- Improve list of available renderers to only those that handle dataset scheme, checkmark next to active render type.
5.3.4. GUI
- Optional x-axis (time) range controller on canvas tab. (done)
- Option to use $Y-$j day of year for ticks instead of $Y-$m-$d (done)
- Links over to help documentation (done)
- Flip y axis (Garcia) (NOTYET)
- Labels option to only face two directions, many publishers require this. (Bill K.) (NOTYET)
5.3.5. Layout
- Automatic layout inserts extra spaces for multiline plots and axes. (Need to specify how this is going to work.) developer.autolayout
- Handle stack plots that don't have x ticks.
- plot above/plot below, insert plot below, insert plot above. (done)
- convert current plot to M by N
- easy way reduce/enlarge height
5.3.6. DOM version 1.8
Everyone has figured out there there are lots of accessible Das2 features that don't make it into the DOM. Bob has requested that "everything" be in the DOM, and to the extent that this is possible, I'll do this. These features include:
- legend position (NOTYET)
- plots.xaxis.ticklen (NOTYET)
- overriding cadence (NOTYET)
- Flip y axis (Garcia)
- Labels option to only face two directions (THIS MISSED the 2011 branch, but is a trivial change. I'll probably introduce it for 2011a_8 as a user preference.
5.3.7. User Options
- basic/expert mode hides large sections of the DOM to help to get started. [DONE]
- basic mode will:
- hide the controller nodes
- show [?] in section labels to link to documentation
- hide the "data" "script" "console" tabs
- show "add plot from...cdaweb..." in basic mode
- expert mode will:
- show the "data" "script" "console" tabs
- show [?] in section labels to link to documentation
- see developer.novice
5.4. Autoplot 2010
5.4.1. Major Limitations of Current Autoplot
A couple of easily described limitations of autoplot present a nice use case and hopefully help to specify the beefy version.
- Only one dataset can be viewed at a time.
- no stack plots
- no overplots
- no correlation plots
- Some properties can be changed, but changes are not saved.
- row, column layout
- contour numbers
The more configurable the application becomes, the more pressing the need for a beefy system for storing application state. This has been a lingering problem with das2 for years.
5.4.2. Scalability from Delegation
The way complex state is achieved is to delete object state storage out to the objects. The application knows how to store itself by delegating to the objects that implement it.
5.4.3. DOM Tree for Autoplot
Upper-case nodes represent nodes with children, lower case nodes are leaf nodes that can be set. 'Italics' are simply comments about the node. An example node may exist below a parent to show an example child, and an ID will show how it might be accessed.
Style is a node below many nodes, which contain properties that affect the application view but not the model. DOMs may be deserialized while ignoring these, leaving user preferences in place. Further "Style Sheets" may be applied to a DOM with affecting the interpretation the application state.
EmSpec is a string like 50%+5em, which are to be interpreted relative to a column or row. 20% means 20% of the way from min to max. 5em means 5 ems, the height of the font. 20px or 20pt means 20 pixels.
Propose/Specify DOM for autoplot application.
5.4.4. First Implementation
Here's how the DOM is shaping up:
Some differences to note:
- less hierarchical than first conceived. It's easier to manage when the model's structure doesn't change, so have plots contain panels by reference rather than object hierarchy. panel.plotId contains the plot that contains it. This also means moving a panel is just resetting the plotId.
- many controllers. Back in the Spring I expected there would be one massive controller, but it works nicely to have a controller for each node.
- controllers in the tree. Right now, the controllers are accessible in the tree with each node's getController method. This may change since it pollutes the model with the controller. But it does make it very easy to find things.
- application.options rather than application.style. This is closer to the legacy structure and this may change as the idea of "style sheets" is introduced. Options are per-user inter-session persistent.
5.4.5. Controller for Autoplot
The controller contains methods for controlling the DOM.
5.4.5.1. Methods
- bind( 'Canvas[0].Plot[0].Xaxis.range', property( DOM.getElementById('annotation_1'), 'text' )
- property( Object o, String property )
- insertRow( ) adds below the current focus
- insertColumn( )
- focusElement( )
- addDataSource( DataSource )
5.4.6. Use Cases
- plot with two datasets overlayed.
- correlation between two datasets.
- add ContoursRenderer
- add TimeRangeSelector
- add page for collecting digitize times
- run daily plots for year
- style sheets demo similar to CSS Zen Garden
5.4.7. Layout Specification
"Rows" and "columns" are used to control the position of plots. Autoplot uses strings "5em,100%-10em" to control the position of these objects. The comma delineates min and max specifications, which correspond to top,left and bottom, right for rows and columns respectively. The text "100%-10em" means 100% of the way from the way left (min) to right (max), and the -10em means come back 10 em heights.
6. Autoplot Objects
6.1. DOM Controllers
The DOM Controllers are objects that manage nodes of the DOM. For example, when the slice index of a DataSourceFilter node is changed, the controller slices the original dataset to create a new dataset, and fires off an event notification of the new dataset.
DataSourceController: Converts the DataSourceFilter node URI string into a dataset. (Filter comes from that this once would slice as well, but that function may be removed. We may allow one node to simply filter another node.)
PlotElementController: A PlotElement has references to a DataSourceFilter,a Plot node, and render type string identifying the rendering method. This takes the QDataSet produced by the DataSourceController and renders it onto the plot. Note that this may also filter the data. When vectors are displayed, several PlotElements are added for each component. Also, each PlotElement contains default plot ranges and labels.
PlotController: The Plot node has range and label settings. This controller will detect when autoranging should occur and combines the plot ranges into a default plot range.
TimeSeriesBrowseController: When a DataSource indicates it has the TimeSeriesBrowse capability, this controller is created to manage loading new data when new ranges are displayed.
ApplicationController: Its job is to simply add and remove listeners as the dom is manipulated, to keep track of the application's history, and to implement load and save.
Note a number of things are handled as part of the Das2 graphics library or Java:
- Updates to the plot when axes are moved or labels changed via property editor.
- Connectors between plots are painted and managed by das2.
- Bindings are simply Java Beans Bindings between property nodes.
6.2. DOM Nodes
6.2.1. syncTo
syncTo method should set all the properties of a node to those of another node. This should include the id. This should include references to id's.
6.2.2. diffs
diffs returns an array of diff descriptor objects that can do and undo the diff. The returned diffs should include the id. The returned diffs should include changes in references to other
6.2.3. copy
copy is a deep copy of the node, including id and references. However, this is often implemented with clone() it's extremely important that the controller not be copied. Most copy implementations will explicitly set the controller to null.