help
From autoplot.org
Contents |
1. Introduction
Autoplot is an interactive browser for data on the web. In the same way that you browse content stored in .html, .jpeg, and .png files on the web, Autoplot allows you to interactively browse data stored in .cdf, .nc, ascii, and many more data file formats.
Autoplot tries to create a sensible plot given a URL or a name of a data file on your computer. Axes and plot settings are quite interactive, so you can quickly get your data from data file to publishable graphic within minutes.
To get started, click this link Start Autoplot and then select some of the demo bookmarks found under "Bookmarks" in the menu bar.
2. Installation
Autoplot is typically run using Java Webstart, which is a technology available with any recent java installation. (When a web server sends over a .jnlp file, webstart will handle the file by downloading and running the application.) In some cases this doesn't work, and an alternate "single-jar" method can be used.
2.1. Failsafe (single-jar non-Webstart)
Windows and Mac
Download http://autoplot.org/jnlp/latest/autoplot.jar to your desktop and then double click it. Autoplot should launch if you have java installed.
Linux
Download http://autoplot.org/jnlp/latest/autoplot.jar and then java -jar autoplot.jar. (IcedTea/OpenJDK versions of Java may work, but we recommend installation of Oracle's version of Java [1])
Note this will by default have limited memory and will use the Java-based CDF reader, which has very minor limitations. To launch with more memory:
java -Xmx4G -jar autoplot.jar
will launch with 4 gigabytes of memory available. The native CDF reader can be accessed as well, and proxies set, but that is beyond the scope of this document. Please email autoplot@groups.google.com for more information about this.
2.2. Webstart
Java applications and Autoplot have the goal of being installationless, in that there's a trivial procedure to start the application. Most operating systems come with Java pre-installed and pre-configured so that Autoplot should work immediately with a launcher that is integrated with your operating system. The installation instructions below reflect specific aspects of this procedure at this time (Spring 2012). The installation instructions below assume that you have a recent version of Java installed (a version from 2005 or later). You may install Java by going to http://java.com/en and clicking "Free Java Download".
First click this link Start Autoplot and follow the instructions given below for you browser/operating system combination.
See also help#WebStart
To get more memory (4 GBytes in this case) with webstart:
http://www.autoplot.org/autoplot.jnlp?max-heap-size=4G
Note 32 bit Java environments are limited to 1GB.
2.2.1. Windows 7
2.2.1.1. IE9
Click "Open" (If you do not see the "Open" option, you need to install Java by going to http://java.com/en and clicking "Free Java Download".)
2.2.2. Mac
People are having problems using Java applications on Mac OS X due to changes in how Java applications are handled. Here are notes about webstart installs.
2.2.3. Chrome
When you see this appear on the bottom of the browser window, Select "Keep"
then click on this
To have Autoplot start in the future without requiring a click on the downloaded jnlp file, select the arrow on the right-hand side of the downloaded file and select "Always open files of this type".
2.2.4. Safari
You will see this pop-up. If Autoplot does not start Automatically, click on autoplot.jnlp.
2.2.5. Firefox
When you see this pop-up window, select OK: (If you do not see the "Java Web Start" as an option in the drop-down list, you need to install Java by going to http://java.com/en and clicking "Free Java Download".)
2.2.6. Linux
The default configuration of many modern Linux operating systems uses a non-Sun implementation of WebStart and Java (e.g., IcedTea/OpenJDK), which works poorly with Autoplot, and sometimes not at all. (Autoplot requires standard features of Java that have not yet been implemented in many non-Oracle versions of Java.)
To launch Autoplot from a web browser on Linux, follow these steps:
- Install Sun/Oracle Java (do not use IcedTea/OpenJDK)
- Find the location of
javawswith the word sun in the path, e.g.,/usr/lib/jvm/java-6-sun-1.6.0.24/bin/javaws. (This is usually the first result listed after enteringlocate "/bin/javaws" | grep sunon the command line.) - Click here.
- When prompted with an "Open With" dialog, enter the full path to
javaws(note that on Chrome you will need to click on the downloaded file to get the "Open With" dialog) - If you still have problems, see [2].
2.2.7. Errors
Sometimes updates in Autoplot's dependencies cause this error:
Unsigned application requesting unrestricted access ...
To fix, start javaws -viewer and delete all CDF and Autoplot applications and then delete all CDF and Autoplot resources. Webstart is touchy software and resetting things almost always seems to help.
3. Navigation
3.1. Address Bar
In the address bar at the top of the GUI you can enter the name of a supported format type in a data address. If the data address is an URL, http or ftp is used to make the remote file local. A fully-qualified address will contain both the location of the file and additional arguments to read in the data. For example, it may contain a CDF filename, plus "?BGSM" to indicate the variable to plot. Together these form a URI, which is the precise address of the data. Autoplot will help build URIs by providing graphical editors to pick the data. Autoplot URIs are designed to give a concise way to refer to data: put a data file in a web location, and send the URI to a colleague so he or she can look at it too.
Note Autoplot has a basic and expert mode. In basic mode, the address bar is not visible and is replaced with a controller for the time range.
Sometimes Autoplot URIs do not refer to files. They still locate data, but one of Autoplot's plug-ins interprets the address. For example, vap+cdaweb:ds=CRRES_H0_MEA&id=FLUX&timerange=1991-10-09 points to data from the CRRES instrument provided via CDAWeb at NASA/Goddard. vap+audiosystem:spec=512&len=2.5 samples from the local machine's soundsystem for 2.5 seconds.
3.2. Bookmarks
Bookmarks may be selected in two ways
- By selecting the
Bookmarksmenu item and then browsing a tree of bookmarks. - By selecting
Bookmarks->Browse and Manageand clicking on a bookmark.
Note that the section option allows you to view metadata associated with a bookmark (if available) by clicking on the Detailed Description button. In addition, the second option allows you to place a plot for a given bookmark in a separate panel or on top of the current panel in view.
3.3. GUI
Basic user interactions including: Left-click and drag to zoom, Middle-click and drag to pan, Click on axis and drag to constrain interaction to one axis, Ctrl-Z / Ctrl-Y to undo/redo changes (CMD-Z/CMD-Y on Mac), Shift-Click on plot to move plot box.
More complex interaction with plots, including: mouse wheel control, keyboard entry of axis range, context overview plots
Here are more videos, described below: http://www.youtube.com/user/AutoplotDevelopers
3.4. Keyboard
Keyboard shortcuts.
- ctrl-Z undo ctrl-Y redo
- ctrl-R reset zoom
- ctrl-shift-R reset Y zoom
- ctrl-shift-X reset X zoom
- ctrl-shift-Y reset Y zoom
- ctrl-plus increase font size
- ctrl-equals increase font size
- ctrl-minus decrease font size
- ctrl-tab next plot element
- ctrl-shift-tab prev plot element
3.4.1. General
- CTRL + play button (green arrow button) adds a plot below.
- Shift + play button (green arrow button) adds an overplot.
- CTRL + R resets the zoom of the plot in focus.
- TAB while typing in address bar to show list of completions.
- Example: http://autoplot.org/data/ + TAB will show a list of files in that directory
- Example: http://autoplot.org/data + TAB will only show a list of files that you have accessed previously that match the string
3.4.2. Axes Shortcuts
When pointer is over an axis
- Middle mouse button drags the axis.
- Mouse Wheel zooms in and out.
- Ctrl + Mouse wheel pans the axis. This allows for rapid scanning of the data.
3.5. Menu Bar
Here are some more useful items from the menu bar.
3.5.1. File->URI History
Autoplot keeps track of everything you have plotted in the file HOME/autoplot_data/bookmarks/history.txt. The URI history dialog is a tool that tries to sort out the history and provides a search tool for locating lost datasets. For example, if you know that you were plotting a txt file, set the filter to "vap+txt" to see all the matching URIs.
3.5.2. File->Export Data
Data can be exported to a number of data types. (In addition to reading in data, some data sources will export data as well.) Only the current selection is exported, so select the item to export, then File->Export will bring up the dialog. From here, you can select the format (e.g. txt for text file) and whether the original data, the processed data (sliced, fftPower,etc), or just the visible data should be exported.
Note that while only the currently selected plot element's data is exported, in the future you will have more options for exporting data.
3.5.3. Options->Auto->Autoranging
Sometimes you want to look at different data on the same axis. This provides a method that turns off Autoplot's autoranging. Note that a future version of Autoplot will include a "reluctant" autoranging mode where axes will be reset only when units change or ranges change dramatically (set dom.options.autorangeType='reluctant' to play with this).
3.5.4. Bookmarks
Autoplot will keep track of data URIs in bookmarks.
3.5.5. Tools->PNGWalk
The PNGWalk tool is an application for looking at series of images found on websites. You can also run a pngwalk from the current product by giving a timerange and a template for each filename (e.g. $Y$m$d.png).
3.5.6. Tools from HOME/autoplot_data/tools (User extensions)
The bottom section of this submenu are all scripts found in HOME/autoplot_data/tools/*.jy. Example scripts can be found here http://autoplot.org/data/tools/.
4. Tabs
4.1. Axes
The Axis tab contains basic controls for the two axes of the plot in focus, the plot title, and the colorbar title (the colorbar may not be visible but it always exists).
|
|
Ranges for the x-, y-, and colorbar are strings that are parsed using the current axis units (such as time or dimensionless). For example,
- For an axes with time units: "2000 through 2010", "Sept 16 2009", "2009-009 10:00 to 11:00", and "2009-009T00:00 to 2009-009T10:00" "2012-12-14T23:00/2012-12-14T24:00".
- For physical ranges, use the keyword "to" to delimit two real numbers. When the axis has a physical unit, such as "m/s" the range can be qualified with a unit, as in "0 to 100 cm/s."
- An automatic test showing example ranges is here.
All labels may include "Granny" strings defined by Grandle and Nystrom [3]. A few of these are
- !A superscript
- !B subscript
- !E superscript with smaller font size
- !D subscript with smaller font size
- !N normal position
- !C carriage return (newline)
so that !A2!Ns!A-2!N results in m2s-2.
Labels may include Unicode (indicated using its entity number, e.g., Å). In addition, the named html entities listed at [4] and [5] may be used. The list includes:
- Greek letters: Α Β Δ α β δ π ρ ω
- Math symbols: ∑ (∑) ± (±)
When selected, Isotropic constrains the x- and y-axis to have the same data-to-pixel scaling. This option is ignored when the axes do not have compatible units.
When Legend Label is checked, a legend is added to the plot, and an icon for this panel is added. Note the plot also has a legendLabel property that when set to False ([menubar]->Edit->Properties) will suppress the label.
4.2. Metadata
The Metadata Tab provides information about the data. It is a tree view that contains a number of nodes of information.
The Metadata node is metadata provided by the data source. In general, Autoplot doesn't understand the metadata and simply provides it for reference by the user. However, if the data source also identifies a "metadata model," this is indicated in parenthesis and Autoplot will use the metadata to get axis labels and ranges.
The Statistics node provides basic statistics about the data. For example it shows the mean and standard deviation, and the number of valid points. Cadence indicates the typical spacing between points and is used to insert breaks in time series.
The Dataset node allows some inspection of the dataset itsself. The size of the dataset, its named dimensions, and their size are identified. Metadata for plotting found within the dataset is available as well. You can look at the digital data (though only the first twenty or so points) as well.
4.3. Style
The Style tab contains controls for the style of the plotted element.
Autoplot tries to pick reasonable default settings such as data rendering method (series versus spectrogram), but also default symbol sizes and connectors. To further specify how a plot should appear, click on the plot element to modify its color or symbol, for example, or for spectrograms the colorbar (e.g. grayscale vs colors).
Note that you can set the rendering method and default settings via the plot context menu (right-click), under Plot Style.
Autoplot will swap out controls based on the rendering method.
5. Adding Plots
Right click->Add plot can be used to add additional plot on the plot. Axes from multiple plots can be bound together so that adjusting one triggers a change in others. This video shows how to add a second plot and bind the y-axes together. Note Autoplot tries to bind time axes together automatically, since this is a common mode of operation. (A future version of Autoplot will allow redundant x-axis labels to be removed to save space when many plots are used.)
5.1. Layout
The Layout Panel (enable with Options->Enable Feature->Layout Tab) provides access to all of the plots and plot elements on the canvas. This allows you to swap plot position, delete plot elements, add groups of plots, etc.
5.1.1. Plots
The Plots area contains a simplified view of the plots on the page. Each plot has a context menu that allows the plot to be edited or deleted. Also a group of plots can be added below, for example a block of 3 by 3 plots.
Multiple plots can be selected by holding the control button down while clicking. Two plots can be selected and Plots->Swap Position will exchange their positions. Also multiple plots can be edited at once, for example to set all the titles the same and then make tweaks for each plot.
5.1.2. Bindings
This Bindings area contains a list of bindings for the application. These are properties that are bound together. For example, a stack of plots with a common x axis all have their x axes' range bound to the DOM property "app_0.timerange". If one right-clicks on a binding, it can be deleted.
5.1.3. Plot Elements
Plot elements have the job of rendering the data onto a plot. The Plot Elements section allows them to be individually controlled and deleted. Note when a vector is plotted (e.g. Bookmarks->Demo 5), then a plot element is added for each component, and these are all children of an inactive parent component. The inactive component can be used to control all the children at once. For example, adjust the line thickness of the parent and all the children will be modified.
5.2. Data
The Data Panel (enable with Options->Enable Feature->Data Tab) contains controls for processing the dataset after it is loaded.
For example, the fill value can be set manually when the dataset has failed to identify it.
The Data Source sub panel controls the data source node that has focus. The URI of the data is indicated for convenience in a small font. Valid Range explicitly overrides the valid range of the dataset.
Data Post Processing specifies the operations that should applied before displaying the data. Originally introduced to allow for slicing of high-dimensional (high "rank") data sets, this also has a number of operators that are described below.
When the Operations field is "|slice<dim>(<index>)" the slice dimension and transpose gui controls become enabled to provide a more usable control for the slice. For example, turning the mouse wheel over the slice index control or operations text field will rapidly adjust the slice position. (Undock the data tab to see the slices as the adjustment is made.) Slice dimension controls which dimension should be sliced. Note that the plot element will automatically set this when a high dimensional dataset is plotted. Transpose will flip the dimensions of the data before plotting.
When a vector component is plotted, component simply identifies the component to plot.
Additional Operators When the operations field starts with the pipe (|) character, it is a list of filters that are to be applied to the data. These filters include:
|histogram perform an "auto" histogram of the data that automatically sets bins.
|logHistogram perform the auto histogram in the log space.
|log10 take the base-10 log of the data.
|exp10 plot pow(10,ds)
|slice0(index) slice the data on the zeroth dimension (often time) at the given index.
|slice1(index) slice the data on the first dimension at the given index.
|collapse0 average over the zeroth dimension to reduce the dimensionality.
|collapse1 average over the first dimension to reduce the dimensionality.
|transpose transpose the rank 2 dataset.
|fftWindow(size) plot power spectrum by breaking waveform data in windows of length size (experimental, not for publication).
|fftPower(size) plot power spectrum by breaking waveform data in windows of length size (experimental, not for publication).
|hanning(size) apply hanning filter before fftPower.
|smooth(size) boxcar average over the rank 1 data.
|diff finite differences between adjacent elements in the rank 1 data.
|accum running sum of the rank 1 data. (opposite of diff).
|grid grid the rank2 buckshot but gridded data into a rank 2 table.
|flatten flatten a rank 2 dataset. The result is a n,3 dataset of [x,y,z]. (opposite of grid)
|negate flip the sign on the data.
|cos cos of the data in radians. (No units check)
|sin sin of the data in radians. (No units check)
|toDegrees convert the data to degrees. (No units check)
|toRadians convert the data to radians. (No units check)
These filters can be chained together like so: |slice1(10)|histogram. The GUI in Autoplot2011 contains a popup list for convenience. Trigger a popup on the editor to see the list. The final version of Autoplot2011 will have a GUI for editing this string. In the DOM, this is the same as the "component" property of a plotElement.
5.3. Script
The script panel (enable with Options->Enable Feature->Script Panel) provides a convenient place for writing and reviewing the Jython scripts to control the application and load data. It provides a simple completion and right-click to see additional options. This is intended to provide full access to advanced users.
Note a .jyds file executed in the "data source context" and loads data from a number of places and performs operations to create a new dataset. .jy files are executed in the "application context" and control labels, add components, run batches and plot ad-hoc datasets.
Another use case for scripting is to add new functionality to the GUI. Scripts that run in the application context can be put in the folder "HOME/autoplot_data/tools/", and will appear under the tools menu when the app is reloaded (or getViewWindow().reloadTools() is run). For example, save this in the file "HOME/autoplot_data/tools/flashFocus.jy:
# label: Flash Focus #cause the selected plot element to flash r= java.util.Collections.singletonList( dom.controller.plotElement ) for i in range(3): dom.controller.canvas.controller.indicateSelection( r ) java.lang.Thread.sleep(500)
Reload the tools, and a new menu item "Flash Focus" should appear. The script will cause the focus plotElement to flash three times. Note the URI "script:http://autoplot.org/data/tools/flashFocus.jy" will display a verification GUI that also allows the script to be installed in the tools area.
Jython scripting is being finalized and documented. More documentation is coming soon.
6. Cookbook
The cookbook page describes quite a few useful tricks, such as resizing plots and putting exponents in labels.
7. Advanced Topics
7.1. Layout fine-tuning
Autoplot's canvas consists of a number of plots which are the two-dimensional spaces on which data is rendered. Plot elements connect data to the plots, by identifying the data source, plot, and method for rendering data. A plot can contain a number of plot elements, as is the case when two datasets are drawn on the same axes, (also referred to as an overplot.) A color bar is connected to all plots so that multiple spectrograms can be rendered using the same color map. The color bar is hidden when it is not needed.
The cookbook page describes quite a few useful tricks, such as resizing plots and putting exponents in labels.
Autoplot uses rows and columns to specify the location of components on the drawing canvas. These objects are controlled with strings that efficiently specify the location. For example, the horizontal (column) layout from the left is specified as a percentage plus a shift from the left L%+S, where
- L is the percentage of canvas to place the left side of the plot box (only data, no labels)
- S is the shift (in px or em)
To have the labels on the y-axis rendered outside of the canvas, use 0%+0
The layout may be specified in the URI sent to the Autoplot server. In this case, the left and right positions are comma separated, as in L%+S,R%+S. The default is 0%+8em,100%-6em. For the bottom and top positions, the default is 100%-3em,0%+2em.
Note that rows and columns are children of the marginRow and marginColumn. Instead of specifying the normal position with respect to the canvas, these rows and columns position are specified with respect to the marginRow and marginColumn. The autolayout feature automatically adjusts the position of the marginRow and marginColumn to make the plot fit on the page.
7.2. Time Parsing / Formatting
One code for parsing and formatting times into strings is used throughout Autoplot to:
- parse times within a file
- parse times in a filename (and URI)
- generated times in scripts
Note this is different than the code that parses ranges to control axes. That one must guess the format, then parse. This one is given a time format, and then and parse strings to get timeranges or vice-versa.
Example times:
- $Y-$m-$d 2010-06-23
- file:///tmp/$Y$m$d.dat file:///tmp/20100623.dat
Field types (see #Wildcard_codes for full list and definitions):
- $Y $m $d $H $M $S $(milli) $(micro)
Field qualifiers:
- $(H;span=4) digit indicates beginning of 4 hour interval
- $(m;Y=2004) The year 2004 is implicit
Note %{m;Y=2004} may be used instead
7.3. Aggregation
Data coming from files can typically be combined, or aggregated. For example, a single CDF file can be read (using the CDF plug-in) with the URI:
vap+cdf:ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/swe/2008/ac_k0_swe_20080620_v01.cdf?Np
If this is one of a series of files, the variable Np can be plotted across several files using the notation:
vap+cdf:ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/swe/$Y/ac_k0_swe_$Y$m$d_v$v.cdf?Np&timerange=2008-June
Most of the letters used for the wildcards for time are the same as the Unix date command, for example $Y means a four-letter year while $y means a two-letter year.
Any file-based reader that returns time for the first dimension can be aggregated. Note "Tools->Aggregate..." will attempt to create an aggregation spec from a file URL.
7.3.1. Aggregation GUI
Aggregation URIs will have an additional controller in the data source editor. This allows timeranges to be browsed and selected. The field "Time Range for Aggregation" controls the span that will be aggregated. This is a time range parsed like other timeranges in Autoplot: 2010 is a year, 2010-Jan is a month, 2010-Jan-01 is a day, 2010-Jan-01 07:00 to 08:00, and so on. Below are three droplists that allow an interval to be selected. The GUI will query the server to see which intervals are available. First select from the available years, then from available months within that year, and then a day. "Copy" will then copy the value into the time range text field. The reduce checkbox enables the experimental data reduction feature where data is reduced to axis resolution as it is read in.
7.3.2. Wildcard codes
Aggregation URIs will contain from the following wildcard codes:
- $Y four-digit year
- $y two-digit year
- $j three-digit day of year
- $m two-digit month
- $b three-letter month (English locale only)
- $d two-digit day of month
- $H two-digit hour
- $M two-digit minute
- $S two-digit second
- $(milli) three digit milliseconds since second boundary. (please use subsec instead of this)
- $(micro) three digit microseconds since millisecond boundary. (please use subsec instead of this)
- $(subsec,places=3) three digits are milliseconds. $(subsec,places=6) means the 6 digits are microseconds.
- $v version number, decimal sort. $(v,alpha) is for alpha sort, and $(v,sep) for "x.y.z"
- $o orbit number, arguments like $(o,id=crres) make this useful. see http://www.autoplot.org/developer.orbitTimeSpec#Orbits_in_Time_Ranges
A version wildcard is allowed. Versioned URIs have the form:
vap+cdf:ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/swe/$Y/ac_k0_swe_$Y$m$d_v$v.cdf?Np&timerange=2011-June
Also, parenthesis can be used to modify fields. For example:
- $Y$m$d-$(H,span=12) indicates the hour is the beginning of a 12 hour interval indicated by the name.
- $Y$m$d_$H$(M,span=10) indicates the files are in ten-minute blocks.
- $(m,Y=2011)$d indicates the year field is implicit. TODO: bugs prevent this from working, since we look for $Y to detect aggregations...
7.4. ASCII Editor
Using the ASCII Table Data Source Editor GUI
Details on how the ASCII table reader is implemented is described at ascii_data_source.
A GUI editor can be used to make it easier to read in data. A table is displayed showing how the reader would parse the file. Darker lines that are not broken into columns are lines identified as non-records, such as header lines.
7.4.1. Header tab: identify records and non-records
Skip Lines: the number of lines to skip before attempting to parse records. Press "Select" then click on the first parseable line.
Comment Prefix: lines starting with this are treated as non-records.
7.4.2. Times tab: specify how times are parsed
Time Format: format for interpreting times. For example $Y+$m+$d means the first three fields are interpreted as year, month, and day separated by a space. $Y$m$d means the first field is an 8-digit date. Press "Select" and highlight the fields in one of the records. This will copy the fields up into the Time Format text field. Then in the text field, highlight each field and use the droplist to identify the field type. If the field is already ISO8601 compliant, manual specification of the time format is not required. See #Wildcard_codes for a full list of time codes.
Note that fractional times may be given in the ASCII file, e.g.,
2001 01 1.5 -44.1
will be interpreted as a value of -44.1 at noon on January 1st.
7.4.3. Data tab: identify data to plot and interdependence
Column: the column to plot. Note if the first record is column labels, then these labels can be used to identify columns. Press "Select", then click on a field to select the column. If a several adjacent fields are selected, then a rank 2 table is plotted (e.g. as a spectrogram)
Depends On: the column that tags the column to plot. This is the independent parameter upon which the dependent parameter depends. Press "Select" then click on the field to select the column.
Time checkbox: identify the "Depends On" as times.
Data can also be formatted to ascii tables. Presently data is formatted to a comma-separated-values (csv) file, and no formatting options are provided. Use Autoplot's "File->Export Data", then select a file name with a .csv, .dat, or .txt extension.
7.5. Plug-in Data Sources
Plug-in data sources are used to load data into Autoplot using a URI specification. The URI is used to identify the plug-in and then the plug-in uses the URI and its parameters to read in the data (into Autoplot's internal data model, QDataSet). Some plug-in data sources have graphical (GUI) editors that make it easier to compose URIs. For these plug-ins, clicking the folder icon next to the address bar will launch the editor, otherwise the URI completions is used to provide the user with the available parameters.
For example, the ASCII Table reader URIs have the form
vap+dat:<file>?<params>
and the params identify how to parse the ASCII table with parameter names like skipLines and delim. The
vap+dat:
explicitly requests a plug-in (a guess is made if not given). (Details on how the ASCII table reader is implemented is described at ascii_data_source.)
Note data from servers that use query parameters (with URLs containing ?) cannot be read directly, since the query parameters are interpreted by the Autoplot reader plug-in. Either read the data to a local file (changing the name so that it does not contain a question mark) and read that, or a Jython script (.jyds) can be used to read the data.
7.6. Discovery Plug-Ins
A discovery plug-in helps the user develop a valid URI to enter into the address bar.
Discovery plug-ins are activated by entering their vap prefix (e.g. vap+cdaweb:) into the address bar and are typically accessed using a GUI by selecting "File->Add Plot From ...".
In some cases, these plug-ins use knowledge of conventions used for the layout of a data bases to allow the user to browse all available data from a given data provider. In other cases, these plug-ins help the user plot data from one or more data providers, data directories, and/or data files without having to enter a complex URI into the address bar.
Data providers interested in making their plug-ins discoverable should see Adding_Data_Sources.
7.6.1. CDAWeb
URI prefix: vap+cdaweb:
File->Add Plot From->CDAWeb or by clicking this linkThe CDAWeb group at NASA/Goddard provides a large volume of data in CDF files which Autoplot can read. This plug-in knows how to query the database to see what is available and provides a GUI for searching and filtering the list.
To see the list,
- Click this link to start Autoplot with the list shown, or
- in Autoplot, select
File->Add Plot From->CDAWebas shown in the top image to the right.
7.6.2. das2server
URI prefix: vap+das2server:
Das2 servers are used to supply data to Das2 applications for the Plasma Wave Group and its collaborators. Since Autoplot is a Das2 application, it's easy to make these data sets available for plotting in Autoplot. Note quite a few of these IDs were introduced for small studies and no longer work. The group is working to add additional metadata to identify working products. Data sources that identify an example range should work for this range, and they will be adding automated tests to ensure this.
Note the Das2 Server sends data to Autoplot via das2streams or QStreams.
7.7. Remote Bookmarks
A bookmarks file can be put in a web-accessible location, and team members will receive updates made to this file. For example, bookmarks:http://virbo.org/meta/viewDataFile.jsp?docname=3BC4F65B-8D59-8046-09F2-976533A13CD9&filetype=data are the set of demo bookmarks, and if we add a bookmark to the list, everyone will get the update. You can create a remote bookmarks file by exporting a single folder (besides the root) with the bookmarks manager.
7.8. Formats Read
The data source type is determined using the mime type in the HTTP headers; if the mime type is not specified, then the file extension is used. The data source type can be explicitly specified by prefixing the URI with "vap" plus an extension followed by a colon. For example: vap+dat:http://www.autoplot.org/data/autoplot.asc tells Autoplot to treat the data returned by this URI as ascii-table-formatted.
Once the data source type (most often the file format type) is identified, the URI's parameter string is interpreted. Different URI types may have different parameter arguments. Additional formats may be added - see the tutorial for adding a data source to Autoplot: Adding_Data_Sources
Note that if a parameters string is not given, the parameters in the file or data stream can often be listed by clicking the folder icon to the right of the address bar, or by pressing Tab to trigger completions.
7.8.1. ASCII Table
The ASCII Table reader reads in a flat ASCII file with one record per line. Each line of the file is identified as a record or non-record. Autoplot URIs are the name of the ascii file and parameters that specify how to parse the file, listed below. A GUI is also provided that allows the URI to be created graphically. This does not provide access to all the available controls, but is much easier to use.
- mime type: text/plain
- extensions: .dat, .txt, .csv
- example URLs:
ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_1963.dat?column=field17&timerange=1963 http://goes.ngdc.noaa.gov/data/avg/2004/A1050412.TXT?skip=23&timeFormat=$y$m$d+$H$M&column=E1&time=YYMMDD ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/omni/omni2_$Y.dat?column=field17&timerange=1963&timeFormat=$Y+$j+$H&time=field0&validMax=999
7.8.1.1. Parameters
- fixedColumns an optimized parser should be used since each row of the file has a fixed column width. By default the row are split into columns using a delimiter. The value may take several forms:
- value contains ",": specify column locations as in "0-10,20-34"
- value is int: specify the number of columns in each row. Actual may be less.
- value is unspecified: the first row is split to determine the column locations.
- columnCount override the number of columns.
- column identifies the field in each record to treat as the dependent variable. By default the columns are named field0, field1, field2, etc.
- depend0 identifies a field as the independent variable.
- rank2 when in the URL indicates that the dataset produced should be a rank 2 dataset, and the number of fixed columns indicates the number of elements per row. When the value contains a colon (:), this indicates a range. The range is <firstColumn>:<lastColumn exclusive> or <firstColumn>-<lastColumn inclusive>.
- value may be empty, meaning use all columns
- 1: means the second row and up.
- -5: means the last five rows.
- 20:24 means the four rows starting at the 21st row.
- field3:field6 three columns
- B_x_gsm-B_z_gsm three columns.
- depend1Labels indicates the first record contains the labels for the rank 2 dataset. (<firstColumn>:<lastColumn exclusive>)
- depend1Values indicates the first record contains the values for the rank 2 dataset (so it is displayed as a spectrogram). (<firstColumn>:<lastColumn exclusive>)
- skip is an integer indicating the number of lines that should be skipped before processing begins.
- fill is the number that indicates missing or fill data.
- timeFormat specifies the time format, based on the Unix
datecommand. "ISO8601" means the times are ISO8601 conforment, or use template with fields from #Wildcard_codes. - time specifies the field that is the time record. This also sets the independent variable.
- delim identifies the delimiter character. By default, the first record is inspected for commas, tabs and then whitespace.
- comment prefix string that indicates records to ignore.
- fill values to be treated as fill data.
We are planning on adding additional parameters, more here: developer.newAsciiParams
7.8.1.2. Notes
column identifiers
Columns are referenced by field identifiers, and the reader will create valid identifiers when the header is not valid. So for example BX-GSM is BX_GSM, etc. "field0" will always refer to the first column, etc.
range notation
Ranges of columns are specified for several keywords, such as depend1Labels and rank2. These are specified as follows:
- 1-3 columns 1 through 3
- field1-field3 columns by generic field name
- BX-BZ columns by field name (BX and BZ are example column identifiers, see notes above)
- 1:4 columns 1 through 3 (colon is exclusive to be consistent with Jython, etc.)
- -5:-1 last columns, but not the very last one.
- -5: last columns
7.8.2. Binary table
BinaryTableReader reads in datasets from binary files. Data are assumed to be in records of fixed-length.
- mime type:
- extension: .bin
- example URL: file:///media/mini/data.backup/examples/bin/fromidl.bin?type=float&fieldCount=2&column=1
- supports formatting: yes, rank 1 and 2, written out as doubles.
- parameters
- byteOffset int, number of bytes skip before reading the data. default is 0.
- byteLength int, total number of bytes to read. default is the content length.
- recLength int, the number of bytes per record.
- recOffset int, the byte offset into the record.
- fieldCount int, number of fields per record, when recLength is not specified. default is 1.
- column int, field number for the dependent parameter, then recOffset is not specified.
- type data type for the dependent parameter. double, float, long, int, short, byte, ubyte. Default is ubyte.
- depend0Offset int, byte offset into each record for the independent parameter.
- depend0 int, field number for the independent parameter when depend0Offset is not specified. Default is no independent parameter and the byte offset is the independent parameter.
- depend0Type data type for the independent parameter. double, float, long, int, short, byte.
- byteOrder data type byte order (endianness). little or big. Default is little.
- validMin, double minimum valid value
- validMax, double, maximum valid value
- recFormat, string, format specifier for each record.
- "d,13f" 8-byte double followed by 13 (4-byte) floats
- "i,s,ub" int, short, unsigned byte
- "x,ub,ui" skip byte, unsigned byte, unsigned int
7.8.2.1. Use-cases
Reverse engineer binary format How to reverse engineer a binary file with the binary reader:
- Plot the data using the default 1-byte data.
- Look for the courser repeating pattern, which are probably records. Timetags will typically identify records, because they are far from zero. Set recLength= this length. rank2 can be used to look at each records bytes in a spectrogram.
- Look for a repeating pattern. This implies a data type. For example, if every fourth point is a peak, then try type=float. If the period is 15 bytes long, then try to identify the data type of the first field by using recLength=15&type=float, recLength=15&type=double, etc. Once the first field's data type is appearent, it's easy to see if the recLength is correct. Use recOffset to look at other bytes within each record.
- If you're getting repeating data with large exponents, then try byteOrder=big
- Once the byteOrder is determined, then it's much easier to identify fields.
- Example where the query parameter were figured out by the above approach: [6]
Unicode to ASCII
Netbeans writes out a Unicode file of its output window. (I could tell this because ascii values were interleaved with zeros.) iconv failed to convert the file, so I tried converting it with Autoplot:
ds= getDataSet('vap+bin:file:///tmp/output1249405460816')
# skip every other record, skipping the zeroth record.
ds2= ds[1::2]
# save it out as a binary stream.
formatDataSet( ds2, 'vap+bin:file:///tmp/output1249405460816.txt?type=ubyte' )
7.8.3. CDF
Reads in a variable from a Common Data Format file.
- mime type: application/x-cdf-file (Note some servers advertise a content type of application/x-netcdf.)
- extension: .cdf
- Example URL: [7]
- Supports formatting: Yes, rank 1 and 2, though not yet ISTP compliant.
- Parameters
- The parameter is the name of the cdf variable.
- If not specified, a list of possible variables is given.
- Wildcards and aggregation
In the following, we tell autoplot that the file name has a four-digit year, a two-digit month, and a two-digit day. Then we ask it to plot data on the day 20000109 using the time wildcards described in [8].
The first part of the url is
http://cdaweb.gsfc.nasa.gov/istp_public/data/polar/hyd_h0/
the file part of the url is
2000/po_h0_hyd_$Y$m$d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20000101
to specify more than one day, use
2000/po_h0_hyd_$Y$m$d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20000101 - 20000103 (end is not inclusive)
to allow access to data that crosses a year boundary, use
$Y/po_h0_hyd_$Y$m$d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20001231 through 20010101
7.8.4. NcML
NcML is an XML representation of netCDF metadata.
- Example URL: [9]
7.8.5. SPASE
- SPASE: If the URL or filename corresponds to a SPASE XML file with a NumericalData element, an attempt is made to plot the first AccessURL in the AccessInformation section. (In the future, default behavior will be to look for a URL in the Granule element and to use the information in the NumericalData node to populate the metadata tree.)
- Example URL: [10]
7.8.6. VAP
VAP: a ViRBO Autoplot files save the state of the application. This is just like a PowerPoint file which is your whole presentation, or an HTML document which is the whole page of a browser, this contains plot positions, data source references, and style settings.
Note if a data source URI is entered on the address bar, the whole application is reset to the new vap file settings. However, if "File->Add Plot From..." is used to access a vap, data URIs from within the vap are accessible.
- Example URL: [11]
7.8.7. Das2Streams and QStreams
- Das2Streams and QStreams: Das2Streams are self-describing, streaming format developed and used by the Plasma Wave Group at the University of Iowa. "Streaming" is the requirement that at any point along the stream, you have a all the data you need have a valid and complete stream. They are generally useful for serializing and deserializing data in das2's internal data model, and Autoplot's QDataSet model. The design goal is any dataset that can be represented in Autoplot can be serialized into a das2Stream. "QStream" refers specifically to a das2Stream with a QDataSet on it, and has the extension .qds. Legacy das2Streams have a .d2s extension.
- Mime type: application/x-das2stream
- Extensions: .d2s .das2stream .qds
- Example URLs: [12] [13]
- Supports Formatting: yes, rank 1, 2, 3 to QStream
- parameters
- arg_0 is the named parameter within the stream, if not specified then default parameter is used.
- output parameters
- type
- binary means format to binary stream
- type
7.8.8. CEF File Reader
Reads data in the Cluster Exchange Format CEF for Cluster. This is experimental, but should work fine for at least 1-D data.
- Example URL: [14]
7.8.9. netCDF file reader
Reads in a variable from a netCDF file.
- mime type: application/x-netcdf
- extension: .nc, .ncml, .hdf5, .h5
- Example URL (described on the unidata web page): [15]
- parameters
- The parameter is the name of the netcdf variable.
- If not specified, a list of possible variables is given.
Times can be encoded in NetCDF files as ISO8601 times, using a 2-D array of characters. The second dimension should be between 14 and 30 characters, for example 20110101T00:00 is 14 chars long, 2011-Jan-01T00:00:00.000000000 is 30 characters long.
7.8.10. OPeNDAP
Reads in data from OpenDAP servers. When data files are served from an OPeNDAP server, subsets of a variable in a file may be requested.
- mime type:
- extensions: .dds and .dods
- Example URL: [16]
- Parameters
- The parameter following the question mark is the name of the OPeNDAP variable. If not specified, a list of possible variables is given.
- Numbers in brackets optionally control the cadence in the notation [min:step:max] or [min:max]. See the OPeNDAP/DODS documentation.
7.8.11. Excel Spreadsheet
Data is read from a Microsoft Excel spreadsheet using Jakarta POI.
- mime type: application/vnd.ms-excel
- extension: .xls
- Example URL: [17]
- Supports formatting: yes, rank 1 and 2
- parameters
- column is the name of the column (e.g. "C"), and in brackets the starting and ending rows.
- depend0 is the name of the column identifying the X values for each value.
- plane0 is a third parameter. Use with caution, this will probably go away.
- firstRow is the first row to read (1 is the first row)
- sheet is the name of the sheet within the workbook to read.
- recCount limit the number of records to read.
7.8.12. TSDS
Data is read from a TSDS server.
- Mime type:
- Extension: tsds (use vap+tsds:http://...)
- Example URL: [18]
- parameters
- None
- Dependencies
- BinaryDataSource
7.8.13. FITS
Flexible Image Transport System [19]. This is using a library that limits what it can access. Soon a new implementation will enable more data sources.
7.8.14. Images
This data source reads images using java's ImageIO library. The images are mapped to a QDataSet internally.
- mime type:
- extension: .gif .jpg .png
- Example URL [22]
- Parameters
- channel specifies how components should be combined to produce the rank 2 dataset.
- If channel is not specified, then a rank 3 dataset with original image channels results.
- Example channels include:
- red, green, blue
- hue, saturation, value (brightness)
- greyscale is faster to calculate than value
- channel specifies how components should be combined to produce the rank 2 dataset.
7.8.15. Wav Files
Plots the waveform within wav files.
- extension: .wav
- supports formatting: yes, rank 1 to 16 bit PCM. xtags must be seconds or UT time.
Example URLs: [23] See also #Multiple_plot_panels
- Parameters:
- channel the channel to plot (by default, only the 0th channel is plotted as a time series).
- offset the number of seconds into the file to start at
- length the number of seconds to plot
7.8.16. Jython Files
Jython scripts can be used to create new datasets by combining other datasets. See also #Jython.
- extension: .jyds
- Example URLs:
- Parameters:
- <name>=<expr> defines parameters for use within the script. Scripts use getParam method to get method parameters.
- <expr> after running the script, this expr is evaluated and plotted. The default expression is "data".
Example Script (/home/user/script.jyds):
amplitude= getParam( 'amp', 1.0, 'the amplitude of the waveform' ) data= amp * sin( linspace( 0, PI*2, 100 ) )
This is called like so:
/home/user/script.jyds?amp=2
An editor is provided that will allow editing of the script, and when getParam( name, default, description ) is used, an entry form is generated for the parameters.
Note if the parameter "timerange" is used, then Autoplot will call the script with new time ranges as the time axis is adjusted.
URIs can also use the resourceURI part to point to a resource that is parsed by the script, such as:
vap+jyds:file:///opt/project/cassini/new_hfr_level2_level3/2010_091_180/n2/P2010118.17?script=file:///home/jbf/project/cassini/jared/autoplot/readL2Files3.jyds
Here the "script=" switch points to the script. This is useful to define a parser for a file type, which allows aggregation to be used.
7.8.17. Inline
Data and short jython commands can be embedded in a URI for demonstrations or quick-and-dirty annotations. An editor is automatically generated for scripts. Any parameter read from the getParam function will be provided in the GUI.
- extension (prefix): vap+inline:
- Example URIs:
- vap+inline:1,2;3,4;5,6;7,2;9,0 # five 2-D points
- vap+inline:1,3,5,7,9;2,4,6,2,0 # same five 2-D points
- vap+inline:1,3;2,4 # the points [1,3] and [2,4]
- vap+inline:ripples(100,100) # demo function from jython support
- vap+inline:linspace(0,1,100),linspace(0,1,100),ripples(100,100) # jython expressions allowed
- vap+inline:ripples(100,100)+randn(100,100)/10 # jython expressions allowed
- vap+inline:ripples(100,100)+randn(100,100)/10&RENDER_TYPE=nnSpectrogram # specify QDataSet properties
- vap+inline:getDataSet('http://autoplot.org/data/autoplot.ncml')&RENDER_TYPE=nnSpectrogram # "decorate" other datasets (TODO: bugs when ? in URI)
7.9. Managing Your Local Cache
Autoplot caches remote files to the local directory $HOME/autoplot_data/fscache. Not only is having local copies necessary for many formats, such as CDF files, but this allows Autoplot to be used when no internet connection is available. This area will grow as more data is read, and presently there is no mechanism to automatically remove infrequently used resources.
7.9.1. Data Providers
7.9.1.1. ro_cache.txt
Data providers will want to know that you can set up a link between your local file cache and the local file location of the data, so that the local cache is not used. For example, http://papco.org/~jbf/1wire/data/furnace.$Y$m$d.d2s is a resource I want to make available to everyone, but in /home/jbf/public_html/1wire/data I have the data locally. In my local file cache, I would use the link file to avoid downloading the data to my local cache, making a second copy. When Autoplot looks for the resource, it will use files from here before downloading.
papco> cat /home/jbf/autoplot_data/fscache/http/sarahandjeremy.net/~jbf/ro_cache.txt /home/jbf/public_html
7.9.1.2. keychain.txt
Often we need to run batch processes, and the keychain.txt allows data providers to store credentials for batch processing. Note only a minimal amount of thought has gone into this, so please be careful with these facilities and use them at your own risk! developer.headless talks about this more.
7.10. Applet
The Autoplot applet requires Java 1.5 or higher and that your browser has Java enabled. The applet typically takes less than 2 seconds to load if the background pre-loading is successful. If the background pre-loading is not successful, the applet will take about 10 seconds to load the first time.
If you are having problems starting the applet, see the Applet Guide or ask for help by sending an email to the Autoplot forum at http://groups.google.com/group/autoplot.
The Autoplot applet has a number of features that are disabled, due to security restrictions enforced by web browsers. When using the applet, you cannot save data to disk or save a pdf or png file of you image. For more information on Java applets, see http://en.wikipedia.org/wiki/Applet or http://java.sun.com/applets/.
7.11. WebStart
When you start Autoplot using WebStart, the full Autoplot application (~10 MB) is installed on your computer by clicking the "Start Autoplot" link. This download and install typically takes about 30 seconds.
Subsequent clicks on "Start Autoplot" links will result in Autoplot being loaded from your computer (unless a new version of Autoplot has been released; in this case additional libraries will be downloaded).
For more information on WebStart, see [28] or [29].
If you are having problems starting Autoplot with WebStart, send an email to the Autoplot forum at http://groups.google.com/group/autoplot.
7.12. Abbreviations
- CDF Common Data Format
- CEF Cluster Exchange Format
- JYDS Jython Data Source. Jython script used to load data into Autoplot.
- JY Jython Script. Jython used to operate Autoplot as a batch mode.
- RTE Run-Time Exception. This is an error that occurs that the user should not see, and is given the opportunity to submit the bug to developers.
- RFE Request For Enhancement, or feature request.
![[22]](http://autoplot.org/jnlp.cgi?http://autoplot.org/data/image/Capture_00158.jpg?channel=greyscale){kind=link}