Autoplot Application (20160902b)

Contour, Digital and Spectrogram plotElements make this plot.

Autoplot attempts to allow plotting of data from many data sources, including remote CDF files and OpenDAP servers. Datasets are identified with URIs, and by the URI's extension, data are loaded into the internal data model. Data are then displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data. Data can then be additionally processed with other operations like smooth and FFT, and Jython scripting provides a capable programming environment.

This is the Autoplot2016a branch. This is an update to the production version containing bugfixes and minor new features. Java 7 is now required. Java 8 will be required sometime soon.

Single-jar version, autoplot.jar, can be run without network access and from the command line.

Experimental Mac Release is available.

Bugfixes (2016-09-01 r19377):

• 1499: Brian's events file (two ISO8601/line) fails to parse.
• 1667: color icons not shown in events list tool.
• rfe521: devel-release had a bug where events list couldn't be aggregated
• 1666: Jython ds[i,j] would return rank 1 when i was integer and j was rank 0 dataset.
• rte_1104366814: format .wav data extent is too great message didn't consider scale.
• ImageDataSource would use Datum is TYPICAL_MAX and TYPICAL_MIN, instead of Number
• slices GUI formatted command that was then not supported (: vs ':')
• code to repaint layout panel was triggering itself, causing unnecessary paints.
• 1665: new jumpiness when click-n-drag on x axis.
• 1661: new version of CDF library supports export to CDF using append now works on Windows.
• 1653: clean up "Add additional ticks" dialog.
• 1662: smooth function fails when fill exists in the first boxcar. Thanks Kristoff!
• 1660: Jython setItem assignment of ds[i,j] to rank 0 dataset. Thanks Kristoff!
• rte_0520679247: don't do separate check for timeSeriesBrowse part of URI, which produces confusing messages. Thanks, Mark!
• 1612: recent change would cause RTE when all data was fill.
• 1659: "monitor finished was called twice" when reading das2stream from das2server. Thanks Masafumi!
• 1658: failure to start jython because pylisting.txt, clarify error message
• 1633: ImageDataSet renderer recoded to extract just the visible portion
• 1653: add explicit "clear" button to the ephemeris dialog.
• rfe499: SciPy Plasma colorbar added. Thanks, Steve!
• rfe509: maybeCreateDasPeer should check headless before adding an export data button to the horizontal and vertical slice.
• 1646: correction to overplot of TSB of TSB without timeaxis flips back to timeaxis. Thanks, Ivar!
• 1649: autoranging failure where code to support TSBs without timeaxis inadvertently would allow plot below TSBs to reset axis. Thanks, Bill!
• 1647: changes to HTML source to support "3.4 sec" broke test 140.
• 1648: new issue with URI handling, where the trailing / gets encoded, and don't look for 2014/.gz automatically.
• bugfix in medianFilter, which was never tested thoroughly. Thanks, Ivar!
• 1643: pngwalk file basename property would fail when when product and folder contained "JNO-IONS"
• 1645: add listener for the timerange.
• 1642: add bindGuiSafe, since GUI bindings need to be done on the event thread.
• corrections and improvements to HtmlDataSource.
• DISPLAY_LOG_LEVEL property of Options included in syncToAll so we have more control over pngwalk output, to turn off message bubbles
• 1640: AutoplotDataServer d2s output drops metadata.
• 1635: xaxis fails to unbind.
• 1639: new bug shown by non-timeaxis-TSB script where TSB rebinding is not done. (Never released.)
• 1638: Jython getParam scraping doesn't support Chris' file with no indentation on closing paren.
• dataset( PyObject, Units ) was thoroughly implemented.
• 1632: copy over copy over SCALE_TYPE, TITLE, etc when writing das2streams.
• 1637: binary source failed to read large JADE files gt 268,435,456 bytes.
• 1634: AutoplotDataServer misformats NoDataInIntervalException
• 1629: recent change for setItem for datasets in Jython needed to account for sloppy codes that mishandled units.
• 1490: RichAscii delim was not set properly in one branch (unreleased bug).
• 1630: turn on flag that suppresses reset of TSB, which only affected some of the Hudson tests.
• 1566: rte_0610709130: load vap supports folder names with UTF-8 characters with local files and http.
• Juno Waves survey dataset showed a bug with the LanlNNRebin mode for spectrograms.
• copy plots down initializes DSF TSB properly. The bug would occasionally cause a fault in test100_042.
• ImageDataSource alpha channel was upside-down, relative to the other channels.
• CSV and ASCII table reader use skipLines consistently (instead of skip), and this is presented in completions.
• 1629: jython index bug, where two lists of indeces and one index are used.
• 1624: local .gz files are now uncompressed automatically. Create-parent-directories portion needed to be synchronized.
• in-lining weights calculation of histogram routine, after seeing use of WeightsDataSet doubles execution time.
• support compressed das2streams with zero-length compressed portion. This would hang before the fix.
• 1628: vap+inline:ripplesTimeSeries(100000) hangs in AutoHistogram.
• 1627: (rte_0554827090) reduce memory requirement by handling bundle dataset using just the last bundled dataset to keep track of fill.
• 1604: Tools Manager has overplot and plot below buttons.
• 1626: das2server dataset parameters are reset as id is changed.
• make block that creates /home/Dragon/autoplot_data/jython synchronized, to avoid errors seen during tests.
• das2stream adapter for rank 3 join of tables supports xTagWidth.
• 1625: writeHTML PngWalk crashes when --update is used. Thanks, Jon N.!
• 1569: check for zero in __add__ function, to support Jython sum operation.
• 1623: Jython index bug, where Jython list was misinterpretted as the single index for each dimension.
• 1621: wrong ordinal selected when table is sorted by ordinal.
• test005_014 corrected, as old site http://satdat.ngdc.noaa.gov/sem/goes/data/avg/$Y/A105$y$m.TXT appears to be gone.
• 1616: hooks in pngwalk to allow for scripting new apps. Add switch to disable automatic digitizer recording.
• rte_1585690286: code failed to check if null was found for units.
• 1541: Larry's Jython code where dom.dataSourceFilters would result in Concurrent exception

Older Bugfixes:

• 1618: botched bugfix breaks File->"Add Plot From..."->CDAWeb
• startDigitizer is a public method now, and startQC can be called multiple times without complaint.
• rte_0400646374: Index out of bounds when QC is running and pngwalk is empty. Thanks Ivar!
• rte_0370111585: SliceFilterEditorPanel.validateFilter assumed data was a QUBE.
• rte_0724070161: address inconsistent state associated with Time Series Browse observed by two people.
• corrections in digitizer (DataPointRecorder) when column sorts are used.
• corrections to pngwalk digitizer when digitizing plots without JSON metadata.
• catch annoying 403 error message that shows at LANL.
• 1582: tweak to the guessCadence logic shown by Rosie's data.
• 1594: HDF reader cannot find data in sweswi_data_12min_year1999.hdf. Now groups are supported.
• Rich ASCII code would run on any comment block containing {}, causing error message popup for some files.
• 1490: Improvements to Rich ASCII output following meeting with LANL folks.
• rte_1147829159: update the default pngwalk bookmarks, connect to github bookmarks file for easier maintenence.
• 1612: Mark's example showed where very negative Long for TT2000 was treated inconsistently and mucking up autoranging.
• updates to example scripts in svn.
• 1268: back out until this can be fixed carefully. (Bug is avoid download on the event thread in addPlot dialog)
• Recent change to Add Plot dialog would result in incomplete URIs and other strange errors.
• warning about using a timerange with a non-TSB getDataSet call removed. It wasn't effective.
• 0899: avoid extraneous load of data when createPngWalk is used to create single pngs.

Features:

• add fillValue, validMin and validMax keywords to NetCDF reader.
• AMDA had a .nc file where the times were 1997223000009297, so special code is added to look for this case.
• script:/tmp/foo.jy, /tmp/foo.jy, are all treated as "--script /tmp/foo.jy" to support running from the command line.
• BinaryDataSource has fillValue and units keywords to support PDS4.
• abstract dataset trim(0,len) returns self, improving performance slightly
• unwrap function to mimic SciPy and Matlab function
• add hilbertFrequency, hilbertPhase, and hilbertEnvelope filters.
• hilbert function added.
• unbundle supports rank 2 unbundle.
• addPlot(rowSpec,columnSpec) command added
• rfe523: format to wav now scales data by default.
• rfe521: aggregation of images is supported with new rules for aggregation.
• Improvements to applot.py, which supports plotting from C Python.
• Introduce ifft, for inverse transforms, to support hilbert function.
• CDF data source URI keyword replaceLabels=T will use the channel values to label (e.g. 14eV instead of ch01). Thanks, Cindy!
• BinaryDataSource editor supports tab
• trim1 which trims elements from the first dimension, not just the zeroth.
• smooth1 which smooths over the first dimension, not just the zeroth.
• add matplotlib plasma colorbar
• rfe515: add DOTS where the dots/spacing is [0,3]
• mashup has sqrt function.
• rfe509: horizontal and vertical slices support export data
• tearoffTabbedPane allows tab to be selected programmatically by title.
• rfe505: Jython setItem will correctly convert units as well as implicitly assign them.
• operations processor supports datums in histogram command.
• rfe504: install tool now checks the script for the label and description of the script.
• allow clients to declare new types of colorbars.
• UInt24DataSet added so that we can read pairs of two nybbles to get data from LWA telescope.
• rfe511: BinaryDataSource can now read data as array of nybbles.

Less recent features:

• NearestNeighborTcaFunction to support using calculated data for ticks. Thanks, Kristoff!
• PNGWalkTool provides more hooks for new Jython apps, including magnifier and auto-increment.
• planet.physics.uiowa.edu is the default das2server now.
• Refactor HTTPFileSystem to make the code a little easier to follow.
• PNGWalkTool provides a method for programmatically adding QC metadata.
• HDF reader supports structures.
• PngWalkTool digitizer now includes column identifying the image.
• PngWalkTool provides mechanism for extension, where scripts can get events and can plug in small GUI.

Known issues:

• see Source Forge

This completes the feature set for the new production branch. No major features will be added before branching this version. A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of autoplot.org will run this version. Feedback on the function and quality of this release is greatly appreciated!

Data is input from:

• ASCII files (.dat,.txt)
• CSV files (.csv)
• Binary files (.bin)
• CDF files(.cdf)
• Excel spread sheets (.xls)
• das2Streams (.d2s or .das2Stream)
• ViRBO's TSDS server
• CDAWeb data server (vap+cdaweb:)
• das2servers (vap+das2server:)
• QStreams (.qds), QDataSet on a das2Stream
• Plasma Wave Group das2 Server
• NetCDF files (.nc,.ncml)
• HDF5 files (.h5,.hdf5)
• DODs servers (.dds)
• Cluster Exchange Format files (.cef)
• Fits Format Files
• Images (.jpg, .png, .gif)
• Wav audio files (.wav)

This is not complete, and limitations include:

• Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
• Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.

autoplot.org is the wiki website, and contains more information than this launch page.