Autoplot Application (v2016a_7)
New image parsing challenge: monitoring home water use with $8 webcam and $30 Raspberry PI.
Autoplot attempts to allow plotting of data from many data sources, including remote CDF files and OpenDAP servers.
Datasets are identified with URIs, and by the URI's extension, data are loaded into the internal data model. Data are then
displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data. Data can then
be additionally processed with other operations like smooth and FFT, and Jython scripting provides a capable programming environment.
This is the Autoplot2016a branch. This is an update to the production version containing bugfixes and minor new features.
Java 7 is now required. Java 8 will be required sometime soon.
Single-jar version, autoplot.jar, can be run without network access and from the command line.
Experimental Mac Release is available.
Bugfixes (2016-09-15 r19429):
- blurTsbUri would fail when a data source was removed, but one of its URIs was still in the history.
- 1677: interpolate2D should allow interpolation along edges when the opposite edge contains fill
- interpolate2D with two rank 0 datasets for xx and yy would cause RTE.
- 1672: layout panel was failing to repaint when plots were added.
- 1636: if statements in Jython scripts could mess up code that would extract the getParam calls to create GUI. Thanks Ivar!
- 1671: contour insert fill values but property is not set
- 1670: spectrogram averaging code assumed that 0*NaN would be 0. Thanks, Ivar!
- 1669: "common path" code, which tries to obscure local references in .vap.zip files, would use "" for data filename (and it would work, surprisingly).
- add a bit of fuzz to the fill value test, after seeing a d2s with 1e31 failing to detect fill
- 1499: Brian's events file (two ISO8601/line) fails to parse.
- 1667: color icons not shown in events list tool.
- rfe521: devel-release had a bug where events list couldn't be aggregated
- 1666: Jython ds[i,j] would return rank 1 when i was integer and j was rank 0 dataset.
- rte_1104366814: format .wav "data extent is too great" message didn't consider scale.
- ImageDataSource would use Datum is TYPICAL_MAX and TYPICAL_MIN, instead of Number
- slices GUI formatted command that was then not supported (: vs ':')
- code to repaint layout panel was triggering itself, causing unnecessary paints.
- 1665: new jumpiness when click-n-drag on x axis.
- 1661: new version of CDF library supports export to CDF using append now works on Windows.
- 1653: clean up "Add additional ticks" dialog.
- 1662: smooth function fails when fill exists in the first boxcar. Thanks Kristoff!
- 1660: Jython setItem assignment of ds[i,j] to rank 0 dataset. Thanks Kristoff!
- rte_0520679247: don't do separate check for timeSeriesBrowse part of URI, which produces confusing messages. Thanks, Mark!
- 1612: recent change would cause RTE when all data was fill.
- 1659: "monitor finished was called twice" when reading das2stream from das2server. Thanks Masafumi!
- 1658: failure to start jython because pylisting.txt, clarify error message
- 1633: ImageDataSet renderer recoded to extract just the visible portion
- 1653: add explicit "clear" button to the ephemeris dialog.
- rfe499: SciPy Plasma colorbar added. Thanks, Steve!
- rfe509: maybeCreateDasPeer should check headless before adding an export data button to the horizontal and vertical slice.
- 1646: correction to overplot of TSB of TSB without timeaxis flips back to timeaxis. Thanks, Ivar!
- 1649: autoranging failure where code to support TSBs without timeaxis inadvertently would allow plot below TSBs to reset axis. Thanks, Bill!
- 1647: changes to HTML source to support "3.4 sec" broke test 140.
- 1648: new issue with URI handling, where the trailing / gets encoded, and don't look for 2014/.gz automatically.
- bugfix in medianFilter, which was never tested thoroughly. Thanks, Ivar!
- 1643: pngwalk file basename property would fail when when product and folder contained "JNO-IONS"
- 1645: add listener for the timerange.
- 1642: add bindGuiSafe, since GUI bindings need to be done on the event thread.
- corrections and improvements to HtmlDataSource.
- DISPLAY_LOG_LEVEL property of Options included in syncToAll so we have more control over pngwalk output, to turn off message bubbles
- 1640: AutoplotDataServer d2s output drops metadata.
- 1635: xaxis fails to unbind.
- 1639: new bug shown by non-timeaxis-TSB script where TSB rebinding is not done. (Never released.)
- 1638: Jython getParam scraping doesn't support Chris' file with no indentation on closing paren.
- dataset( PyObject, Units ) was thoroughly implemented.
- 1632: copy over copy over SCALE_TYPE, TITLE, etc when writing das2streams.
- 1637: binary source failed to read large JADE files gt 268,435,456 bytes.
- 1634: AutoplotDataServer misformats NoDataInIntervalException
- 1629: recent change for setItem for datasets in Jython needed to account for sloppy codes that mishandled units.
- 1490: RichAscii delim was not set properly in one branch (unreleased bug).
- 1630: turn on flag that suppresses reset of TSB, which only affected some of the Hudson tests.
- 1566: rte_0610709130: load vap supports folder names with UTF-8 characters with local files and http.
- Juno Waves survey dataset showed a bug with the LanlNNRebin mode for spectrograms.
- copy plots down initializes DSF TSB properly. The bug would occasionally cause a fault in test100_042.
- ImageDataSource alpha channel was upside-down, relative to the other channels.
- CSV and ASCII table reader use skipLines consistently (instead of skip), and this is presented in completions.
- 1629: jython index bug, where two lists of indeces and one index are used.
- 1624: local .gz files are now uncompressed automatically. Create-parent-directories portion needed to be synchronized.
- in-lining weights calculation of histogram routine, after seeing use of WeightsDataSet doubles execution time.
- support compressed das2streams with zero-length compressed portion. This would hang before the fix.
- 1628: vap+inline:ripplesTimeSeries(100000) hangs in AutoHistogram.
- 1627: (rte_0554827090) reduce memory requirement by handling bundle dataset using just the last bundled dataset to keep track of fill.
- 1604: Tools Manager has overplot and plot below buttons.
- 1626: das2server dataset parameters are reset as id is changed.
- make block that creates /home/Dragon/autoplot_data/jython synchronized, to avoid errors seen during tests.
- das2stream adapter for rank 3 join of tables supports xTagWidth.
- 1625: writeHTML PngWalk crashes when --update is used. Thanks, Jon N.!
- 1569: check for zero in __add__ function, to support Jython sum operation.
- 1623: Jython index bug, where Jython list was misinterpretted as the single index for each dimension.
- 1621: wrong ordinal selected when table is sorted by ordinal.
- test005_014 corrected, as old site http://satdat.ngdc.noaa.gov/sem/goes/data/avg/$Y/A105$y$m.TXT appears to be gone.
- 1616: hooks in pngwalk to allow for scripting new apps. Add switch to disable automatic digitizer recording.
- rte_1585690286: code failed to check if null was found for units.
- 1541: Larry's Jython code where dom.dataSourceFilters would result in Concurrent exception
- rfe247: experiment with allowing the data source to be edited without resetting everything else. Thanks Chris, Bob, and Bill!
- Tooltips indicate description of filters in Filter Chain. Thanks, Chris!
- rfe526: add test for rank join of rank 2 waveforms, to support cases where sampling rate changes for example.
- dataset index range checking flag is enabled by default now, improving error message precision.
- sin,cos,atan,etc added to Mash Up tool.
- add fillValue, validMin and validMax keywords to NetCDF reader.
- AMDA had a .nc file where the times were 1997223000009297, so special code is added to look for this case.
- script:/tmp/foo.jy, /tmp/foo.jy, are all treated as "--script /tmp/foo.jy" to support running from the command line.
- BinaryDataSource has fillValue and units keywords to support PDS4.
- abstract dataset trim(0,len) returns self, improving performance slightly
- unwrap function to mimic SciPy and Matlab function
- add hilbertFrequency, hilbertPhase, and hilbertEnvelope filters.
- hilbert function added.
- unbundle supports rank 2 unbundle.
- addPlot(rowSpec,columnSpec) command added
- rfe523: format to wav now scales data by default.
- rfe521: aggregation of images is supported with new rules for aggregation.
- Improvements to applot.py, which supports plotting from C Python.
- Introduce ifft, for inverse transforms, to support hilbert function.
- bug1664: CDF data source URI keyword replaceLabels=T will use the channel values to label (e.g. 14eV instead of ch01). Thanks, Cindy!
- BinaryDataSource editor supports tab
- trim1 which trims elements from the first dimension, not just the zeroth.
- smooth1 which smooths over the first dimension, not just the zeroth.
- add matplotlib plasma colorbar
- rfe515: add DOTS where the dots/spacing is [0,3]
Less recent features:
- mashup has sqrt function.
- rfe509: horizontal and vertical slices support export data
- tearoffTabbedPane allows tab to be selected programmatically by title.
- rfe505: Jython setItem will correctly convert units as well as implicitly assign them.
- operations processor supports datums in histogram command.
- rfe504: install tool now checks the script for the label and description of the script.
- allow clients to declare new types of colorbars.
- UInt24DataSet added so that we can read pairs of two nybbles to get data from LWA telescope.
- rfe511: BinaryDataSource can now read data as array of nybbles.
This completes the feature set for the new production branch. No major features will be added before branching this version.
A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of
autoplot.org will run this version.
Feedback on the function and quality of this release is greatly appreciated!
Data is input from:
- ASCII files (.dat,.txt)
- CSV files (.csv)
- Binary files (.bin)
- CDF files(.cdf)
- Excel spread sheets (.xls)
- das2Streams (.d2s or .das2Stream)
- ViRBO's TSDS server
- CDAWeb data server (vap+cdaweb:)
- das2servers (vap+das2server:)
- QStreams (.qds), QDataSet on a das2Stream
- Plasma Wave Group das2 Server
- NetCDF files (.nc,.ncml)
- HDF5 files (.h5,.hdf5)
- DODs servers (.dds)
- Cluster Exchange Format files (.cef)
- Fits Format Files
- Images (.jpg, .png, .gif)
- Wav audio files (.wav)
This is not complete, and limitations include:
- Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
- Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.
autoplot.org is the wiki website, and contains
more information than this launch page.