Autoplot Application (20160621b)

PNGWalk tool default bookmarks are updated and now maintained at GitHub.

Autoplot attempts to allow plotting of data from many data sources, including remote CDF files and OpenDAP servers. Datasets are identified with URIs, and by the URI's extension, data are loaded into the internal data model. Data are then displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data. Data can then be additionally processed with other operations like smooth and FFT, and Jython scripting provides a capable programming environment.

This is the Autoplot2016a branch. This is an update to the production version containing bugfixes and minor new features. Java 7 is now required. Java 8 will be required sometime soon.

Single-jar version, autoplot.jar, can be run without network access and from the command line.

Experimental Mac Release is available.

Bugfixes (2016-06-21 r19068):

• 1490: Improvements to Rich ASCII output following meeting with LANL folks.
• rte_1147829159: update the default pngwalk bookmarks, connect to github bookmarks file for easier maintenence.
• 1612: Mark's example showed where very negative Long for TT2000 was treated inconsistently and mucking up autoranging.
• updates to example scripts in svn.
• 1268: back out until this can be fixed carefully. (Bug is avoid download on the event thread in addPlot dialog)
• Recent change to Add Plot dialog would result in incomplete URIs and other strange errors.
• warning about using a timerange with a non-TSB getDataSet call removed. It wasn't effective.
• 0899: avoid extraneous load of data when createPngWalk is used to create single pngs.
• bugfix: simplifyScriptToGetParams didn't consider comments, creating a confusing situation where it wasn't clear why it was failing to parse.
• 1603: check for ordinal units in closestIndex function.
• putProperty( ds, QDataSet.SCALE_MIN, rank0qds ) supported.
• Code put in to automatically name the default plane of das2stream didn't consider that there might already be a plane "y", and uses the name property instead for the default value. Thanks Chris and Bill!
• "UNKNOWN ERROR" message was printed when CreatePngwalk was cancelled.
• add noCheckCertificate setting to work around where LANL's certificate is suddenly not working.
• dataset([d1,d2]) where d1 and d2 are Datums of different but convertible Units created incorrect dataset. Thanks Ivar!
• 1599: experiment with invokeLater to avoid hang seen with TimeRangeEditor.

Older Bugfixes:

• 1596: initial feedback on the mash up GUI. Thanks, Craig!
• isSimpleTableDataSet failed with Juno/JADE dataset with rank 2 DEPEND_1. Thanks, Chris!
• 1595: If the Time Format in PNGWalk was not specified, then one png of years 1000-9000 would be run, creating problems for many data servers. Thanks, Chris!
• improve error feedback when a component ("Bx") cannot be found in the das2stream.
• improve feedback for low memory problems, with link to Autoplot.org.
• CDF reader never called monitor.finished(), relying on sloppy clean-up code to call finished.
• 1590: Autoplot's runtime exception handler is used throughout Das2 code.
• 1587: jython error location when callbacks are used.
• 1589: javadoc popup window view-in-browser broken with Illegal character in fragment
• new autoplot.py would use new function getAutoplotScriptingVersion, preventing old Autoplots from running. (never released)
• rte_1262492318: csv reader with bundle=field1-field5 had issues. Thanks Masafumi!
• new Data Sources area in Layout could prevent selection of a plotElement. Thanks Chris!
• rte_0520679223: "####" splits to zero-length array, so input caused index out of bounds. Thanks Bob!
• Jython datumRange('2015-06-25 11:00 to 13:00') would fail to parse as time range. Thanks Kristoff!
• Jython popup documentation box size was not stored on disk between sessions. Thanks Ivar!
• Jython plot command can plot None again, to create a placeholder for data later in script.
• Jython Data Source Time Series Browse mistaken when commented out. Thanks Ivar!
• check that the monitor passed into reference cache park isn't already finished. Thanks Ivar!
• 1584: jyds scripts get 2010-01-01 for timerange on reset, because the TSB controller was still listening. Thanks Masafumi!
• further cleanup of .jyds handling, where exception could occur outside of try/finally block, leaving parked JythonDataSource read hanging. Thanks Ivar!
• rte_1270364473: index out of bounds when handling URI because MMS server doesn't quote the filename like VxOWare code would, improve feedback.
• require that 99% of the data tags be increasing or decreasing to even allow a cadence to be measured. This was 90%. Thanks George!
• improve support for empty VOTable result from PDS/PPI node. Thanks Jenkins!
• spectrogram updatePlotImage would make a local copy of the colorbar for thread safety, but this was not used consistently.
• extraneous calls to monitor.finished would cause noise on stderr.
• 1583: when creating PNGWalk from list of events, make sure orbit timeranges are used so that the pngs can refer to them
• 1582: Juno trajectory segment failed to paint properly because of cadence calculation.
• 1425: jump back into Autoplot button is enabled, but the timerange cannot be resolved.
• das2servers allow realms with HTTP access, limiting the number of times authentication is needed.
• 1560: copy plot when vap+internal is used.
• 1581: null result of ReferenceCache.getDataSet is ambiguous.
• TrimDataSet's trim operator was incorrect when start gt 0, creating error messages.
• 1580: support multiple orbit selection in the TimeRangeEditor
• 1579: old caching mechanism with .jyds scripts would fail to coordinate many simultaneous reads.
• DasProgressPanel multiple calls to finished warning stack trace correctly prints the two calls.
• EventsBar renderer Gantt mode supports where column would be less than two pixels.
• TrimDataSet's trim operator (trim of trim) would fail when the start was greater than 0.
• Jython dataset(QDataSet) wouldn't work, because there was no test for the PyJavaInstance of TrimDataSet.
• trim operator would fail when second index was past the end of the array with incorrect error message.
• link(x,y,z) would fail because of incorrect check when Y was rank 1 and Z was rank 2. Thanks, Masafumi!
• zFill supported for multiple interleaved tables in das2stream. Thanks Chris!
• do early check for unique names in das2stream. This would misformatted das2stream result in strange error.
• Feb2013 change of "asOrderOneUnits" to prevent ms->msec conversion also prevented hr->hr, so that the original unit was never used, even if it is the best.
• script shortcut cheatsheet was missing save.
• butterworth GUI would reject float parameters, which are valid, and would clobber the scientist's input. Thanks, Kristoff!
• revalidate after TSB is loaded and trimmed to timerange, since the trim might break some datasets.
• the timerange editor should load orbits off the event thread.
• When adapting legacy VectorDataSets to QDataSet, name the default plane "y".
• 1576: pointed out extraneous load when undo was pressed caused by leftover TSB listener. Thanks Masafumi!
• das2stream containing new unit would fail to read, whoops!
• bundleNames function would fail to add element when it found an unnamed dataset, now returns "ch_i". Thanks Ivar!
• Jython "recently used files" area shows both .jy and .jyds files, so you don't have to think about the current context.
• 1570: make sure the axis timerange is returned in %{TIMERANGE} macro, not the loaded timerange.
• corrections to triangulation routines. Scale data to be around one before tesselation.
• 476: add plot below resulted in NullPointerException.
• 1567: das2server URIs fail to embed within .vap.zip file, other problems. Also see if a common file path can be calculated.
• rebin mishandled NaN data, resulting in weights dataset which was inconsistent with the data.
• 1278: das2server URIs fail to embed within .vap.zip file
• rte_0405096901: initialization of TickCurveRenderer caused NullPointerException
• rebin(ds,int,int) would fail under test data.
• rfe #453 underimplementation show both timerange and data set URI.
• 1566: improve support for international desktops, with UTF-8 filenames.
• send a separate graphics object to each renderer, so that sloppy renderers (like the orbit mode) don't affect others.
• das2stream cache tag was not propagated up to the QDataSet
• plot timerange GUI on axis tab goes insensitive when no one is listening.
• 1565: dataset selector use off the event thread.
• plotCommand with four parameters (index,x,y,z) would cause error
• reset the pointer cursor from BUSY to default when cancel is pressed in inspect-vap path
• reset the current folder for vap files when a vap is loaded.
• bugfix 1562: the trailing DD was mistaken for a unit. Thanks, Chris!
• bundle(ds1,ds2) never supported rank 0 bundled to rank 1 dataset.
• rte_0990623712: set the focus URI and support edit function on vap+internal dsf
• an orbits file with a comment would result in an unterminated loop. Thanks, Ivar!
• 1557: corrections to the SAMP listener. Thanks, Babtiste!
• csv reader properly sets column headings to field0..fieldN
• remove restriction that asynchronous (slow) filters cannot be used in the dataSourceFilter node, since there is no feedback of the problem.
• add experimental dep0 mismatch test when joining two TSBs with vap+internal:data_0,data_1.
• don't accept the click context (x,y) if no events were plotted
• events bar supports fill values for xtags.
• events bar supports rank 0 dataset.
• findbugs cleanup to get back down to 100.
• 1553: slice1 is dropped from cdaweb uris, components not shown for GSE_POS
• 1549: filter GUIs only fire events when focus is lost, or if they manually fire the event.
• 1545: render type fails to switch because of extraneous event firings from data panel.
• 1547: path reduction code would fail to connect dataset of two points with same X.
• version is identified as v2016a
• ISTP-CDF metadata node dataset/METADATA/GlobalAttributes is now shown as a map.
• When a TSB trims the data to the range because there is no X-axis, an extra point was trimmed from the end.
• Events bar should allow for units sloppiness when X values are Stevens' ratio scale units.
• don't assume that rank0 datasets will use digital renderer, they my use eventsBar.
• printing to PDF with a bottomDecorator would result in stderr chatter as it failed to update.
• 1542: after printing with background decorator, the cacheImageBounds would be incorrect but this was not resolved, getting stuck in a loop.
• 1509: popup windows below screen were not moved up sufficiently. Thanks Scott!
• 1541: jython code showed where looping over dom.dataSourceFilters could cause a concurrent modification problem. Thanks Larry!
• 1540: catch RTE from "bookmrks:" typo, when http or other valid protocol is not used. Thanks Bob!
• layout tab updates to current canvas within ~200 ms.
• axes tab plot timerange is now enabled when the app timerange is *not* the default.
• Larry pointed out his plots now show "slice1=1", which is because the loaded data set has a CONTEXT_0. Thanks Larry!
• boolean property types in das2streams are supported. Thanks Chris!
• vap+inline for Y(X) list was broken since vap+inline:4 now returns rank 0 dataset, not rank 1 dataset.
• rte_0110509791: when rank 1 dataset get into code to autorange spectrogram render type.
• rte_0123356543: runtime error when no records are returned from das2server Mars_Express/ASPERA/ELS. Thanks Ed!
• ImageDataSource now correctly extracts red and green components.
• 1536: CDF Themis virtual variables slice1=i doesn't work. Thanks Cindy!
• automatic completions-based GUI doesn't show "null" when label is not available for option. Show documentation.
• rte_1240063716: error: java.lang.String cannot be cast to org.virbo.dataset.QDataSet seen at Juno meeting. Thanks Rob!
• corrections to the AutoplotDataServer, where "NoDataInInterval" was not thrown properly in reduce.
• checkStatusLoop was not started in child applications, so the busy indicator used the old logic.
• bugfix where QStream had references to data that could not be resolved would prevent aggregation of JADE data.
• bugs when loading vap files with time series browse datasets that are not visible with a time axis.
• FilterChainPanel now limits the flashing of intermediate state by allowing 300ms to process before the color is changed.
• 1531: BufferDataSet append assumed compact data.
• swap needs to turn off TCAs. Experiment with fixLayout.
• 1530: support error bars in X, add control to turn off error bars with drawError=F. Thanks Lawrance!
• CDF where constraint works when data is rank 4 and filter data is rank 0.
• 1524: mashup handles when dataset loads fail. thumbnails are calculated more independently of one another.
• 1526: PDS/PPI didn't support quoted ISO8601 times
• annotation "point at" operation doesn't move the annotation, so "point at" and "move" are orthogonal operations.
• turn on timerange editor when vap+inline: is TimeSeriesBrowse
• events bar autoranging of one time datum should expand out to one hour duration
• the vertical slice "dump data set" now forces the .d2s extension so Autoplot can read this result.
• 1528: add an invisible plotElement when adding a hidden plot, so it can be turned on and provide overlays.
• vap+inline:3 now returns a rank 0 scalar.
• rte 1695465728: single-tab tear off on Mac would result in RTE.
• setDepend0Cadence would not properly set the units
• webstart on Linux/Mint creates a launcher that likes pass in %U when there is no file argument. -open xx (with any two characters) is now ignored.
• 1517: Sebastian's script hangs when calling DataPointRecorder.save
• 1515: remove extraneous user permissions dialogs caused when user@ added to it automatically.
• add PENDING_CHECK_REJECT so that the waitUntilIdle function properly detects that the application is not idle. Thanks Kenzie!
• improve feedback while reading CDF file data, MMS has large datasets that take non-trivial time to extract
• update the crosshair digitizer when we install RBGImageRenderer, so that R,G,B values are reported.
• treat http://user@... the same as http://... when credentials are needed either way. PNGWalkTool had a bug where user@ was added automatically, causing a second login dialog.

Features:

• PngWalkTool digitizer now includes column identifying the image.
• PngWalkTool provides mechanism for extension, where scripts can get events and can plug in small GUI.
• Jython plot command gets control for spectrogram color table. Thanks Babtiste and Kristoff!
• add control to set the maximum thread count, useful for Ivar's strange .jyds bug
• add plots dialog allows up to 6 by 6 plots now, useful for things like hourly plots of a day (6x4=24).

Less recent features:

• About low memory link on the About Autoplot page, when less than 700Mb of memory is detected.
• 381: pdb should be supported in jython editor. Loosen requirements for matching lines.
• APDataSet adapter for IDL and Matlab allows a filter process to be applied.
• Add ConsoleTextProgressMonitor, for examples and to support IDLMatlabBridge
• whiteBlueBlack colorbar added. Thanks, Masafumi!
• add convertUnitsTo for DatumRange and Datum. Thanks, Ivar!
• dataSetBuilder nextRecord with rank 1 data.
• smoothFit supports rank2 data to match the smooth function
• 360: Jython scripts have setOutputParam and requireAutoplotScriptingVersion, so that scripts can identify what they need.
• julianDay units added (days since noon, January 1, 4713 BC) to support Masafumi's data.
• 491: allow %{TIMERANGE,FORMAT=$Y$m$d} and %{TIMERANGE,FORMAT=$o}
• 389: buttons indicate the address bar or timerange selection.
• plot command gets lineWidth alias for lineThick, since this is a more consistent name.
• 482: Jython script save as allows the old name to be deleted, renaming the file.
• NetCDF/HDF reader now allows units to be specified in the URI, to support time tags with units=t1970, etc
• Jython supports Datum instances as PyDatums, supporting inline operators (like +)
• Auralizer code supports rank 2 waveform data
• 486: IDL/Matlab interface loadDataSet(uri) loads the URI with one command, returning status code.
• 485: run script shortcut key, experiment with F6.
• addPlotElement code added for scripting, to support overplots.
• X axis with %{CONTEXT} property will show DEPEND_0 context.
• Y axis with %{CONTEXT} property updates properly.
• screenshotsTool supports manual calls to get the screenshot, so that scripts can write documentation.
• jython popup documentation box size can be set.
• rank 3 flatten and rebinBundle routines added.
• kludge in controls like %{TIMERANGE,NOORBIT} for Ivar's study.
• FileUtil.find added which allows regular expressions to be used when looking for files.
• buckshotInterpolate introduced, adding support for 2-D and 3-D interpolations of buckshot data.
• ProGAL library added, which supports 2-D and 3-D tesselations.
• 24-bit wav files can now be read. 24-bit and 32-bit wav files can be exported. Thanks, Sarah!
• circle command now allows x and y positions
• update to SAMP 1.3.5, which is able to initialize a hub so javascript can talk to Autoplot.
• org.das2.util.FileUtil supports find file with regular expressions.
• Add setFillValue and setValidRange filters and GUIs
• Add flattenWaveform to the filters, which was needed for testing.
• Annotations support macros like %{USER_PROPERTIES.Name}
• DataSource nodes on layout tab now have a shiny blue icon and tooltip indicating Time Series Browse.
• DataSource focus can be set on the layout tab, allowing access to composite data components.
• fontSize property is more uniform for annotation, orbit plot, and digital plot
• interpolate supports rank 3 data.
• Image mode type now has control panel, and nearest neighbor rendering is provided.
• ImageDataSource now hints at the colorbar to use for red, green and blue components.
• Orbit mode now has control panel.
• Orbit mode allows tick label size to be controlled.
• Events list GUI is now two-column table. Thanks Chris for the suggestion!
• 479: consider if interpolateMod should allow range (e.g. -180 to 180).
• html table data source no longer requires a header row, first row will be interpretted as headers if no numbers are found.
• DataSetBuilder nextRecord method accepts array of doubles, as well as array of objects (datums)
• bug 1530: add control switch for error bars.
• support now-P1D/now+PT1H
• plug-in filters can now validate the string, allowing mousewheel on pipes string to be limited.
• BinaryDataSet supports "time21," fore where the ascii bytes of an ISO8601 time are found within binary PDS data.
• 476: control for data source filter's filter string is provided.
• linspace() supports rank 0 datasets.
• rank 4 random functions
• vap+inline: automatically supports timeSeriesBrowse
• events bar autoranging of one time datum should expand out to one hour duration
• add Units.nT.
• Autoplot servlet allows multiple colors to be specified for stack of traces.
• Data Mash up is now accessible from menus. Improvements to mashup including thumbnails.
• Autoplot servlet experiment with checking for exceptions in the plot and putting them in the response headers.
• 472: allow $x to be used in aggregation without time fields.

Known issues:

• see Source Forge

This completes the feature set for the new production branch. No major features will be added before branching this version. A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of autoplot.org will run this version. Feedback on the function and quality of this release is greatly appreciated!

Data is input from:

• ASCII files (.dat,.txt)
• CSV files (.csv)
• Binary files (.bin)
• CDF files(.cdf)
• Excel spread sheets (.xls)
• das2Streams (.d2s or .das2Stream)
• ViRBO's TSDS server
• CDAWeb data server (vap+cdaweb:)
• das2servers (vap+das2server:)
• QStreams (.qds), QDataSet on a das2Stream
• Plasma Wave Group das2 Server
• NetCDF files (.nc,.ncml)
• HDF5 files (.h5,.hdf5)
• DODs servers (.dds)
• Cluster Exchange Format files (.cef)
• Fits Format Files
• Images (.jpg, .png, .gif)
• Wav audio files (.wav)

This is not complete, and limitations include:

• Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
• Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.

autoplot.org is the wiki website, and contains more information than this launch page.