Autoplot Application (20160428c)

Code cleanup of URI handling allows characters like ß to be used.

Autoplot attempts to allow plotting of data from many data sources, including remote CDF files and OpenDAP servers. Datasets are identified with URIs, and by the URI's extension, data are loaded into the internal data model. Data are then displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data. Data can then be additionally processed with other operations like smooth and FFT, and Jython scripting provides a capable programming environment.

This is the Autoplot2016a branch. This is an update to the production version containing bugfixes and minor new features. Java 7 is now required. Java 8 will be required sometime soon.

Single-jar version, autoplot.jar, can be run without network access and from the command line.

Experimental Mac Release is available.

Bugfixes (2016-04-28 r18801):

• When adapting legacy VectorDataSets to QDataSet, name the default plane "y".
• 1576: Masafumi pointed out extraneous load when undo was pressed.
• das2stream containing new unit would fail to read, whoops!
• bundleNames function would fail to add element when it found an unnamed dataset, now returns "ch_i"
• Jython "recently used files" area shows both .jy and .jyds files, so you don't have to think about the current context.
• 1570: make sure the axis timerange is returned in %{TIMERANGE} macro, not the loaded timerange.
• corrections to triangulation routines. Scale data to be around one before tesselation.
• 476: add plot below resulted in NullPointerException.
• 1567: das2server URIs fail to embed within .vap.zip file, other problems. Also see if a common file path can be calculated..
• rebin mishandled NaN data, resulting in weights dataset which was inconsistent with the data.
• 1278: das2server URIs fail to embed within .vap.zip file
• rte_0405096901: initialization of TickCurveRenderer caused NullPointerException
• rebin(ds,int,int) would fail under test data.
• rfe #453 underimplementation show both timerange and data set URI.
• 1566: improve support for international desktops, with UTF-8 filenames.
• send a separate graphics object to each renderer, so that sloppy renderers (like the orbit mode) don't affect others.
• das2stream cache tag was not propagated up to the QDataSet
• plot timerange GUI on axis tab goes insensitive when no one is listening.
• 1565: dataset selector use off the event thread.
• plotCommand with four parameters (index,x,y,z) would cause error
• reset the pointer cursor from BUSY to default when cancel is pressed in inspect-vap path
• reset the current folder for vap files when a vap is loaded.
• bugfix 1562: the trailing DD was mistaken for a unit. Thanks, Chris!
• bundle(ds1,ds2) never supported rank 0 bundled to rank 1 dataset.
• rte_0990623712: set the focus URI and support edit function on vap+internal dsf
• an orbits file with a comment would result in an unterminated loop. Thanks, Ivar!
• 1557: corrections to the SAMP listener. Thanks, Babtiste!
• csv reader properly sets column headings to field0..fieldN
• remove restriction that asynchronous (slow) filters cannot be used in the dataSourceFilter node, since there is no feedback of the problem.
• add experimental dep0 mismatch test when joining two TSBs with vap+internal:data_0,data_1.
• don't accept the click context (x,y) if no events were plotted
• events bar supports fill values for xtags.
• events bar supports rank 0 dataset.
• findbugs cleanup to get back down to 100.
• 1553: slice1 is dropped from cdaweb uris, components not shown for GSE_POS
• 1549: filter GUIs only fire events when focus is lost, or if they manually fire the event.
• 1545: render type fails to switch because of extraneous event firings from data panel.
• 1547: path reduction code would fail to connect dataset of two points with same X.
• version is identified as v2016a
• ISTP-CDF metadata node dataset/METADATA/GlobalAttributes is now shown as a map.
• When a TSB trims the data to the range because there is no X-axis, an extra point was trimmed from the end.
• Events bar should allow for units sloppiness when X values are Stevens' ratio scale units.
• don't assume that rank0 datasets will use digital renderer, they my use eventsBar.
• printing to PDF with a bottomDecorator would result in stderr chatter as it failed to update.
• 1542: after printing with background decorator, the cacheImageBounds would be incorrect but this was not resolved, getting stuck in a loop.
• 1509: popup windows below screen were not moved up sufficiently. Thanks Scott!
• 1541: jython code showed where looping over dom.dataSourceFilters could cause a concurrent modification problem. Thanks Larry!
• 1540: catch RTE from "bookmrks:" typo, when http or other valid protocol is not used. Thanks Bob!
• layout tab updates to current canvas within ~200 ms.
• axes tab plot timerange is now enabled when the app timerange is *not* the default.
• Larry pointed out his plots now show "slice1=1", which is because the loaded data set has a CONTEXT_0. Thanks Larry!
• boolean property types in das2streams are supported. Thanks Chris!
• vap+inline for Y(X) list was broken since vap+inline:4 now returns rank 0 dataset, not rank 1 dataset.
• rte_0110509791: when rank 1 dataset get into code to autorange spectrogram render type.
• rte_0123356543: runtime error when no records are returned from das2server Mars_Express/ASPERA/ELS. Thanks Ed!
• ImageDataSource now correctly extracts red and green components.
• 1536: CDF Themis virtual variables slice1=i doesn't work. Thanks Cindy!
• automatic completions-based GUI doesn't show "null" when label is not available for option. Show documentation.
• rte_1240063716: error: java.lang.String cannot be cast to org.virbo.dataset.QDataSet seen at Juno meeting. Thanks Rob!
• corrections to the AutoplotDataServer, where "NoDataInInterval" was not thrown properly in reduce.
• checkStatusLoop was not started in child applications, so the busy indicator used the old logic.
• bugfix where QStream had references to data that could not be resolved would prevent aggregation of JADE data.
• bugs when loading vap files with time series browse datasets that are not visible with a time axis.
• FilterChainPanel now limits the flashing of intermediate state by allowing 300ms to process before the color is changed.
• 1531: BufferDataSet append assumed compact data.
• swap needs to turn off TCAs. Experiment with fixLayout.
• 1530: support error bars in X, add control to turn off error bars with drawError=F. Thanks Lawrance!
• CDF where constraint works when data is rank 4 and filter data is rank 0.
• 1524: mashup handles when dataset loads fail. thumbnails are calculated more independently of one another.
• 1526: PDS/PPI didn't support quoted ISO8601 times
• annotation "point at" operation doesn't move the annotation, so "point at" and "move" are orthogonal operations.
• turn on timerange editor when vap+inline: is TimeSeriesBrowse
• events bar autoranging of one time datum should expand out to one hour duration
• the vertical slice "dump data set" now forces the .d2s extension so Autoplot can read this result.
• 1528: add an invisible plotElement when adding a hidden plot, so it can be turned on and provide overlays.
• vap+inline:3 now returns a rank 0 scalar.
• rte 1695465728: single-tab tear off on Mac would result in RTE.
• setDepend0Cadence would not properly set the units
• webstart on Linux/Mint creates a launcher that likes pass in %U when there is no file argument. -open xx (with any two characters) is now ignored.
• 1517: Sebastian's script hangs when calling DataPointRecorder.save
• 1515: remove extraneous user permissions dialogs caused when user@ added to it automatically.
• add PENDING_CHECK_REJECT so that the waitUntilIdle function properly detects that the application is not idle. Thanks Kenzie!
• improve feedback while reading CDF file data, MMS has large datasets that take non-trivial time to extract
• update the crosshair digitizer when we install RBGImageRenderer, so that R,G,B values are reported.
• treat http://user@... the same as http://... when credentials are needed either way. PNGWalkTool had a bug where user@ was added automatically, causing a second login dialog.

Older Bugfixes:

• openBrowser code assumed that the Java 6 system could be used, but it isn't supported on Linux (Mint).
• 1512: I/O Error reading CDF files on 32 bit machines. Fall back to alternate method for opening files.
• RGB indication on crosshair renderer corrected for images without DEPEND_1 tags.
• 1511: resetting the data set URI in a stack of plots would reset the timerange. Thanks, Bill, for pointing this out!
• relax timeout on read to 60 seconds, because sometimes the das2server would take more 10 seconds before sending out data.
• 1510: pngwalks of aggregations of sparse data, limitation is no longer needed, and we just don't look for gaps when this would be impractical.
• correction to PNGWalkTool thread sync bug, where check failed to see that thumbnail wasn't ready.
• keep track of separate control for HTTP read timeout, before there was just timeout for connection.
• pngwalk write to html corrected to make pngwalk more portable.
• 1506: allow subsequent repeat of lsd to reset the multiplier, fixing $Y$(m,delta=4)/$Y$m$d.dat
• grayPeaks didn't draw in StackedHistogram mode, because the lightGray.brighter() would result in white.
• undo operation was performed on event thread, which could easily lock the application
• CDAWeb result was missing CACHE_TAG, so result would be reloaded when it wasn't needed.
• CDAWeb uses web service to more accurately identify needed files.
• 1509: do sanity check to make sure new dialogs are not positioned outside of screen bounds.
• export to rich ascii will allocate name when dataset doesn't have name property.
• Screenshots tool would clip the pointer when it was partially outside of the GUI boundaries.
• 1437: export to rich ascii must preserve as much metadata as possible
• check for missing/problem with Rich PNG JSON, Where Dan's pngwalk contained bad metadata that showed in a demo at AGU.
• dashup tool gets quick-n-dirty rename variable code
• screenshots tool would clip the pointer when it was on the edge of window
• time unit t2000 works in das2stream again.
• clean up DataPointRecorder's use of threads, to avoid RTE.
• 1500: two autoplot script editors, pointing at same file, detect changes incorrectly
• 1498: pngwalk digitizer jumps ahead when delta=15 should have been used
• add typographicPoints unit, and correct meters to inches conversion, which was using inverse.
• bugfix: offline status wasn't respected in one code, causing hang.
• Netbeans is able to compile dasCore again, the project metadata was messed up.
• set the bounds properly when fontSize is used to enlarge font. Thanks Rich!
• add code to ensure that ephemeris data is monotonic and non-repeating.
• 1495: detect when maximized state should be used when restoring .vap files
• minor improvements to status bar messages
• 466: copy-n-paste plots. Feature now does error checking
• Updates to Rich ASCII parser to support Firebird files.
• "getParam (" wasn't parsed properly when generating GUI. Thanks Rich!
• Aggregation listings constrained to timerange, when timerange is in the URI.
• das2stream supports yFormat property
• annotations now visible in pngwalks and with the server.
• qds formatter also handles BIN_MINUS and BIN_PLUS
• corrections to autorange hints
• annotation bounds were not set properly for pngwalks and with the servlet.
• formatDataSet to CDF on Windows with Webstart failed because of move, so we fall back to copy.
• rte 1586184923: user exceptions were confusing because RTE dialog showed
• try/finally block used with web communication to get mime type.
• 1486: minor corrections to layout tab to improve usability
• corrections to Dashup tool caused by minus sign vs negation in Jython AST.
• "and" and "or" must be implemented as inline operators in Dashup tool
• introduce rank0 transform methods to support Jython rank 1 iterator returns rank 0 datasets instead of doubles, breaking Juno/Waves planning script.
• rfe 457: add hidden plot would intercept mouse events, making the connected plots difficult to work with.
• rfe 464: update the initial colors.txt template so that it is not confusing.

Features:

• X axis with %{CONTEXT} property will show DEPEND_0 context.
• Y axis with %{CONTEXT} property updates properly.
• screenshotsTool supports manual calls to get the screenshot, so that scripts can write documentation.
• jython popup documentation box size can be set.
• rank 3 flatten and rebinBundle routines added.
• kludge in controls like %{TIMERANGE,NOORBIT} for Ivar's study.
• FileUtil.find added which allows regular expressions to be used when looking for files.
• buckshotInterpolate introduced, adding support for 2-D and 3-D interpolations of buckshot data.
• ProGAL library added, which supports 2-D and 3-D tesselations.
• 24-bit wav files can now be read. 24-bit and 32-bit wav files can be exported. Thanks, Sarah!
• circle command now allows x and y positions
• update to SAMP 1.3.5, which is able to initialize a hub so javascript can talk to Autoplot.
• org.das2.util.FileUtil supports find file with regular expressions.
• Add setFillValue and setValidRange filters and GUIs
• Add flattenWaveform to the filters, which was needed for testing.
• Annotations support macros like %{USER_PROPERTIES.Name}
• DataSource nodes on layout tab now have a shiny blue icon and tooltip indicating Time Series Browse.
• DataSource focus can be set on the layout tab, allowing access to composite data components.
• fontSize property is more uniform for annotation, orbit plot, and digital plot
• interpolate supports rank 3 data.
• Image mode type now has control panel, and nearest neighbor rendering is provided.
• ImageDataSource now hints at the colorbar to use for red, green and blue components.
• Orbit mode now has control panel.
• Orbit mode allows tick label size to be controlled.
• Events list GUI is now two-column table. Thanks Chris for the suggestion!
• 479: consider if interpolateMod should allow range (e.g. -180 to 180).
• html table data source no longer requires a header row, first row will be interpretted as headers if no numbers are found.
• DataSetBuilder nextRecord method accepts array of doubles, as well as array of objects (datums)
• 1530: add control switch for error bars.
• support now-P1D/now+PT1H
• plug-in filters can now validate the string, allowing mousewheel on pipes string to be limited.
• BinaryDataSet supports "time21," fore where the ascii bytes of an ISO8601 time are found within binary PDS data.
• 476: control for data source filter's filter string is provided.
• linspace() supports rank 0 datasets.
• rank 4 random functions
• vap+inline: automatically supports timeSeriesBrowse
• events bar autoranging of one time datum should expand out to one hour duration
• add Units.nT.
• Autoplot servlet allows multiple colors to be specified for stack of traces.
• Data Mash up is now accessible from menus. Improvements to mashup including thumbnails.
• Autoplot servlet experiment with checking for exceptions in the plot and putting them in the response headers.
• 472: allow $x to be used in aggregation without time fields.

Less recent features:

• About screen includes a bullet explicitly identifying 32bit Java and provides a link to fix this.
• html version of pngwalk uses precalculated thumbnails.
• 470: DasProgressPanel shows the thread name to improve serviceability
• CDAWeb now shows availability.
• StackedHistogram gets style panel
• StackedHistogramRenderer name fixes
• CDF editor dialog indicates internal type to improve transparency and support creating new files.
• 468: unplot can use iso8601 times.
• Canvas size GUI supports "7 inch" or "30 centimeters" for dimensions, not just pixels.
• add LSpec rebin that uses bundle of buckshot (T,L,Z) for the input.
• 467: add accessory to allow override timerange when loading vap files.
• das2server items can now be a comma-delimited list of indeces.
• Single filesystems can be reset now, double-clicking in FileSystem manager will trigger reset.
• format to QDS supports BIN_PLUS and BIN_MINUS properies
• 443: autorange hint log=T supported, corrections to initial implementation of center and includeZero
• 466: copy-n-paste plots.
• script editor allows entry into data source editors with right-click→actions→"inspect URI"
• ISO8601 parser now supports time zone offsets.
• Clean-up and logging of OpenDap code so more predicatable failures can be studied.
• nc4 extension is recognized as NetCDF.

Known issues:

• see Source Forge

This completes the feature set for the new production branch. No major features will be added before branching this version. A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of autoplot.org will run this version. Feedback on the function and quality of this release is greatly appreciated!

Data is input from:

• ASCII files (.dat,.txt)
• CSV files (.csv)
• Binary files (.bin)
• CDF files(.cdf)
• Excel spread sheets (.xls)
• das2Streams (.d2s or .das2Stream)
• ViRBO's TSDS server
• CDAWeb data server (vap+cdaweb:)
• das2servers (vap+das2server:)
• QStreams (.qds), QDataSet on a das2Stream
• Plasma Wave Group das2 Server
• NetCDF files (.nc,.ncml)
• HDF5 files (.h5,.hdf5)
• DODs servers (.dds)
• Cluster Exchange Format files (.cef)
• Fits Format Files
• Images (.jpg, .png, .gif)
• Wav audio files (.wav)

This is not complete, and limitations include:

• Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
• Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.

autoplot.org is the wiki website, and contains more information than this launch page.