Autoplot Application (20150709b)

Funky polar spectrogram screenshot.

Autoplot attempts to allow plotting of data from many data sources, including local files and OpenDAP servers. Datasets are identified with URIs, and by the URI's extension a module is used to load the data into the internal data model. Data is then displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data.

This is the Autoplot2015a branch. This is an update to the production version containing bugfixes and minor new features. Java 7 is now required. Java 8 will be required sometime soon.

Single-jar version, autoplot.jar, can be run without network access and from the command line.

Bugfixes (2015-07-09 r17091):

• check to see that the bookmarks are loaded before assuming they had been, to support running scripts within 10 seconds of launch.
• total of total showed a bug where the weights were not properly accounted for.
• 1422: filters GUI will run filters on the event thread, hanging app for slow filters.
• old code for managing the tick formatter assumed that milliseconds would be sufficient.
• CollapseFilterEditor panel would loose the selected index when resetting the dimension names.
• add progress monitor to InlineDataSource
• das2server GUIs would drop intrinsic and item keywords.
• add missing sqrt and pow filters to FilterChain GUI.
• vap+inline: supports more complex URIs that contain getDataSet calls with ampersands within.
• ds1 + '5nT' will preserve the DEPEND_0 of ds1.
• SpectrogramRenderer would incorrectly guess the limit when NN rebinning was used, after new code was added earlier this Spring.
• USER_PROPERTIES copied over for Juno das2stream in odd branch.
• filters tool keeps alpha list and tree list synchronized to the same selection.
• 1347: finally make the result from reducex have uniform timetags.
• next/prev buttons check log property so that we avoid RTE with negative numbers.
• rte_2038937185: handle case where WWWAuthenticate is not provided by HTTP server.
• use "alphabetical" where "alpha" was used before. Brevity didn't help and could confuse.
• format to CDF with append multiple sets of timetags.
• Vertical slice on two-record spectrogram can now access the first record.
• 1415: plotting one day of data from a yearly aggregation. The exact URI timerange now appears.
• 1414: previous scan button needs to check for log property
• slices have properly hidden next/previous buttons
• 1408: make sure .vap file name is in address bar after loading.
• move getCompletions and sleep over from the application context to data source context. Thanks, Simon!
• 1408: put the script arguments on the address bar and in history. Thanks, George!
• label for atan2 function was acos, left over from cut-n-paste code. Thanks, Dave!
• rte_0484501685: don't allow format to XLS when the dataset is more than 65000 records, because it will crash. Thanks, Mary!
• rte_1830226672: check for valid name in reject method. Thanks acho!
• 1409: subsetting with new CDF library fails, with rank 1 DEPEND_1.
• rte_1765707187: export to file when filename is relative is allowed now. Thanks Ananya!
• rte_0874741227: nullPointerException with Jython scripts, thanks Bill and Dan!
• setDepend0Cadence supports rank 3 joins

Older Bugfixes:

• SpectrogramRenderer used code that ignored CADENCE property to judge the distance between measurements
• 1403: properly support brackets in filenames (rte1830227625)
• 1310: fix accidental java import with "import java.lang.Thread as _Thread instead of "import java.lang.Thread"
• 1404: fix occasional Jython GUI error caused by review GUI being created off of the event thread.
• 1393: WebStart caching has a bug where updates aren't seen, urlc.setUseCaches(false) fixes.
• 1310: keep track of scripts that have been approved by the operator, reducing tedium and enhancing security
• 1402: recent change in ascii URIs breaks editor when DEPEND_0 is not the zeroth column.
• append will preserve BufferDataSets when they are the same type.
• CDF editor only allows one item to be selected.
• 1349: 32-bit machines will use heap memory instead of accessing native memory.
• 1390: createEvent with the arguments createEvent( ds, DatumRange, color, label )
• 1390: eventds.slice(0).slice(0) now has correct units.
• 1399: minor das2server bugs such as +20 fixed. This may introduce new bugs...
• 1398: join of tables containing length 1 dataset would fail to limit interpolation length.
• 1398: xTagWidth property wouldn't be converted into QDataSet for MultiTableDataSet
• 478: list subset preserves DEPEND variable.
• 1395: out-of-native memory error when lots of data is loaded rapidly because of Java GC doesn't clear references.
• 1394: remove excessive traffic with the HTTP server, where GETs were done on the remote folder every ten seconds during aggregation
• 1394: expensiveOpCache for getting HTTP freshness didn't authenticate
• Empty Tools submenu will have "filters" added.
• TickCurveRenderer code that detected breaks initializes properly now.
• reset() method clears top and bottom decorators as well.
• top and bottom decorator access is thread-safe
• 1392: reducex GUI resets with "60s"
• Slice filter GUI now supports slice at time or Datum.
• 1383: double slash in history on Windows
• 1387: vertical slicer should keep labels
• context overview doesn't mess up the menus like it has done since 2008.
• 1385: VOTables contained lower case tr tags, where the parser assumed they would be upper case.
• 1381: rebin propagates the labels and units.
• 1380: correct the number of bins reported when DEPEND_2 is rank 2
• bugfix: correct place where blurTsbURI can return null.
• 1377: corrections and clarifications with inline editor panel and URIs.
• 1376: RTE caused by inline URI very hard to diagnose in script.
• QStreams with just one rank 2 dataset would flatten the QDataSet to rank 1 mistakenly.
• bugfix: screenshots tool would always grab from the first display.
• slice filter editor panel shouldn't use commas in the index.
• 1375: operations GUIs are incorrectly set as focus is switched.
• remove long operations from the droplist of recent operations. They mess up the layout of the GUI, and are probably difficult to interpret anyway.
• add round(n,1) to match Python round function.
• add more keyboard shortcuts and cheatsheet in Jython editor.
• allow single-letter variable name lookups in Jython editor.
• OpenDAP support would collapse rank 3 and rank 4 data, because early version of Autoplot only supported rank 2 data.
• switch to latest version of Java CDF library
• Autoplot server only accepts connections from localhost, and doesn't allow imports.
• correct branch where names were used instead of labels for x,y,z plots from das2server.
• kludge in update to force repaint when valid range and fill value fields are changed.
• rte_1630095413: clean up transition by clearing input and filter. Before the wrong dataset was sent to the filter.
• 1366 DasProgressPanel constructor should be protected. Jython scripts would sometimes not have a showing progress bar
• aggregation of two QStreams would fail because of change to when join datasets are flattened.
• 1362: memory leaks shown by Kenzie's FFT process.
• 1359: scan buttons get into funky state demonstration
• clarify smax<smin message in eventsRenderer
• bugfix when very small times were used and seconds was 0. with parse and format times.
• correct where $(subseconds,places=6) was ignoring the fractional part of the seconds field, and was just using the micros.
• 1359: trim() supports waveform dataset, which was causing scan to next to fail.
• unsigned bytes, shorts and ints in HDF5 files are corrected.
• minor performance enhancements for huge scatter renderer
• 1357: avoid nasty messages about immutable datasets, where CDAWeb plugin was modifying data that will soon be immutable.
• 1356: clean up code where the listener would add another listener.
• 1355: old renderer would paint new dataset, which could hang the event thread
• 1353: corrections to huge scatter where large waveform packets were preventing it from painting individual points.
• 1354: edit style properties now brings up custom editor.
• bugfix: single-tag dataset for ephemeris would result in RTE.
• HtmlTableDataSource name changes, improved support for dates.
• huge scatter properly implements flipped
• magic number check on CDF files would fail with old CDF files
• IndexGenDataSet is now final, more operations try to preserve internal types to save memory.
• CDF file units can now contain LaTeX code as well.
• 1350: Corrections to QStreams containing bundles.
• rte 2071287514: cancel dialog by closing window caused RTE. Check for other places where it is assumed at least one element is returned. Thanks Bob!

Features:

• add rank 4 ones and replicate functions.
• 1421: filters can now be cancelled.
• add fft and magnitude to the filters.
• intrinsic checkbox in das2server editor panel. Bold font used to improve readability.
• compile in new Dashup GUI for testing.
• in Jython, add datumRange method for coercing various object types into DatumRange.
• allow vap+jyds:http://... when the whitelist contains just http://...
• system property AUTOPLOT_DATA can be used to specify autoplot_data folder location.
• GUI title indicates if problemmatic 32bit Java is used.
• layout tab has add plots and delete plots button.

Less recent features:

• Chris added support for macros (substitutions) in das2streams.
• format to HTML supports rank 1 data.
• Add Tools->Filters... so new users needn't understand the data tab
• add grid to match flatten to filters GUI.
• macros support %{PROPERTIES.UNITS} %{PROPERTIES.CACHE_TAG}
• add separate logger das2.url that logs all external connections.
• add (µs) as a unit.
• UTC is allowed where units are found, so that times can be explicitly parsed as such.
• plotx color keyword parameter can now be a float, int, rank 0 dataset, or Color.
• 426: extract macro converter logic, and make plot title and plotElement label macros consistent.
• 426: support macros for METADATA and USER_PROPERTIES, like %{METADATA.GlobalAttributes.Mission_group}
• putProperty( ds, QDataSet.USER_PROPERTIES, PyDict )
• stairSteps mode doesn't show dot plot symbols, to better support histograms.
• add more shortcuts to jython editor, and screen showing shortcuts.
• applot.py matured and improves support for SciPy.
• Autoplot Server port only listens to localhost, and does not allow imports.
• update RBSP orbits to 2015-05-01
• finish off applot.py support for Autoplot in native SciPy applications, using Autoplot server.
• export to CDF supports UINT1, UINT2, and UINT4 data types.
• add more examples to InlineDataSource editor panel, to add colored events bar.
• add PutPropertyFilterEditorPanel, which allows valid ranges and missing metadata to be specified
• introduce DatumRangeUtil.parseDatumRange(String) which detects ISO8601 times or units equal to UTC.
• automatic hang detection writes RTE to autoplot_data/log/ when app hangs for 10 seconds.
• added BindingSupport.printStatus() to help find memory leaks caused by bindings
• get rid of extraneous InterruptedExceptions which were thrown from the bowels, and needn't be propagated to the more abstract waitUntilIdle
• renderer style panels for Contour, PitchAngleDistribution, and EventsBar styles.
• more dasCore code cleanup

Known issues:

• see Source Forge

This completes the feature set for the new production branch. No major features will be added before branching this version. A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of autoplot.org will run this version. Feedback on the function and quality of this release is greatly appreciated!

Data is input from:

• ASCII files (.dat,.txt)
• CSV files (.csv)
• Binary files (.bin)
• CDF files(.cdf)
• Excel spread sheets (.xls)
• das2Streams (.d2s or .das2Stream)
• ViRBO's TSDS server
• CDAWeb data server (vap+cdaweb:)
• das2servers (vap+das2server:)
• QStreams (.qds), QDataSet on a das2Stream
• Plasma Wave Group das2 Server
• NetCDF files (.nc,.ncml)
• HDF5 files (.h5,.hdf5)
• DODs servers (.dds)
• Cluster Exchange Format files (.cef)
• Fits Format Files
• Images (.jpg, .png, .gif)
• Wav audio files (.wav)

This is not complete, and limitations include:

• Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
• Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.

autoplot.org is the wiki website, and contains more information than this launch page.