Autoplot Application (20151017a)

The new annotations feature set is done. Here a static image (left) is used to annotate the figure, and the text annotation (right) is being modified.

Autoplot attempts to allow plotting of data from many data sources, including local files and OpenDAP servers. Datasets are identified with URIs, and by the URI's extension a module is used to load the data into the internal data model. Data is then displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data.

This is the Autoplot2015a branch. This is an update to the production version containing bugfixes and minor new features. Java 7 is now required. Java 8 will be required sometime soon.

Single-jar version, autoplot.jar, can be run without network access and from the command line.

Bugfixes (2015-10-16 r17722):

• 1473: create pngwalk by orbit without rescale fails
• 1472: HTTPFileSystem fails to query for username after typo
• 1470: de-internationalize the three codes that were accidentally internationalized.
• 1468: Chris pointed out the setting for the default browser in the settings, which pointed out that Java handles this now, since Java6.
• 1465: check before closing an editor with changes.
• invokeScriptSoon dialog would leave progress monitor hanging if there's a exception thrown in the script..
• remove extra question mark when reporting data was exported.
• corrections to hrinterval and time format, correctly using the stop time and supporting end modifier.
• clean up address bar as vap is loaded.
• "2014-001 to 2015-001 UTC" is supported now.
• 1462: result of guessCadence mishandled in hugeScatter rendering mode
• 1463: null value in USER_PROPERTIES would result in NullPointerException
• 1440: vap+inline:ripp<C> works now.
• null value in USER_PROPERTIES would result in NullPointerException
• Export to CDF didn't properly support append keyword.
• HugeScatter wouldn't to paint the exception if that's why it didn't have a dataset
• vap+inline each line needs to be URL-decoded. Add monitor so downloadResourceAsTempFile can be used.
• if the first filter was an implicit unbundle, then it Filters GUI wouldn't update properly
• FFTPower GUI supports slide of 8 (7/8 overlap)
• use case where GCD didn't work to calculate the needed cadence for ephemeris ticks, fall back to difference
• CACHE_TAG was getting through the subset operator, causing strange caching problems.
• copyTo needs to reset the limit of the new buffer, I believe because of append.
• handle case where there is a CONTEXT_0 with Units.us2000 and DEPEND_0 with Units.seconds in fftFilter.
• 1458: valid function would return result with fill_value of its input for reference, but this could be zero, which is incorrect. SUGGEST_FILL property added.
• properly size the x-axis when running headless jobs with TimeSeriesBrowse-capable datasets.
• properly format the leap second with 23:59:60
• support for huge (>2GB) CDF files, switch to 20150828 CDF library version
• improvements to Huge Scatter mode allow 67 million points to be plotted interactively (somewhat).
• sloppy code in spinner progress bar be left behind when loading ephemeris.
• refactor ascii parser to make it more consistent.
• screenshots tool includes Windows as well as Frames when masking for privacy.
• 1453: AggregatingDataSource now uses the time constraint when identifying the delegate file for retrieving metadata, avoiding extraneous load.
• 1448: allow capitalized extentions like ".PNG" when explicit filenames are used.
• 1433: double check on timetags was too restrictive and throwing out too much data from RBSP waveform data, George pointed out.
• 2015 leap second was applied 64 seconds too early.
• Ephemeris ticks now use resolution when tick labels are sourced from TimeSeriesBrowse.
• PNGWalk "View in Autoplot" button will look at PNG metadata to extract the timerange.
• More zaxis properties are loaded in das2streams.
• 1165: create pngwalk uses user prefs for grid and other options.
• xls files would drop the last record.
• xls files would have times shifted by local time zone.
• null pointer when no color was selected in pick colors dialog on Linux.
• rank 3 data in QStream wasn't properly read in.
• Autoplot dialogs like the "additional operators" don't have goofy default question mark icons.
• output of contour hard to use, breaks in non-monotonic data. Thanks Sebastian!
• rte 2091983907: thread safety in pngwalk. Thanks usuario!
• rte_1604359977: limit the total number of filePollUpdates threads that can be started to 10000. Thanks, Bob!
• completions within a function call are corrected. plot( ripples<C> ) corrected.
• 1439: add "delete selected items" to edit menu of DataPointRecorder
• Sebastian demonstrated that QStream with DEPEND_0 having DEPEND_0 would cause failure.
• Sebastian demonstrated that embed data failed when the URI was ''.
• Kristoff demonstrated that embed data doesn't work when the file came via http.
• 1442-6: camelCase for the plotx command.
• 1442-7 symbolSize instead of symSize
• 1442-10: height and width boxes on mac
• resizeForCanvasSize didn't consider that the canvas may already be disconnected from the application
• 1433: Kristoff pointed out that when records contain fill or are not long enough for the interval, then the ytags would be left with a different length.
• bugfixes with rich ascii.
• 1088: clip off the extra stuff when we do completions in the middle of a parameter.
• tolerate mixed case DISPLAY_TYPE metadata in CDF files.
• 1434: metadata propagation through operators is inconsistent, inconvenient
• 1433: allow waveform cadence to change in fftPower.
• 1432: list of list to declare double arrays (and rank 2 datasets) in Jython.
• 1431: "Cannot create PyString from null" popup in editor.
• Ops.labels renamed to Ops.labelsDataset, to avoid name clashes.
• completions within command, like plotx(<C>) where we need to look up the documentation for plotx.
• more support for color argument in plotx
• plot and plotx command are finally one and the same.
• check for null in setSymbolConnector, setColor, setFillColor.
• cdfn and cdfj loggers combined to "cdf" since there's just one cdf source.
• bug where cdf editor had no visible variables because of bugs with CDF editor (Thanks Mark and Matt).
• bug where labels were array [N] instead of [1,N] corrected.
• ds[rank0ds] handled in Jython
• flattenRank2 handles rank 1 DEPEND_1.
• correction where FFT window was applied twice corrected.
• qds supports BIN_PLUS and BIN_MINUS properties, WEIGHTS.
• verify the qstream packet length is less than 1000000, to aid in debugging.
• warning icon persists, where it was cleared mistakenly before.
• scriptContext formatDataSet allows progress monitor.
• imports are allowed with server port again.
• updates to many of the example scripts.
• bugfixes in block from 2015-07-15 to 2015-07-26
• FFTPower would apply the window twice, affecting the data. Thanks, Chris!

Older Bugfixes:

• support where=IsoDateTime.within(2012-11-01T03:00/09:00) in ascii plugin.
• minor tweaks to property editor
• update RBSP orbits to 2827
• hide the "filters" name which is meaningless to the scientists, and use "operations" (which is just as meaningless, but there's one fewer name).
• check to see that the bookmarks are loaded before assuming they had been, to support running scripts within 10 seconds of launch.
• total of total showed a bug where the weights were not properly accounted for.
• 1422: filters GUI would run filters on the event thread, hanging app for slow filters.
• old code for managing the tick formatter assumed that milliseconds would be sufficient.
• CollapseFilterEditor panel would loose the selected index when resetting the dimension names.
• add progress monitor to InlineDataSource
• das2server GUIs would drop intrinsic and item keywords.
• add missing sqrt and pow filters to FilterChain GUI.
• vap+inline: supports more complex URIs that contain getDataSet calls with ampersands within.
• ds1 + '5nT' will preserve the DEPEND_0 of ds1.
• SpectrogramRenderer would incorrectly guess the limit when NN rebinning was used, after new code was added earlier this Spring.
• USER_PROPERTIES copied over for Juno das2stream in odd branch.
• filters tool keeps alpha list and tree list synchronized to the same selection.
• 1347: finally make the result from reducex have uniform timetags.
• next/prev buttons check log property so that we avoid RTE with negative numbers.
• rte_2038937185: handle case where WWWAuthenticate is not provided by HTTP server.
• use "alphabetical" where "alpha" was used before. Brevity didn't help and could confuse.
• format to CDF with append multiple sets of timetags.
• Vertical slice on two-record spectrogram can now access the first record.
• 1415: plotting one day of data from a yearly aggregation. The exact URI timerange now appears.
• 1414: previous scan button needs to check for log property
• slices have properly hidden next/previous buttons
• 1408: make sure .vap file name is in address bar after loading.
• move getCompletions and sleep over from the application context to data source context. Thanks, Simon!
• 1408: put the script arguments on the address bar and in history. Thanks, George!
• label for atan2 function was acos, left over from cut-n-paste code. Thanks, Dave!
• rte_0484501685: don't allow format to XLS when the dataset is more than 65000 records, because it will crash. Thanks, Mary!
• rte_1830226672: check for valid name in reject method. Thanks acho!
• 1409: subsetting with new CDF library fails, with rank 1 DEPEND_1.
• rte_1765707187: export to file when filename is relative is allowed now. Thanks Ananya!
• rte_0874741227: nullPointerException with Jython scripts, thanks Bill and Dan!
• setDepend0Cadence supports rank 3 joins

Features:

• rfe453: axis tab has timerange editor for when the timerange isn't an axis.
• rfe454: support arbitrary rank data with HDF5 plugin.
• pngwalk allows &timerange= in URI to constrain the image timeranges.
• 451: keep track of running script so buffer doesn't swap
• 345: first go at annotations
• improvements to $(enum) in timeranges, providing code to read out the enumerated values.
• 449: bitwise operators in QDataSets.
• export to HDF supports append
• trim operator has GUI now. Thanks Armond!
• ascii formatter now allows header=none and doDep=F, depend0Units=seconds+since+1970-01-01 to support import to Matlab and IDL.
• ascii parser depend0Units=seconds+since+1970-01-01
• add ctrl+0 to reset the font.
• 446: add binsWithin function, motivated by Ivar's script.
• ds[rand()*100]= 1 now works.
• plot command [xy]drawTickLabels=False added
• 445: speed enhancements for lanlNNRebinner, experiments with extent
• setWindowLocation command. commands for multiple windows in scripts.
• 441: allow non-time aggregations like data_$x.dat
• 444: data source editors may have list of reasons why the URI was rejected
• 443: autoRangingHints property allows more control over autoranging (https://sourceforge.net/p/autoplot/feature-requests/443/)
• axes have dialog supporting units change to convertible units
• 442: special support for MMS data website, which has uses login and cookies
• Add native memory available to the "About Autoplot" dialog.
• http://autoplot.org//developer.datasetToString clean up
• .pngwalk file has filepattern to support more pngwalks and subdirectories
• URI about:classpath added for debugging
• 441: allow $v and $x to trigger the split index.
• Introduce "flaky server" code to model slow servers so Autoplot can deal with them more effectively.
• 1448: explicit control of output filenames in pngwalk.
• where keyword support in HDF5
• HDF5 now has an editor panel.
• improvements to completions in Jython editor panel.
• plot named parameters have consistent capitalization and index named parameter introduced.
• embed data found in data sources that does not have a URI ("vap+internal:")
• opposite property for axes.
• support $x/$Y_$m for file storage models
• rfe 439: Need standard method for getting a second Autoplot app in a script.
• avoid filesystem completions with inline, cdaweb, or pdsppi.
• completions are working somewhat with vap+inline.
• within parameter added to CDF
• plotx command legendLabel named parameter added.
• plotx linestyle added.
• improvements to DashupTool.
• update the port reported when the server is on, to support automatic selection of port with --server=0.
• support Units.RGBColor for the units of the colorBy dataset.

Less recent features:

• improvements to feedback in CDF parameter editor panel.
• 1425: implicit events file format allows any text file with ISO8601, ISO8601 for the first two columns to be implicitly treated as an events file.
• add rank 4 ones and replicate functions.
• 1421: filters can now be cancelled.
• add fft and magnitude to the filters.
• intrinsic checkbox in das2server editor panel. Bold font used to improve readability.
• compile in new Dashup GUI for testing.
• in Jython, add datumRange method for coercing various object types into DatumRange.
• allow vap+jyds:http://... when the whitelist contains just http://...
• system property AUTOPLOT_DATA can be used to specify autoplot_data folder location.
• GUI title indicates if problematic 32bit Java is used.
• layout tab has add plots and delete plots button.

Known issues:

• see Source Forge

This completes the feature set for the new production branch. No major features will be added before branching this version. A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of autoplot.org will run this version. Feedback on the function and quality of this release is greatly appreciated!

Data is input from:

• ASCII files (.dat,.txt)
• CSV files (.csv)
• Binary files (.bin)
• CDF files(.cdf)
• Excel spread sheets (.xls)
• das2Streams (.d2s or .das2Stream)
• ViRBO's TSDS server
• CDAWeb data server (vap+cdaweb:)
• das2servers (vap+das2server:)
• QStreams (.qds), QDataSet on a das2Stream
• Plasma Wave Group das2 Server
• NetCDF files (.nc,.ncml)
• HDF5 files (.h5,.hdf5)
• DODs servers (.dds)
• Cluster Exchange Format files (.cef)
• Fits Format Files
• Images (.jpg, .png, .gif)
• Wav audio files (.wav)

This is not complete, and limitations include:

• Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
• Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.

autoplot.org is the wiki website, and contains more information than this launch page.