Autoplot Application (20170320c)

Data Mash Up and HAPI server used for quick study of house temperatures. Happy Spring!

Autoplot attempts to allow plotting of data from many data sources, including remote CDF files and OpenDAP servers. Datasets are identified with URIs, and by the URI's extension, data are loaded into the internal data model. Data are then displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data. Data can then be additionally processed with other operations like smooth and FFT, and Jython scripting provides a capable programming environment.

This is the Autoplot2016a branch. This is an update to the production version containing bugfixes and minor new features. Java 7 is now required. Java 8 will be required sometime soon.

Single-jar version, autoplot.jar, can be run without network access and from the command line.

Mac DMG Release is available.

Bugfixes (2017-03-21 r20555):

• PDS/PPI uses https now. Editor properly sets the parameter when returning to the editor GUI.
• downloadResourceAsTempFile now includes http or https (or ftp) in generated filename.
• trimStrideWrapper dropped metadata.
• Units.getAllUnits would return a list of strings when its declaration indicated that it would return a list of Units. Thanks, Jon!
• RTE reports (and provide feedback) must now submit to jfaden.net:80, not jfaden.net:8080, because of firewall issues.
• 1794: correction to branch when github is accessed.
• some spectrogram and color scatter assumed data and colorbar zunits were the same.
• https://bugs-pw.physics.uiowa.edu/mantis/view.php?id=478: ascii time transfer types could not have differing units.
• test100 showed where an immutable dataset was modified with the "hanning" filter.
• &y=bgsm[:,1] was unnecessarily URL-encoded, reducing readability.
• where parameters list was incorrect when the HDF5 file contained structures
• link(x,y,z) dropped z, when x y or z is an object (not dataset).
• issue a warning when das2stream units change, which can cause problems.
• where with HDF5 files containing parameters with dots in the name was never tested.
• where with HDF5 files containing structures was never tested.
• csv editor panel has selection for delimiter.
• corrections to HAPI reader, which was broken after rank 2 DEPEND_1 support was added.
• addDataPoint.jy script updated to use DataPointRecorder (not DataPointRecorderNew), and supports keystroke mode.
• log "ImageIO used to create" at less than info, to avoid clutter in server logs.
• rte_1490216564: don't throw runtime error when taskSize is zero, just ignore it.
• 1785: unable to format bundle to CDF. (Append is now handled properly.) Thanks, Kristoff!
• 1789: PngWalkTool correct feedback about writing to writable fs. The code didn't distinguish between Read-Only and Read-Write, and assumed that local filesystems were read-write. Thanks, Sebastian!
• rte_0147649225: fall back after failing to find "field6" in bundleDescriptor, because HAPI metadata was mistaken for Rich ASCII.
• rte 0857617456: Null pointer when selecting time field type but time format column has never been set. Thanks, Shiva!
• corrections to format to .wav gui, which wasn't initialized properly to indicate automatic scaling.
• digitizer DataPointRecorder "new" operation fires dataset update event. Thanks, Kris!
• bugfix: vap+cdaweb:ds=IM_K0_SIE&id=SI_PIXELS had image>THUMBSIZE>50 for the RENDER_TYPE, which Autoplot doesn't support as "image"
• Experimental support for rank 2 DEPEND_1 datasets with HAPI. HAPI uses sampleStartDate for discovery.
• check for vap+internal:data_1,data_2 when switching to mash-up
• 1754: Correctly support http://autoplot.org/data/AMSR_E_L3_SeaIce6km_B06_20070307.hdf.gz
• 1754: Java now reuses https connections properly and garbage collection requests are not needed.
• improvements to command line history.
• DownloadResourceAsTempFile never properly indicated progress, showing progress bar with incorrect position.
• Script panel would ask if you want to reload script when time stamp changed, even if the contents did not change.
• 1754: explicitly close Network connections, and call System.gc while interacting with CDAWeb.
• 1754: cache metadata requests so that the metadata for a given file is only retrieved once every ten seconds.
• 1754: corrections to 301 redirects to support old URLs pointing to CDAWeb.
• 1754: Use https to access CDAWeb. Java 1.8.0r102 is now required to access CDAWeb.
• "October 2016" now parses properly. Thanks, Sebastian!
• 995: colorbar with color scatter now appears consistently.
• corrections to labels when X,Y->Z is used. Thanks, Babtiste!
• 1191: restore mousewheel behavior to the secondary mouse module.
• 1769: make sure event things happen on the event thread DataPointRecorder
• rte_1889988765: null pointer path in DataPointRecorder load
• bug 1770, catch the NullPointerException and print it to stderr
• clean up DataPointRecorder new so that it provides all the function of DataPointRecorderNew.
• rte_0809841297: Sebastian noticed this harmless runtime exception, which occurs when the last guy listening to the service was closed.
• don't warn about empty script editor which has been modified.
• Some loggers were still logging at INFO, which shows up in web logs and could cause problems.
• 1765: properly handle carots in URIs. Thanks Ivar!
• 1764: vap+inline GUI with events corrected, so that you can re-enter the GUI.
• 1749: GUI generated supports (kinda) positional parameters
• 1763: spectrogram codes should accept X,Y->Z (rank1 X,Y,Z). Thanks Babtiste and Chris!
• 1762: add support for URL argument types.
• 1760: remove "here" from rebinning code, which would hang the Mac release with stdout noise. Thanks, Sebastian!
• corrections to scatterRebinner to support more data types.
• 1730: correct code for adding label when slice1=i is used. Thanks, Tami!
• file:///tmp/foo.zip/Chipmunks.wav wouldn't work because wav file wasn't accessed properly within the zip file.
• check that processedData checkbox is enabled before setting it because originalData is not enabled.
• 1759: support rank2 DEPEND_2 in expressions like ds[r,:,:]. Thanks, Ivar!
• total supports rank 0 data. Thanks, Ivar!
• 1566: improper escaping meant that pngwalk tool may not have been using thumbs400 directory.
• rte_0064480313: index out of bounds in pngwalk tool when using limited range. Thanks Laura!
• improvements to Orbit mode, for example when the last point is fill.
• "file not found: afile.txt.gz" when afile.txt is requested.
• POLAR_HYDRA_MOMENTS-14SEC have missing parameters in 2008 (ELECTRON_DENSITY) because v02 master files were not used.
• 1753: datum/dataset now results in correct units.
• 1752: renderControl property needed to be handled after the style node when loading a file, so orbit plot settings would load properly
• 1752: more corrections to the orbit plot renderer, such as rounded strokes, tick positioning, and icon
• 1749: correct code that handles unnamed command-line arguments for scripts, which dropped the first argument.
• spectrogram code assumed there were valid Y tags. Thanks, Seth!
• rte_0579587635: single pipe character (|) in operations would cause RTE.
• Hapi client had a branch that would result in class cast exception RTE popup.
• 1750: ds[-1,r]=0 didn't work, where negative indeces weren't supported. Thanks, Ivar!

Features:

• Data Mash Up GUI has bundle operator so that events lists can be constructed.
• Introduce high-rank trim, so you can trim on any index now.
• HAPI Server bins can have either centers or ranges specified.
• HAPI Server support for enumerated data (Strings) supported.
• refactor to create action "Edit Plot"->"Replace plot with clipboard plot"
• metadata now has "copy value" popup menu item.
• support ds[:,1] as well as ds&slice1=1, so that X and Y can be components of vectors.
• 563: HDF and CDF support X and Y datasets. Thanks Kris and others who have requested this.
• bugfix:negative DELTA_PLUS, which is invalid, is handled.
• layout tab indicates selected plot.
• export-to-hapi formats catalog and info responses.
• export-to-wav has timeScale parameter for scaling time tags, useful for auralizing B-Field data.
• AutoplotServlet simpleServlet has symbolSize to support sparklines.
• HAPI server allows external configuration (but it must be in /home/jbf/hapi/ for now!)
• 559: screenshots tool now uses a separate thread to save images, improving responsiveness.
• experimental support for high-rank datasets in HAPI servers
• dom.getElementById added, to remove confusion about indeces.
• provide useful record-by-record reform command that allows reforming within each record.
• getCompletions command provides completions, so that function can be tested immediately.
• annotations allow images to be used when http, https, or file URLs are used.
• x(t),y(t)->z(t) (as well as x(i),y(j)->z(i,j)) supported in spectrogram renderer.
• add method for creating zip files, from within scripts. unzip is deprecated, and it is replaced with gunzip.
• rfe556: support semicolon in ascii files and csv files.
• Util.getBuildTime returns the build time, if available.
• support apply_qflag for Timas data in CDF.
• spectrogram scatter mode. Thanks, Matt!
• cdawebHttps=true environment variable allows use of https for web transactions with CDAWeb.
• logging improvements to CDF reader.
• LinFit has getSlope and getIntercept methods, in anticipation of proper SI units library.

Older Bugfixes:

• improve drawing of orbit plot ticks.
• data mashup tool ignores IDs with empty URIs.
• HAPI client was doing web interactions on the event thread, causing possible hangs
• aggregation avail=T attempted to download granule metadata when it is not used.
• render topDecorator is cleared when reset is called.
• example Jython scripts in VirboAutoplot/src/scripts updated and now work for everyone.
• Jython getParam position argument (as opposed to named argument) works again.
• remove extra rows in Events List when events list is a single file
• 1740: two-columns of iso8601 do not read properly as events list. Thanks George!
• improvements to ASCII editor, including examples and improved guessing.
• demo 14 now uses https.
• dep0units can be used to explicitly set the units, and does this correctly for times now as well.
• 1743: make sure all datasets entering Autoplot are immutable.
• 1746: dasPlot created off event thread would occasionally cause RTE 11148*.
• 1739: include the scientist's home directory in the list of completions. Thanks, Sarah!
• scripting had bind command, but no unbind.
• corrections to ELEMENT_DIMENSIONS property for bundles
• Jython completions detect static helper classes and don't complete to argument-less constructor, like Schemes()
• 1738: sequence using ".." in URI completions, fix unstable code.
• 1735: correct synchronization errors which would lead to occasional errors when loading .vap files
• 1734: noisy "alarming extrapolation in findex" warning is more lax.
• 1728: change color to black from custom
• 1732: Bookmarks when a single plot is already plotted
• 1731: get screenshot for second display.
• rte_2095724092: don't go into the completions editor when there is no extension.
• 1727: break into stack of plots reconnects time series browse to new x-axis.
• units on bin_plus and bin_minus in HAPI client.
• rte_2071759301: rename file when old file wasn't set caused NullPointerException.
• in data mash up, handle cancel in "add function" dialog.
• setXAxis of DasPlot was incorrect. Thanks, James!
• corrections to HAPI support for CDAWeb.
• 1724: autoplotapp.py is constantly rewritten, causing occasional "File not found" popups.
• IDL/Matlab interface provides access to more datasets in qstreams.
• 1722: qstreams with arbitrary qdataset properties.
• 1718: HDF5 supports DELTA_PLUS_VAR, DELTA_MINUS_VAR metadata.
• 1718: LANL NN with log-spaced x-tags.
• test002 showed where -INFs were getting into AutoHistogram and causing unnecessary warnings.
• 1714: slice function of NetCDF (and HDF) result didn't handle DEPEND meta dataset property.
• 1714: Brian showed where HDF5 variable was ISTP, but its DEPEND datasets were not.
• something changed on Mint Linux display (or java version) and get screenshot didn't work on one of the screens.
• old nbproject areas (ant build) rebuilt from scratch, reducing incremental build time from 20sec to 5sec.
• rte_1736345160: improperly-localized strings caused RTE in layout tab's "make taller" and other buttons. Thanks, Andre!
• rte_0095613254: if the vap+cdf: was missing from the URI, Tool->"Mash Data" would fail.
• stacked histogram correctly handles data where points are more course than the resolution and a peaks plane is found, so show hourly averages.
• Help->"About this release" didn't point at the correct URL.
• 1708: digitizer component fails when the first column was not time, because code assumed times with timeFormatter.
• data set selector component setValue would add "%20" when there was an extra space at the end when cut-n-paste URIs.

Less recent features:

• mouse modules (crosshair,etc) allow up,down,right,left keys for precise control.
• add example values for depend0units and timeFormat parameters in ascii editor panel.
• ASCII table data source format supports "+" in tformat parameter, to output multiple columns
• tweaks to the ASCII editor panel, improving tooltips and providing examples.
• HAPI data source makes SPASE link and DOI links clickable.
• "run batch" feature allows jython scripts to be run for a series of inputs, and has methods for generating inputs.
• rfe 548: Das2 Renderers have a decorator object to allow additional drawing code.
• HAPI shows descriptions if available.
• data mash-up tool supports string arguments and constants like QDataSet.UNITS. Thanks, Kristoff!
• metadata tab allows access to the metadata for each bundled dataset.
• update internal orbits, and include 2017 leap second.
• 20000th SVN revision!
• layout tab has "move to top" menu item
• lookupAxis supports horizontal axis.
• 543: support for time zone offsets in ISO8601 strings.
• HAPI data source supports JSON data response.

Known issues:

• see Source Forge

This completes the feature set for the new production branch. No major features will be added before branching this version. A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of autoplot.org will run this version. Feedback on the function and quality of this release is greatly appreciated!

Data is input from:

• ASCII files (.dat,.txt)
• CSV files (.csv)
• Binary files (.bin)
• CDF files(.cdf)
• Excel spread sheets (.xls)
• das2Streams (.d2s or .das2Stream)
• ViRBO's TSDS server
• CDAWeb data server (vap+cdaweb:)
• das2servers (vap+das2server:)
• QStreams (.qds), QDataSet on a das2Stream
• Plasma Wave Group das2 Server
• NetCDF files (.nc,.ncml)
• HDF5 files (.h5,.hdf5)
• DODs servers (.dds)
• Cluster Exchange Format files (.cef)
• Fits Format Files
• Images (.jpg, .png, .gif)
• Wav audio files (.wav)

This is not complete, and limitations include:

• Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
• Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.

autoplot.org is the wiki website, and contains more information than this launch page.