Autoplot Application (20170401a)
PNGWalkTool is used to investigate occasional failures in nightly testing.
Autoplot attempts to allow plotting of data from many data sources, including remote CDF files and OpenDAP servers.
Datasets are identified with URIs, and by the URI's extension, data are loaded into the internal data model. Data are then
displayed by guessing the rendering method and axis ranges to provide a reasonable, useful view of the data. Data can then
be additionally processed with other operations like smooth and FFT, and Jython scripting provides a capable programming environment.
This is the Autoplot2016a branch. This is an update to the production version containing bugfixes and minor new features.
Java 7 is now required. Java 8 will be required sometime soon.
Single-jar version, autoplot.jar, can be run without network access and from the command line.
Mac DMG Release is available.
Bugfixes (2017-03-31 r20633):
- special handling of parseDecimal("NaN").
- improve handling of space and umlaut (¨) in pngwalk file names.
- rte_0746124665: George saw path that was resulted in NullPointerException.
- one-arg join(ds) was deleted at some point, and is now replaced. Thanks, George!
- HAPI export supports rank 3.
- bundle(ds1,ds2) correctly handles DEPEND_1 and DEPEND_2 when bundled datasets have equivalent values.
- correction to ascii formatting when microsecond resolution is used. Thanks, Kristoff!
- slice2 and slice3 preserve NAME property.
- bugfix: LocalFileObject write capability didn't check to see if it could create a file, with the QC stuff in PNGWalk.
- 1804: trim off-by-one error corrected.
- 1802: checkIsotropic could trigger before axes were sized with AutoplotServer.
- Java 8 Javadocs supported in Jython completions.
- new CDF Jar file fixes problem where unused ThreadGroups were made.
- 1801: dataset iterator is inordinately slow, put in optimal loops for rank 1 data.
- interpolate and interpolateNN now have single-dataset interface as well as arrays. Thanks, Kristoff!
- BufferDataSet remove redundant check of reclen which was also miscoded, using remaining() instead of limit(), producing incorrect warnings.
- tca can be reset before entering the editor panel. Thanks, Kris!
- when bins are specified with ranges, use render type hint to request Nearest Neighbor Spectrograms.
- bugfix: in the case where the HAPI dataset is just one day, the discovery range would be the day before the data range, making it a poor choice.
- PDS/PPI uses https now. Editor properly sets the parameter when returning to the editor GUI.
- downloadResourceAsTempFile now includes http or https (or ftp) in generated filename.
- getConvertibleUnits replaces getConvertableUnits (spelling error), and getConvertibleUnits now sorts the units.
- trimStrideWrapper dropped metadata.
- Units.getAllUnits would return a list of strings when it's declaration indicated that it would return a list of Units. Thanks, Jon!
- RTE reports (and provide feedback) must now submit to jfaden.net:80, not jfaden.net:8080, because of firewall issues.
- 1794: correction to branch when github is accessed.
- some spectrogram and color scatter assumed data and colorbar zunits were the same.
- https://bugs-pw.physics.uiowa.edu/mantis/view.php?id=478: ascii time transfer types could not have differing units.
- test100 showed where an immutable dataset was modified with the "hanning" filter.
- &y=bgsm[:,1] was unnecessarily URL-encoded, reducing readability.
- where parameters list was incorrect when the HDF5 file contained structures
- link(x,y,z) dropped z, when x y or z is an object (not dataset).
- issue a warning when das2stream units change, which can cause problems.
- where with HDF5 files containing parameters with dots in the name was never tested.
- where with HDF5 files containing structures was never tested.
- csv editor panel has selection for delimiter.
- corrections to HAPI reader, which was broken after rank 2 DEPEND_1 support was added.
- addDataPoint.jy script updated to use DataPointRecorder (not DataPointRecorderNew), and supports keystroke mode.
- log "ImageIO used to create" at less than info, to avoid clutter in server logs.
- rte_1490216564: don't throw runtime error when taskSize is zero, just ignore it.
- 1785: unable to format bundle to CDF. (Append is now handled properly.) Thanks, Kristoff!
- 1789: PngWalkTool correct feedback about writing to writable fs. The code didn't distinguish between Read-Only and Read-Write, and assumed that local filesystems were read-write. Thanks, Sebastian!
- rte_0147649225: fall back after failing to find "field6" in bundleDescriptor, because HAPI metadata was mistaken for Rich ASCII.
- rte 0857617456: Null pointer when selecting time field type but time format column has never been set. Thanks, Shiva!
- corrections to format to .wav gui, which wasn't initialized properly to indicate automatic scaling.
- digitizer DataPointRecorder "new" operation fires dataset update event. Thanks, Kris!
- bugfix: vap+cdaweb:ds=IM_K0_SIE&id=SI_PIXELS had image>THUMBSIZE>50 for the RENDER_TYPE, which Autoplot doesn't support as "image"
- Experimental support for rank 2 DEPEND_1 datasets with HAPI. HAPI uses sampleStartDate for discovery.
- check for vap+internal:data_1,data_2 when switching to mash-up
- 1754: Correctly support http://autoplot.org/data/AMSR_E_L3_SeaIce6km_B06_20070307.hdf.gz
- HAPI supports nominal data.
- 1754: Java now reuses https connections properly and garbage collection requests are not needed.
- improvements to command line history.
- DownloadResourceAsTempFile never properly indicated progress, showing progress bar with incorrect position.
- Script panel would ask if you want to reload script when time stamp changed, even if the contents did not change.
- 1754: explicitly close Network connections, and call System.gc while interacting with CDAWeb.
- 1754: cache metadata requests so that the metadata for a given file is only retrieved once every ten seconds.
- 1754: corrections to 301 redirects to support old URLs pointing to CDAWeb.
- 1754: Use https to access CDAWeb. Java 1.8.0r102 is now required to access CDAWeb.
- "October 2016" now parses properly. Thanks, Sebastian!
- 995: colorbar with color scatter now appears consistently.
- corrections to labels when X,Y->Z is used. Thanks, Babtiste!
- 1191: restore mousewheel behavior to the secondary mouse module.
- 1769: make sure event things happen on the event thread DataPointRecorder
- rte_1889988765: null pointer path in DataPointRecorder load
- bug 1770, catch the NullPointerException and print it to stderr
- clean up DataPointRecorder new so that it provides all the function of DataPointRecorderNew.
- rte_0809841297: Sebastian noticed this harmless runtime exception, which occurs when the last guy listening to the service was closed.
- don't warn about empty script editor which has been modified.
- Some loggers were still logging at INFO, which shows up in web logs and could cause problems.
- 1765: properly handle carots in URIs. Thanks Ivar!
- 1764: vap+inline GUI with events corrected, so that you can re-enter the GUI.
- 1749: GUI generated supports (kinda) positional parameters
- 1763: spectrogram codes should accept X,Y->Z (rank1 X,Y,Z). Thanks Babtiste and Chris!
- 1762: add support for URL argument types.
- 1760: remove "here" from rebinning code, which would hang the Mac release with stdout noise. Thanks, Sebastian!
- corrections to scatterRebinner to support more data types.
- 1730: correct code for adding label when slice1=i is used. Thanks, Tami!
- file:///tmp/foo.zip/Chipmunks.wav wouldn't work because wav file wasn't accessed properly within the zip file.
- check that processedData checkbox is enabled before setting it because originalData is not enabled.
- 1759: support rank2 DEPEND_2 in expressions like ds[r,:,:]. Thanks, Ivar!
- total supports rank 0 data. Thanks, Ivar!
- 1566: improper escaping meant that pngwalk tool may not have been using thumbs400 directory.
- rte_0064480313: index out of bounds in pngwalk tool when using limited range. Thanks Laura!
- improvements to Orbit mode, for example when the last point is fill.
- "file not found: afile.txt.gz" when afile.txt is requested.
- POLAR_HYDRA_MOMENTS-14SEC have missing parameters in 2008 (ELECTRON_DENSITY) because v02 master files were not used.
- 1753: datum/dataset now results in correct units.
- 1752: renderControl property needed to be handled after the style node when loading a file, so orbit plot settings would load properly
- 1752: more corrections to the orbit plot renderer, such as rounded strokes, tick positioning, and icon
- 1749: correct code that handles unnamed command-line arguments for scripts, which dropped the first argument.
- spectrogram code assumed there were valid Y tags. Thanks, Seth!
- rte_0579587635: single pipe character (|) in operations would cause RTE.
- Hapi client had a branch that would result in class cast exception RTE popup.
- 1750: ds[-1,r]=0 didn't work, where negative indeces weren't supported. Thanks, Ivar!
- Data Mash Up GUI has bundle operator so that events lists can be constructed.
- Introduce high-rank trim, so you can trim on any index now.
- HAPI Server bins can have either centers or ranges specified.
- HAPI Server support for enumerated data (Strings) supported.
- 1779: refactor to create action "Edit Plot"->"Replace plot with clipboard plot"
- metadata now has "copy value" popup menu item.
- support ds[:,1] as well as ds&slice1=1, so that X and Y can be components of vectors.
- 563: HDF and CDF support X and Y datasets. Thanks Kris and others who have requested this.
- bugfix:negative DELTA_PLUS, which is invalid, is handled.
- layout tab indicates selected plot.
- export-to-hapi formats catalog and info responses.
- export-to-wav has timeScale parameter for scaling time tags, useful for auralizing B-Field data.
- AutoplotServlet simpleServlet has symbolSize to support sparklines.
- HAPI server allows external configuration (but it must be in /home/jbf/hapi/ for now!)
Less recent features:
- 559: screenshots tool now uses a separate thread to save images, improving responsiveness.
- experimental support for high-rank datasets in HAPI servers
- dom.getElementById added, to remove confusion about indeces.
- provide useful record-by-record reform command that allows reforming within each record.
- getCompletions command provides completions, so that function can be tested immediately.
- annotations allow images to be used when http, https, or file URLs are used.
- x(t),y(t)->z(t) (as well as x(i),y(j)->z(i,j)) supported in spectrogram renderer.
- add method for creating zip files, from within scripts. unzip is deprecated, and it is replaced with gunzip.
- rfe556: support semicolon in ascii files and csv files.
- Util.getBuildTime returns the build time, if available.
- support apply_qflag for Timas data in CDF.
- spectrogram scatter mode. Thanks, Matt!
- cdawebHttps=true environment variable allows use of https for web transactions with CDAWeb.
- logging improvements to CDF reader.
- LinFit has getSlope and getIntercept methods, in anticipation of proper SI units library.
This completes the feature set for the new production branch. No major features will be added before branching this version.
A production branch will be started in a couple of weeks, and the "Start Autoplot" link at the top of
autoplot.org will run this version.
Feedback on the function and quality of this release is greatly appreciated!
Data is input from:
- ASCII files (.dat,.txt)
- CSV files (.csv)
- Binary files (.bin)
- CDF files(.cdf)
- Excel spread sheets (.xls)
- das2Streams (.d2s or .das2Stream)
- ViRBO's TSDS server
- CDAWeb data server (vap+cdaweb:)
- das2servers (vap+das2server:)
- QStreams (.qds), QDataSet on a das2Stream
- Plasma Wave Group das2 Server
- NetCDF files (.nc,.ncml)
- HDF5 files (.h5,.hdf5)
- DODs servers (.dds)
- Cluster Exchange Format files (.cef)
- Fits Format Files
- Images (.jpg, .png, .gif)
- Wav audio files (.wav)
This is not complete, and limitations include:
- Quantities of rank 3 dimensionality (e.g. flux(Time,Energy,pitch) are plotted by slicing.
- Quantities of rank 4 dimensionality (e.g. flux(Time,Energy,azimuth,sector) are plotted by slicing twice.
autoplot.org is the wiki website, and contains
more information than this launch page.