developer.cdfguide

From autoplot.org

Jump to: navigation, search

Purpose: outline a short getting started guide for new CDF authors.

Audience: people getting started with creating CDF files who wish to contribute ideas.

See also: http://autoplot.org/developer.cdf which is focused on metadata to Autoplot display.

Contents

  1. Introduction
  2. List of Confusing Ideas to Cover
  3. Outline
  4. CDF as Archival Format
  5. Other resources

1. Introduction

This is not a getting started with CDF guide, but instead it is notes for one that doesn't yet exist.

2. List of Confusing Ideas to Cover

  • rVar/zVar difference -- use zvars
  • representing timetags: Epochs, TT2000 -- use TT2000 and be done with it.
  • metadata that connects variables (DEPEND_0)
  • valid data and plot hints
  • ISTP metadata and skeleton files
  • structure for common examples
  • How to create quick CDF data files from IDL
  • working with SPDF to make your data available to the community

3. Outline

  • Introduction to CDF file format
    • History and goals
    • What is it? Binary format that contains a named arrays and associations between arrays and metadata.
    • Uses compared to ascii data, etc.
    • CDAWeb is like a database but data is ingested in chunks and can be copied out into chunks.
  • Reading CDF files
    • Reading in Autoplot
    • IDL and Matlab
    • Using CDAWeb make your data available to the world
  • Writing CDF files from scratch
    • creating and installing data.
    • first metadata to specify title, fill values and scale settings
    • installing times. TT2000.
    • linking data DEPEND_0
    • multidimensional datasets.
    • varying vs non-varying data.
    • what's not talked about
      • sparse data
      • blocking
      • compression
  • Using skeletons.
    • Separation of roles. Let the scientist specify the metadata, and the programmers handle the data.
    • a complete processing system
    • validating product with skteditor.
  • Ingesting data into CDAWeb
    • naming CDF files. Filenaming conventions, mission, instrument, dates, and versioning.
    • what about missing files? Should an empty file be made?
  • Example schemes for data
    • spectrogram
    • sparse data
  • Limitations -- people have asked a lot about how big can CDF files be? I suggest <300 MB, but I know Autoplot has handled 1GB files.
  • WriteMyCDF
  • Reading and Writing in Java.

4. CDF as Archival Format

  • variables should be single blocks. This provides efficient access with NIO.
  • or cdfs should be compressed by variable, but in this case we use chunks to provide random access.
  • Timetags are stored in TT2000 format, and are not compressed. (Uh-oh, does this contradict last two?)

5. Other resources

Does Bob McGuire have a powerpoint that shows some of these ideas that we could link to?

Tami (CDAWeb) has a page showing different types of data in CDF files: http://spdf.gsfc.nasa.gov/istp_guide/variables.html

Personal tools