# zstream

Purpose: notes about the stream format we talked about at the RPWS group.

Introduction: first das2streams, then qstreams. qstreams still weren't clean and extensible. This attempts to fix the problems.

[T|N|L]
T: the type of packet.  For example, stream descriptor, packet descriptor, or data.
N: the packet identifier.  numbers or zero-length string to indicate anonymous.  This must only be positive and less than 1000, or zero length.
L: decimal length.  Number of bytes following the packet.  Zero is acceptable (heartbeat).  999999999 is limit.


The type (T) of packet can be one of:

• S, a stream header. What follows this will always be a valid stream. (This is the sync capability we revere but never implement.) Once a S is encountered, all Ss can be ignored. This will always be anonymous, so that streams have the magic number "[S||" The identifier (N) indicates the scheme for the stream, e.g. das2stream or QStream.
• H, a packet header. This is an xml description of things that will be on the stream.
• h, a zipped packet header. This can be unzipped to become an H.
• D, a data packet. This is data described by an H with the same identifier.
• d, a zipped packet. This can be unzipped to become a D.
• C, a comment packet. This is intended to provide non-essential metadata such as progress information, etc for a stream.
• c, a zipped comment packet.

• number of packet types should be kept to a minimum. Semantic layers will provide abstraction.
• one type of compression should be implemented at this layer, since it's possible to do compression regardless of content.
• stream header serves as a sync so it should be its own type.
• comments are non-essential metadata that can always be removed, like stderr messages vs stdout.

That's the S11 layer.

• repeat count? This would enable multiple packets to be compressed together.

# 1. Limits

Length of packet content, 0..999999999. (9 digits) Length of packet id, 0..999 (3 digits) Length of header packet, without content, is 1 + 1 + 1 + 3 + 1 + 9 + 1 = 27 bytes. Characters of packet length, must parse as integer, containing only 0..9 and spaces. Characters of packet id, containing 0..9.

# 2. Goals

With the old wiki, there was a list of about 12 things we wanted to accomplish with streams.

• transmit data
• ac-hoc method for storing data, just like .idlsav files.
• provide self-describing but readable format. (ascii format)
• provide efficient means for transmitting data. (binary format, compression.)
• allow client to sync up with the stream. (like telemetry stream)
• allow metadata such as progress information to be inserted into the stream.
• valid stream concatenated with valid stream is valid stream.
• heartbeats or keep-alive messages: comments allow for this...

# 3. Questions

• Can properties be defined in the S packet, indicating stream properties?
 Chris answer:  Of course, the contents of S packets are defined by the
protocol number.  Put whatever you like in there.


# 4. Proposal on Stream Syntax numbers

How about the following for a starter protocol index list:

• 0 - Not a legal protocol ID. (Used to terminate a stream?)
• 1 - Das1 Packets in this chucking format, stream would only have [D] packets, not [H] packets.
• 2 - Das2 Packets in this chucking format
• 3 - QStream Packets in this chunking format
• 4 - Das2.2 server "in-band" queries
• 100 - Chunked binary object transfer, a way to send a CDF file for example

? Also the 900 block is reserved for private use.