Description of SIS Level-One Processing The purpose of this document is to describe the processing that occurs on the SIS data during the level-one processing at the ACE Science Center (ASC). I. Overview of ASC Instrument Level-One Processing The data that is the starting point of the level-one processing has the following characteristics: 1. The data consists of a single stream in chronological (S/C clock) order. The raw S/C data can reach the Mission Operation Center (MOC) and eventually the ASC by several paths. The level-zero processing merges the different paths into a single stream and inserts fill data in places where there is missing data. 2. Each S/C minor frame has associated with it a one-bit quality flag. The quality indicates whether the level-zero processing encountered uncorrectable errors in processing the data for that minor frame. If uncorrectable errors are encountered, the level-zero processing removes the minor frame and inserts fill data. 3. The data stream is expected to contain few dropouts, and the dropouts that occur are expected to cluster during time intervals when receiving conditions are poor. The level-one processing of instrument data is performed inside a loop that calls four C functions per instrument for each S/C minor frame. The four functions associated with SIS are 1. accum_sis_level1, 2. accum_sis_browse, 3. output_sis_level1, 4. output_sis_browse. From the names, it is obvious that accum_sis_level1 and output_sis_level1 are associated with the generation and output of SIS level-one data. The other functions are associated with the generation and output of browse data and will not be discussed in this document. The function, output_sis_level1, directly controls the structure and content of the SIS data that is written to the level-one data set which will be distributed to the instrument teams. It has only indirect control of the format of the data written to the CDROM. The format is controlled by the eight functions 1. write_sis_hskp_rate, 2. write_sis_events, 3. write_sis_diagnostic_event, 4. write_sis_command_echo, 5. write_sis_command_table, 6. write_sis_incomplete_event, 7. write_sis_unprocessed_minor_frame, 8. write_sis_minor_frame_summary. Each of the above write functions is written by a PERL script that is furnished by the ASC. The PERL script takes as input the definition of a C data structure and generates a write function with embedded HDF calls. When that PERL-generated write function is called it writes the data to the level-one data set. The ASC will furnish the inverse read functions that will allow the user to recover the original C data structure from the level-one HDF data set. In addition to the instrument data, the level-one product includes S/C housekeeping data. The ASC will furnish functions that given the level-one S/C housekeeping structure, the functions will return the attitude and postion of the S/C. Also the ASC will furnish a function that converts the S/C clock to utc. II. Overview of SIS Telemtry SIS operates on a 256 instrument minor frame cycle. During each instrument cycle, SIS has a sync bit which remains low for the first 128 instrument minor frames and high for the next 128 instrument minor frames. During each instrument cycle, SIS samples each housekeeping parameter once, each low priority rate once, and each high priority rate eight times. In addition, on even instrument cycles, SIS samples the coarse matrix detector DAC settings and PHA offsets. On odd instrument cycles, SIS samples the fine matrix detector DAC settings and PHA offsets. SIS during its normal processing, outputs a collection of compressed events during the instrument cycle. In general, the compressed events will span instrument minor frame boundaries, but during periods of normal processing will not span instrument cycle boundaries. SIS contains an 1180 byte command table which defines the instrument status. During each instrument cycle, 80 bytes of the command table is readout as part of the housekeeping data. It takes about 15 instrument cycles to readout the entire table. When SIS receives a command, it puts a command response in its telemetry. In general, command response spans minor frames and can span cycle boundaries. Finally, SIS can be put into a diagnostic mode. During diagnostic mode, SIS suspends putting compressed events into its telemetry stream, and substitutes diagnostic event data. The housekeeping/rate and command responses are uneffected. The most important point from the above discussion is that SIS produces five different types of data: housekeeping/rate, compressed event, command table, command responses, and diagnostic event. The SIS level-one processing reflects this data organization. III. S/C Minor Frames and Instrument Minor Frames The ACE S/C produces one minor frame of data per second. Within each S/C minor frame there is a 249 byte allocation for SIS data. SIS produces one 249 byte instrument minor frame per second. SIS uses its 249 byte allocation in the S/C minor frame to output the last two bytes of the previous instrument minor frame and the first 247 bytes of the current instrument minor frame. The ACE S/C maintains a major frame counter and a minor frame counter. The S/C housekeeping data is tied to these counters, and most instruments' telemetry are also tied to the S/C minor and major frame counters. SIS does not use these counters. IV. Overview of SIS Level-One Processing The SIS level-one processing parses the natural occuring data structures from the raw SIS data. Each data structure which is built contains a S/C clock time tag that allows the data to be traced to the original S/C minor frame and to correlate it with other data. The data inside the structures is left in raw units, i.e. ADC values, PHA value, and counts. The details of each structure is covered in the next section. The level-one processing is done on an instrument cycle basis. In other words, 256 instrument minor frames (spanning 257 S/C minor frames) are processed at a time. The criteria used to determine whether a block of minor frames contains a valid instrument cycle is the following: 1. the S/C clock of each S/C minor frame increments by 1 for each of the 257 minor frames present, 2. the S/C major and minor frame counters increment properly for each of the 257 S/C minor frames, 3. the quality flag is low for each of the 257 S/C minor frames present, 4. the first 128 instrument minor frames have their sync bits set low, 5. the next 128 instrument minor frames have their sync bit set high, 6. the sync bit of the next instrument minor frame is low, 7. all events in the event are self-consistent in length, i.e. the sum of the length of the components is consistent with the length of the event. If the block passes the test, then the data is processed and put in the appropriate data structure. If the block fails, the following happens: 1. the oldest S/C minor frame is written to the level-one data set as unprocessed data, 2. another S/C minor frame is added to the end of the block, 3. the block is retested using the above criteria. If the number of unprocessed minor frames grows unacceptably large, then somewhere in the level-two processing the problem can be addressed. The unprocessed minor frames are part of the level-one product. By including the unprocessed minor frames in the product, it is possible to recover the original SIS data stream from the level-one product; however, no software is planned for such a recovery. V. SIS Level One Data Structures For each of the write functions mentioned in section I of this document, there is a corresponding data structure which will be written to the level-one data set using HDF functions. The purpose of this section is to define these data structures and provide some explanation for their existence and form. Each subsection will begin with the actual C structure followed by a brief explanation. A. Compressed Event Data Structure #define MAX_NUM_SIS_EVENTS 2048 #define NUM_SIS_EVENT_BUFFER_BYTES 62336 /* 243*256 + 128 */ typedef struct { uint32 ClockCycle; uint32 ClockMinorFrame[MAX_NUM_SIS_EVENTS]; uint16 EventIndex[MAX_NUM_SIS_EVENTS]; uint16 NumberEvents; uint8 EventBuffer[NUM_SIS_EVENT_BUFFER_BYTES]; uint8 ContinuationFlag; } L1SisEventStruct; /* Definition: */ /* ClockCycle = S/C clock of the first minor frame of the cycle */ /* ClockMinorFrame = S/C clock of the minor frame containing the */ /* start of each event */ /* EventIndex = Starting byte of each event strored in the event */ /* buffer */ /* NumberEvents = Number of events stored in the event buffer */ /* EventBuffer = Event buffer */ /* ContinuationFlag = Flag indicating events from this cycle are */ /* continued in the next structure */ The SIS compressed events occupy most of the SIS data stream. These events come from SIS in a compressed form, and this form is maintained during the level-one processing. Even though it would have been possible to define a fixed data structure to store each event, it was felt that this would result in greatly enlarged data files. Since the work to expand a compressed event should be small compared to the work of analysing the event, it was felt that doing the expansion at analysis time was more appropriate. Standard functions to expand the compressed event into a fixed structure can be furnished to the SIS team members. The actual events have an internal time tag with a resolution of 256 seconds. The S/C clock associated with an event is the time the event was put in the telemetry stream; it is not the time that SIS processed the event. For low priority events, the readout time can be significantly different from the time the event was processed. The continuation flag in the data structure is not expected to be used. It only exists to deal with the pathological case, that the average number of events readout during a minor frame exceeds eight. B. Housekeeping/Rate Structure #define NUM_SIS_HSKP_RATE_BUFFER_BYTES 1536 typedef struct { uint32 ClockCycle; uint8 HskpRateBuffer[NUM_SIS_HSKP_RATE_BUFFER_BYTES]; } L1SisHskpRateStruct; /* Definition: */ /* ClockCycle = S/C clock of the first minor frame of the cycle */ /* HskpRateBuffer = Housekeeping/rate buffer for the cycle */ In the SIS telemetry stream, six bytes from each minor frame are used to output housekeeping and rate quantities. The pattern is the same for each instrument cycle, so a data structure with fixed fields could have been substituted for the byte array in the structure. This would make given standard names to all the data quantities and made the the data slightly more accessible. However, most of these same benefits can be obtained by furnishing a C function that expands the data. The main argument for leaving it as an array is that the expansion can take place at the same time as the conversion to physical units; there is very little benefit in introducting another structure containing raw data. C. Command Table Structure #define NUM_SIS_COMMAND_TABLE_BYTES 1180 typedef struct { uint32 ClockMinorFrame0; uint32 ClockMinorFrame1; uint8 CommandTable[NUM_SIS_COMMAND_TABLE_BYTES]; uint8 MissingDataFlag[NUM_SIS_COMMAND_TABLE_BYTES]; } L1SisCommandTableStruct; /* Definition: */ /* ClockMinorFrame0 = S/C clock of the minor frame containing the */ /* first byte of the command table */ /* ClockMinorFrame1 = S/C clock of the minor frame containing the */ /* last byte of the command table */ /* CommandTable = SIS command table */ /* MissingDataFlag = Missing data flag for each byte of the */ /* command table */ The data in the SIS command table structure is duplicated in the housekeeping/rate structure. In general, duplicating data is undesirable. However, since each housekeeping/rate structure contains only 80 bytes of the command table, the command table is much easier to use if there is a structure that contains the whole table. Since the readout of the command table spans about an hour, it is assumed the occasionally a minor frame of data will be missing. For places where there is missing data, the missing data flag is set. The level-two processing will have to decide what to do with command tables with missing data. The two clocks associated with the command table only indicate when the readout of the command table started and when it ended. They should not be interpreted to mean that in the time interval the command table's value was stable. D. Command Echo Structure #define MAX_NUM_SIS_COMMAND_ECHO_CHARACTERS 242 typedef struct { uint32 ClockMinorFrame; uint8 NumberChars; char8 CommandEcho[MAX_NUM_SIS_COMMAND_ECHO_CHARACTERS]; } L1SisCommandEchoStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame containing the */ /* command echo */ /* NumberChars = Number of characters in the command echo */ /* CommandEcho = Command echo characters */ Whenever a command is received, SIS puts a command response in its telemetry. There is no agreed-upon limit on the size of the SIS command response. Also, the form of the command response is ASCII characters with embedded linefeeds and carriage returns. The command response was designed to be displayed on a "dumb" terminal and to be interpreted by humans. Since command responses usually span minor frames and also can span instrument cycles, there is no simple way to determine where one command response ends and another begins. To keep the level-one processing simple, it was decided to call the portion of the command response that appears in a instrument minor frame a command echo and define a structure to contain it. E. Diagnostic Event Structure #define NUM_SIS_DIAG_EVENT_BUFFER_BYTES 894 typedef struct { uint32 ClockMinorFrame; uint32 NumberBytes; uint8 DiagEventBuffer[NUM_SIS_DIAG_EVENT_BUFFER_BYTES]; } L1SisDiagnosticEventStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame containing the */ /* start of the diagnostic event */ /* NumberBytes = No. of bytes of the diagnostic event in this */ /* structure */ /* DiagEventBuffer = Diagnostic event buffer */ A SIS diagnostic event consists of a raw event and a compresed event. The size of the SIS raw event is 538 bytes. The maximum size of a SIS compressed event is 112 bytes. The remaining bytes in the buffer are used for padding to the end of the instrument minor frame. F. Unprocessed Minor Frame Structure #define NUM_SIS_TELEMETRY_BYTES 249 typedef struct { uint32 ClockMinorFrame; uint8 FormatId; uint8 MajorFrameCount; uint8 MinorFrameCount; uint8 quality; uint8 MainBusVoltage; uint8 SisCurrent; uint8 SisInternalTemperature1; uint8 SisInternalTemperature2; uint8 SisInterfaceTemperature; uint8 SiSCoverTelltale; uint8 NumberBytesUsed; uint8 SisData[NUM_SIS_TELEMETRY_BYTES]; } L1SisMinorFrameStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame */ /* FormatId = Format of the minor frame */ /* MajorFrameCount = Major frame count of the minor frame */ /* MinorFrameCount = Minor frame count of the minor frame */ /* quality = Quality flag associated with the minor frame */ /* MainBusVoltage = Main bus voltage value (0=no data) */ /* SisCurrent = SIS I/F & survial heater current (0=no data) */ /* SisInternalTemperature1 = */ /* SIS internal temperature 1 (0=no data) */ /* SisInternalTemperature2 = */ /* SIS internal temperature 2 (0=no data) */ /* SisInterfaceTemperature = */ /* SIS interface temperature (0=no data) */ /* NumberBytesUsed = Number of SIS data bytes used */ /* SisData = SIS data */ Whenever the SIS level-one processing encounters a S/C minor frame which contains data that is not part of a valid SIS instrument cycle, it writes the above structure to the level-one data set. Most of the fields in the structure are self-explanatory. Because of the relationship of an instrument minor frame to a S/C minor frame, the values which NumberBytesUsed can assume the values 0, 4, and 54. G. Minor Frame Summary Structure typedef struct { uint32 ClockMinorFrame; uint8 FormatId; uint8 MajorFrameCount; uint8 MinorFrameCount; uint8 quality; uint8 MainBusVoltage; uint8 SisCurrent; uint8 SisInternalTemperature1; uint8 SisInternalTemperature2; uint8 SisInterfaceTemperature; uint8 NumberBytesUsed; } L1SisMinorFrameSummaryStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame */ /* FormatId = Format of the minor frame */ /* MajorFrameCount = Major frame count of the minor frame */ /* MinorFrameCount = Minor frame count of the minor frame */ /* quality = Quality flag associated with the minor frame */ /* MainBusVoltage = Main bus voltage value (0=no data) */ /* SisCurrent = SIS I/F & survial heater current (0=no data) */ /* SisInternalTemperature1 = */ /* SIS internal temperature 1 (0=no data) */ /* SisInternalTemperature2 = */ /* SIS internal temperature 2 (0=no data) */ /* SisInterfaceTemperature = */ /* SIS interface temperature (0=no data) */ /* SIS interface temperature (0=no data) */ /* NumberBytesUsed = Number of SIS data bytes used */ The purpose of this data structure is two-folded. The first is to give a summary of the amount of data was processed out of each S/C minor frame. The second is to hold the SIS-specific S/C housekeeping values. H. Incomplete Event Structure typedef struct { uint32 ClockMinorFrame; uint8 NumberEventBytes; uint8 EventBytes[128]; } L1SisIncompleteEventStruct; The main purpose for the existence of this data structure is to be able to claim that 100% of the SIS data is in the level-one data set. Incomplete events can arise in two different manners. The first way occurs when SIS is in normal mode. Since compressed events are of variable length, there is usually one event which will not fit into the event space of the instrument cycle. Since events are not allowed to span cycle boundaries, the event gets truncated if SIS attempts to put it into the instrument minor frame. The second way happens when SIS switches from normal mode to diagnostic mode and back to normal mode. When SIS changes to diagnostic mode, it suspends its normal event processing and starts inserting frames with diagnostic event data. When it resumes normal mode, it will continue at the exact byte that it had last outputted. If for some reason, there is missing data in the cycle that SIS resumes normal operations, the last event will not be completed by the level-one processing.