ACE Weekly 07/23/04 - 07/29/04 All ACE spacecraft subsystems are performing nominally. No attitude maneuvers were executed this reported period. The next attitude maneuver is scheduled for 08/02/04. ============================================================================ Orbit Events: From Craig Roberts (FDF): The next ACE stationkeeping maneuver (SK-33) is tentatively scheduled for 08/20/04, to coincide with a (to be) scheduled attitude reorientation maneuver. Thus SK-33 will be out of the way just as the next SEZ transit--which will have its closest approach to the Sun on 08/30/04 --is beginning. The delta-V for 08/20/04 is currently projected to be about 0.37 m/sec in the anti-Sunward direction. So the burn will be using the top-deck axials 1A and 2A. The burn duration is estimated at 64 seconds. ACE's Lissajous orbit has now transitioned to an opening pattern, so the ACE motion will now be counterclockwise about L1 (and for that matter, the Sun) for quite a long time to come. ===================================================================== OCRs Executed: NONE ===================================================================== Ongoing Activities: TAPS: - Installation of TAPS release 4.3 is occurred on 07/20/04. FOT testing of this release is ongoing. ---------------------------------------------------------------------------- Automation (ROBOTT): On Monday 07/26/2004, ROBOTT v0.7 was installed on String 1 and reinstalled on Strings 2 and 3 (OCR GND-139). ROBOTT v0.7 has been used for daily operations since Tuesday 07/27/2004 (OCR GND-140) with ROBOTT in advisory mode. No data loss or command problems due to ROBOTT have occurred this week. Installation on String 2 had been done on Monday 07/19/2004 and testing was performed until Monday 07/26/2004. To date, 14 of the 26 DRs fixed in v0.7 have been verified and 17 new DRs have been written. A review of operations with ROBOTT will be held on Friday 07/30/2004. ============================================================================ Extra Spacecraft Activities: None ===================================================================== Anomalies: DOY 04205 (07/23/04) ACE Anomaly G04-0068 - GDS 26m (DSS-16) TCP / Commanding Problem At 12:25:00, the FOT ran the procedure G_PRECAL. The FOT didn't receive a echo for the test block or NO_OP command. The FOT informed DSS-16 of the problem. DSS-16 stated they had just configured their system and to try again. DSS-16 didn't see the test block or no_op. After an hour of troubleshooting the station dropped their carrier and reloaded their equipment. At 13:25, a NO_OP command was sent and received by the station. IMPACT: Unable to command S/C for an hour. Delayed pass activities. SSR playback incomplete. DR# G104650. DOY 04205 (07/23/04) ACE Anomaly G04-0069 - GDS 26m (DSS-16) C&DH B Not Receiving Commands Commands were correctly received by RcvrA/C&DH A through the spacecraft's Hi-Gain Antenna. But 69 of 70 commands to RcvrB/C&DH B through the spacecraft's BroadBeam Antenna were not accepted. A second support was scheduled with DSS-27 to determine if the problem was with the station or the spacecraft. DSS-27 was able to successfully command to RcvrB/C&DH B through the BroadBeam Antenna and had no problems. IMPACT: Because all of the data indicated a nominal uplink (AGC, VCO, CMD Mod Index, CDU Bit Lock), the possibility of a spacecraft problem was forwarded to management. If a command was not verified by C&DH B, the RF Watchdog would execute 48 hours after the last real-time command and the C&DH would reset 96 hours after the last poke. A postpass DR was requested from DSS-16. DOY 04206 (07/24/04) ACE Anomaly G04-0070 - GDS 34m (DSS-27) Telemetry Outage / DCC Problem At 13:38, the FOT requested DSS-27 put telemetry online. DSS-27 stated data was online, however the FOT was not receiving any data. At 13:30, the FOT informed the TSS of the problem. At 13:50, the TSS confirmed with JPL COMM that data was going to the CDR. At 13:55, the FOT contacted the CD Manager. The CD Manager performed a down/up (rebooted) on port 55550. However, the CD Manager doesn't see any data coming in from JPL. At 14:05, the FOT informed the TSS that telemetry was still offline. The FOT asked TSS to confirm that data is leaving JPL. The TSS stated that data was going to the CDR, however, they couldn't confirm the data was being sent to GSFC. At 14:15, the TSS stated that DSS-27 was switching to a new DCC. At 14:19, telemetry was online. IMPACT: Telemetry outage from 13:40 - 14:19. Pass activities were delayed. The FOT was unable to complete the playback of SSR-B. The remainder of the dump (slices 490 - 601) completed on day 207. DR# G104653 DOY 04207 (07/25/04) ACE Anomaly S04-0005 - SWICS Limit Violation TAPS detected a limit violation for SWICS PAPS -5V. The value was out of limits from 207-17:20:52 to 207-17:21:16. Jim Raines notified. IMPACT: Unknown. DOY 04207 (07/25/04) ACE Anomaly G04-0071 - GDS 26m (DSS-16) Command Modulation Index Problem At 12:44:45, telemetry dropped out from DSS-16. The station was notified and telemetry resumed at 12:47:49. At 12:50:16, during G_INITIAL_CONTACT, commanding was automatically paused by TPOCC due to commands not being accepted by C&DH-B. This same problem occurred on day 205 with DSS-16. At 12:58, the FOT requested that DSS-16 manually reenter the command modulation index. At 13:00, after reentering the command modulation index, a /resume was performed for the procedure G_INITIAL_CONTACT. At 13:00:30, telemetry dropped out again. The station was informed and telemetry resumed at 13:02:34. The station stated they were reconfiguring their system from a test pass that was conducted. The FOT asked if they should hold off commanding until the reconfiguration was complete. The station agreed and stated everything should be complete in 15 minutes. At 13:12, DSS-16 gave the FOT a go to resume commanding. At 13:14, a /resume was entered and the command was accepted by C&DH-B. The pass continued without any further problems. IMPACT: Pass activities delayed. DR#G104655. DOY 04209 (07/27/04) ACE Anomaly S04-0006 - SWICS Limit Violation TAPS detected 6 limit violation for SWICS PAPS +5V. The value was out of limits from 209-04:06:30 to 209-11:55:55. Notified Jim Raines. IMPACT: Unknown DOY 04209 (07/27/04) ACE Anomaly G04-0072 - ROBOTT Command &Control Failed to Start C&C was unable to allocate enough colors for itself which generated dialog boxes. IMPACT: C&C would not have started without manual intervention. DOY 04209 (07/27/04) ACE Anomaly G04-0073 - ROBOTT Paging Not Starting After Installation CRON was unable to start Paging because the file 'page.log' did not exist. IMPACT: No pages sent to FOT. DOY 04209 (07/27/04) ACE Anomaly G04-0074 - C&C/ROBOTT Failovers C&C requests a Failover if it does not see an approved event message within 14 minutes. If no event messages are received within 6 minutes, C&C issues the TSTOL 'shoval' directive which generates an event message. However, C&C does not issue these 'shoval' directives when it is in Advisory Mode. IMPACT: Multiple failovers occurred. DOY 04209 (07/27/04) ACE Anomaly G04-0075 - ROBOTT CRON button Still Green After Being Killed After the third failover today (Ref AR G04-0074), the RTMON issued a message stating that there is no system to failover to and to please help. At this time cron is also killed. The FOT was unaware that cron was down though, because the cron button was still green. The 'crontab -l' directive was needed to actually see that cron was down. IMPACT: Because the color of the button did not change, the FOT was unaware that cron was down until none of the postpass activities were performed at EOT + 10 min. DOY 04210 (07/28/04) ACE Anomaly G04-0076 - GDS 26m (DSS-16) Antenna Problem At 14:01:33 RCVR A and B dropped out of lock. DSS-16 lost uplink to spacecraft. Had transmitter interlock set with warning: Klystron air flow fault detected. At 14:09:37 RCVR's back in lock and good telemetry online. IMPACT: 2 minutes of degradated data while DSS-16 was bringing carrier up. All data was recovered in playbacks. DR# G104665 DOY 04211 (07/29/04) ACE Anomaly G04-0077 - Corrupted DSN Schedule ROBOTT detected a corrupted DSN schedule for Week 32. FOT was unsure which week and day had the corruption because the message from ROBOTT was not descriptive enough. IMPACT: FOT unable to determine reason for the schedule corruption in a timely manner. DOY 04211 (07/29/04) ACE Anomaly G04-0078 - No Redumps Performed by Automation S_ops_plbk_completed needs that argument for whether g_end_day has been run or not, but this is not updated to a "YES" until after g_end_day completes. So even though g_end_day ran before s_ops_plbk_complete, C&C still passed in a "NO" indicating that g_end_day had not run. IMPACT: The end_day_complete variable is used to determine how much time (20min or 5min) is needed between redumps and EOT. There was not enough time to complete the redumps with Automation because the argument was not correct. There were 3 telemetry hits during the pass, 2 of which were captured on the CDR. 1 hit was not captured which resulted in 6 seconds of data loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Glen A. Miller ACE Flight Operations Observatory Engineer Honeywell Technology Solutions Inc. Goddard Space Flight Center Building 14, Room W10A Mail Stop 428.2 Greenbelt, MD 20771 Office: 301-286-7071 Pager: 800-425-3174 Fax: 301-286-1729 gamiller@pop500.gsfc.nasa.gov ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~