ACE Weekly 10/21/2015 - 10/28/2015

All ACE spacecraft subsystems are performing as expected.

========================================================================
Attitude/Orbit:

Type               Attitude
Date             10/28/2015
DOY                301 2015
Thrusters        2R 4R+ 4R-
Duration       6:34 minutes
Start             13:57:18z
Stop              14:03:52z
ra,decStart    220.11,-20.96
ra,decStop     225.14,-24.66
ra,decExpect   225.34,-24.91
Deviation          0.31 deg (-4.8%)
HGAStart          -4.67 deg
HGAStop            8.79 deg
SunStart          10.95 deg
SunStop           16.85 deg
SpinStart        5.0288 rpm
SpinStop         5.0305 rpm
Nutation           0.07 deg
Firing            33 pulses
FuelUsed         0.1067 lbm
FuelRemain      94.7500 lbm
FinalSCMass    1329.010 lbm

Next attitude maneuver: Tuesday, 11/03/2015

========================================================================
OCRs:

None


========================================================================
Activities:

Data Capture: 99.99%  DOY 291-298 2015

A 2-sec gap in playback data (VC4) was recorded during a data re-dump for
the following timeframe: 2015-297:16:18:53 (UT) - 2015-297:16:18:54 (UT).

========================================================================
Anomalies:

--- SOAR S-ACE-38 ¡© ACE Pass Procedure Repeatedly Started

On Saturday 10/24 (DOY 297), beginning at 2015-297-17:25:45 UTC, the FOT
received repeated messages indicating that the ace pass procedure had
restarted due to ACE MUS receiving data but not ITOS.
    
A review of the event log indicated the MMOC was getting ACE low rate data
at the time the problem was recorded. However, for yet unknown reasons,
ITOS did not generate the expected message which correlates to ITOS getting
data (i.e., the  itos_event_do_action tool was looking and expecting this
message as an indicator that ITOS was receiving data).

Since the itos_event_do_action tool did not see that message, the tool
responded accordingly, and restarted the ace_pass procedure. Given that
this wasn't the problem the fix was not successful and repeated attempts
(from itos_event_do_action) followed.

The playback failed to connect from ITOS to frame_sync_port, but the
connection from frame_sync_port to MUS succeeded.

The root cause is still under investigation, but a problem with an
ITOS background process is suspected.
    
IMPACT:  

On-call FOT personnel came in to the MMOC to perform a manual cleanup
(restarted ITOS and the pass procedure, and re-established connection
around 18:40 UTC). Cleanup operations unfolded nominally and the next
contact on DOY 298 worked (nominally) as expected.


--- SOAR S-ACE-39 - NOAA not Receiving Forwarded Data

On Tuesday, 10/27 (DOY 300), during the 300/21:05 UTC pass, NOAA contacted
the FOT and indicated that NOAA was not receiving data from ACE.

In response to the problem, the FOT ran "netstat" and did not see the
expected outgoing connections.  However, running "ps" to obtain the process
status did show the associated processes. The FOT started a single
forwarding process to NOAA, which successfully allowed NOAA to receive
data.

During the same contact, the FOT was not able to get display pages to open
for the ITOS ACE context - however, it did work for the WIND context.

The pass was nominal other than the noted problems. As a last observation,
the FOT found out the attitude sequential print was generated, but the
file did not get copied into the common shared location area.

IMPACT:

Minimal. The FOT initiated single forwarding process to NOAA and queued-up
a re-start of ITOS shortly after the pass ended.