What Went Wrong?

For normal VLBA correlator operations, the weekly Bulletin A published by the International Earth Rotation and Reference Systems Service (IERS) and available from the US Naval Observatory (USNO) is read by an automatic script that inserts EOP values in the database from which the job generator gathers its required information. The numbers put in the database include predicts and one week's worth of rapid service values. Once a month, the Bulletin A also contains ``final'' values for times at least a month in the past. Those are also put in the data base, but are usually too late for use on the correlator. Ultimately the database includes multiple values of varying quality for each day.

The job script generator is supposed to take the most recently inserted and presumably most accurate value. Instead, starting on May 5, 2003, at the time of a change in the underlying commercial database system, a bug was introduced such that the first value inserted was used. Prior to that change, correlator jobs almost always were based on the rapid service values. Afterwards, they were mostly based on predicts which are of much lower accuracy. Until June 2004, the values for 5 days a week were 4 to 9 day predicts while the other 2 days were 4 and 5 day old rapid service values. After June 2004, the values used were 10 to 16 day predicts, accounting for a significant degradation of the values in mid 2004. The bug was fixed on August 8, 2005 so job scripts prepared after that date should have used rapid service values unless they were prepared too close to the date of observation, before the rapid service values were available (as has happened). All job scripts in the queue at that time were remade. Unfortunately, thanks to a glitch in the automatic reading of Bulletin A, some projects correlated in late September 2005 have poor EOP and will need correction if they use phase referencing.

The correlator uses EOP values from zero hours UT on each of 5 days. Those 5 values are used for a spline fit that is then used to determine the actual value to use for each delay calculation, typically at 2 minute intervals. Thus there is no one value used for each project. Any EOP correction scheme needs to be aware of this and, unless recalculating the total model, be able to duplicate the correlator's interpolation. In fact, the same algorithm should be used so that it will deal with events like steps in the data in the same way as the correlator. For example, after mid 2004, once a week there is a step when jumping from a 16 day predict to a noticably better 10 day predict. Prior to mid 2004, some rapid service values are mixed with the predicts, which also introduced steps.

While investigating the EOP problem, it became apparent that a few observations from before May 2005 and also since the fix in August 2005 also used poor EOP. The reason for this is the jobs were prepared before the rapid service values for the required days were put into the data base. Between the once-a-week updates of Bulletin A and the need for 3 days of EOP beyond the observe date, jobs prepared up to 9 days after observing can end up using predicted EOP values. A suggested change to avoid this is presented in Section 10.