Correcting Data - The Long Version with Full Explanations

There are two methods, in principle, to correct existing data. VLBA data sets contain a record of the total delay model used on the correlator. That record is in the form of polynomials in the IM (Interferometer Model) and CL tables. A very good way in principle to correct any issues with the correlator model is to simply calculate a new total delay model and adjust the data by the difference between that result and the correlator model. That is the scheme used for geodesy and that is why geodesy observations have not been affected by the EOP error. Currently there is no capability to make this style of correction in AIPS. This incident has added to the motivation to add one.

The second method is to calculate the difference between a delay model made using the same EOP that were used in correlation and one that uses improved EOP. The difference is then applied to the data. As long as, other than the EOP, the two models are identical, the difference is not very sensitive to the quality of the overall delay model so a fairly crude model can be used. This is the scheme implemented in CLCOR.

In order to correct for poor EOP, CLCOR uses information from internal AIPS tables, from user input, and from an external file that the user needs to download. First, the EOP actually used on the correlator must be determined and getting that right has revealed some issues with the interaction between the correlator software and the correlator job generator. It has also revealed a weakness, recently corrected, in the correlator model parameters (CT) table in AIPS. The vast majority of users will not be affected by these issues but CLCOR needs to deal with them so they are described here.

Figure 9: The correlator gets one set of EOP numbers each day. For each delay calculation, it interpolates to the actual time using a spline fit. This plot shows the spline fit for the UT1-UTC values used in the August 13 test observations for the correlator pass with ``poor'' EOP. The actual observations spanned a several hour period near 1 day after the reference day.

The correlator obtains its Earth Orientation Parameters from the ``UTC Table'' in a correlator job script. There are typically a few to a few tens of job scripts needed to cover the whole time range of a project. The UTC Table has values for UT1-UTC, Xpole, and Ypole for midnight UT of several days. The correlator reads the first 5 such days, making an assumption that there are only 5 present and that those 5 are the ones it actually wants. The values of the parameters can change significantly from day to day (up to about 1 ms = 15 mas for UT1-UTC) so the correlator interpolates between the provided values to get a value for each individual delay calculation. That interpolation is done with a spline fit. An example of such a spline fit is shown in Figure 9. These are the ``poor'' EOP points from the August 13 test and include a step of the sort that can be induced by the presence of 10-16 day predicted values.

Complications for CLCOR occur when a project crosses midnight. For projects that cross midnight, the job generator provides EOP for more than 5 days for some or all jobs, including those that apply to times before midnight. Usually a project just crosses midnight once and most or all jobs have EOP from 6 days. But there are very occasional projects with more days that have more entries (One late 90's case with 12 has been seen). The correlator always takes the first 5 days of EOP data in the job, regardless of how many days data are there or when scans times are relative to the EOP rows. So to determine what values the correlator used, CLCOR must determine, for any given time, what day's EOP were in the job script covering that time. CLCOR is able to do that for essentially all cases without user help.

In VLBA correlator distribution data, there is usually (maybe always) one file per correlator job. The values of EOP used are in the CT table in that file. When the data are loaded with FITLD, and the distribution files (jobs) are concatenated, only one CT table is written. The rows of EOP data from each of the input jobs are concatenated. In most cases, where the rows from each of the jobs are identical, that means that the CT table contains many rows that repeat the same information. But there are exceptions. It appears that in jobs for times well past midnight (details still a bit unclear), the row for the first day covered in earlier jobs can be dropped. The correlator model for that job then will not be based on the same EOP values used in the earlier jobs. So the routine doing corrections needs to know when this happens.

Unfortunately, the original version of the CT table (before 15 Sept 2005), which simply copies the information from the distribution files, does not include a time range for which each row applies. That time range is obvious from other information for the individual jobs, but was lost once FITLD concatenates the files. The only record of the time ranges of each job that AIPS had was in the history records FITLD writes, which are not really meant for other AIPS tasks to read. As of mid September, 2005, FITLD has been modified to write a new varient of the CT table that includes the time range to which each row applies. Thus for newly loaded data, it will be easy for CLCOR to tell what rows should be used in corrections.

When run CLCOR will attempt to figure out which CT rows to use for the interpolation to each CL table row time. For most observations, all sets of CT table rows are the same. Sets are rows from a single correlator job. They are easily distinguished from other sets by a drop in time between successive CT table rows when the new set starts. CLCOR checks if all sets are the same and, if so, simply uses the first 5 rows for correcting the entire data set. If they are not the same, CLCOR uses the time range column for new type CT tables or, for the old type, reads the history records from FITLD and uses the time information there to assign times to each CT table row. Note that, in the case of non-identical sets, getting the time information right depends on the structure of the CT table relating nicely to the history records from FITLD. If a post-FITLD concatenation operation such as DBCON or VBGLU has been done, this might be problematic and the user should be very alert as to what is going on.

Once CLCOR has the proper 5 days of EOP data, it interpolates to the time of the CL table entry using the same code that is used by the correlator. Thus the interpolation should be accurate to high precision.

There are a small number of projects from years ago that extended over several days. The job scripts contained EOP data from many days -- up to 12 in a standout case. Since the correlator uses the first 5 days EOP regardless, it is likely that later days in the observation were correlated with EOP values actually extrapolated off the end of the time range of the values provided. Spline fits are notoriously poor at extrapolation. After only 3 days or so from the last point used, the EOP would be very poor. Any user with data from many days should look very closely at their situation and it might be best to use a full model recalculation to make corrections.

CLCOR also needs to know the desired ``good'' values of EOP. It is set up to read those from an external file, mentioned earlier, that can be downloaded from http://gemini.gsfc.nasa.gov/solve_save/usno_finals.erp. It is easiest to just feed this file to CLCOR (parameter INFILE) without modification, although it can be edited if you desire. If it is edited, beware of the somewhat odd units of tenths of arcseconds for X and Y and microseconds for UT1-IAT (not the UT1-UTC normally given). Those units must not be changed. The first column is the Julian Day number. Alternate inputs for desired EOP are not yet implemented.

Data correlated with poor EOP has delay and phase offsets that vary with position on the sky. Any calibration that uses the measured phases and delays of a source to derive a correction for that source will remove the effect of the EOP errors. Such calibrations include fringe fitting and, for phases, self-calibration. This is why projects for which all sources are fringe-fitted and/or self-calibrated are not affected by the EOP error. It is also why some calibration steps, like bandpass calibration and pulse cal calibration, which are typically based on fringe fitted sources, are not affected. Where the corrections are needed is in situations where the corrections for one source are applied to another source at a different location. Then the relative delay in the model must be accurate and the EOP corrections need to be applied if poor EOP were used.

When the EOP correction is applied, it will change delays and phases on all sources. If it is applied to a phase calibrator whose phases have been forced to zero already by fringe fitting or self calibration, it will push those phases away from zero. Another round of fringe fitting or self calibration would need to be done to bring them back to zero to preserve the calibator position. All such fringe fitting and self calibration results, from before and after the EOP corrections, need to be applied to the target source(s) too.

It is probably best to apply the EOP corrections to a CL table that does not contain any data based delay or phase calibrations -- that has not been fringe fitted or self calibrated. Such a CL table can contain, for example, the correlator offset corrections (ACCOR), parallactic angle corrections (also CLCOR), absolute gain calibration (APCAL), and ionospheric calibration (TECOR). Flagging will not be affected. Fringe fit and/or self calibration will need to be redone after EOP corrections regardless so it is best to do them only after the corrections. It is also probably best to do pulse cal calibrations (PCCOR) and bandpass calibrations (BPASS), both of which depend on data phases, after the EOP corrections, although that is not strictly necessary because they almost certainly will use fringe fitted sources whose final calibrations are not changed by the EOP corrections.