Interpreting QA2 Scores
This page contains information about how to interpret the default QA2 score generated by the VLA Calibration Pipeline. The pipeline gives an overall QA2 score equal to the worst score reached for any step in the pipeline, and under certain circumstances, this default QA2 score may not be a good indication of the overall quality of the calibration: some of these circumstances are described below. Future improvements to the pipeline should eliminate these cases from causing misleading QA2 scores. In all cases, we recommend a careful check of your data and of the calibration derived by the pipeline before using the pipeline's output for science.
Setup and Pointing Scans
Currently, the pipeline will try to calibrate all spectral windows (spws), including those of setup scans. Since the fraction of spws found that have not been calibrated is one of the metrics used to determine an overall QA2 score for any given run of the pipeline, just the inclusion of these setup or pointing scans in an SB can alter the QA2 score negatively without impacting the actual calibration quality. From the "qa2.html" section of the weblog, you may find a link to the QA2 scores for each step of the pipeline: QA2_scores.txt. Looking here may show that the only steps with a Partial or Fail QA2 score are the "QA2_uncalspws" steps. If any of the following sections apply to your data, this could indicate the QA2 score is not a good indication of the overall quality of calibration from the pipeline.
S band with L band setup scan
When observing at S band, we currently recommend the use of a setup scan other than S band during slew time so the system setup is not affected by potentially strong RFI. In this case, one might have 16 S band spws and another 16 spws from some other band that is only "observed" during the first (setup) scan. This causes the pipeline to see 50% of the spws as uncalibrateable, and gives a Fail QA2 score for the uncalspws checks.
Low Frequency 3bit
3bit observations using either C or X band currently require an 8bit setup scan in addition to the 3bit scans. This might result in 64 3bit spws for C band and another 16 8bit spws from the setup scan. The pipeline will find 20% of "observed" spws are uncalibrateable and give a Partial QA2 score for the uncalspws checks.
Pointing Scans
The pipeline will count pointing scans in the fraction of total spws that cannot be calibrated. If there are a small number of spectral windows used for science, the two spws used for pointing might cause either a Parital or Fail QA2 score for the uncalspws checks, and therefore the pipeline as a whole.
Few Spectral Windows
Some observations make use of only one or a few spectral windows to achieve a desired science goal. In these cases, the pipeline may compute a spectral index of 0.0 and/or a SNR of 0.0 as well. This is unrealistic and causes a QA2 score of Fail; however, this failing QA2 score may give no indication of the actual quality of calibration derived.
