VLA CASA Calibration Pipeline **pre 4.7.2**

The VLA calibration pipeline runs on each completed SB separately; there is currently no provision for it running on collections of SBs.  The pipeline relies entirely on correct scan intents to be defined in each SB.  In order for the pipeline to run successfully on an SB it must contain, at minimum, scans with the following intents:

1. a flux density calibrator scan that observes one of the primary calibrators (this will also be used as the delay and bandpass calibrator if no bandpass or delay calibrator is defined)
2. complex gain calibrator scans.

The SB may also contain scans to be used specifically for bandpass and delay calibration, if desired.  However, if multiple fields are defined as bandpass or delay calibrators, only the first one will be used by the pipeline.  Note that a single scan or field may have multiple intents specified.  Scans intended to be used to set attenuators or requantizer gains should have scan intents of setup intent; they must not have scan intents that may result in their being used for calibration.

In overview, the VLA calibration pipeline does the following:

• Loads the data into a CASA measurement set (MS), applies Hanning smoothing to them, and obtains information about the observing set-up from the MS
• Applies online flags and other deterministic flags (shadowed data, end channels of sub-bands, etc.)
• Prepares models for primary flux density calibrators
• Derives pre-determined calibrations (antenna position corrections, gain curves, atmospheric opacity corrections, requantizer gains, etc.)
• Iteratively determines initial delay and bandpass calibrations, including flagging of RFI and some automated identification of system problems
• Derives initial gain calibration, and derives the spectral index of the bandpass calibrator
• RFI flagging is done on data with the initial calibration applied
• Derives final delay, bandpass, and gain calibrations, and applies them to the data
• Runs the RFI flagging algorithm on the target data
• Runs "statwt" to calculate data weights based on the RMS noise
• Creates diagnostic images of calibrators

For more information about the pipeline calibration, please see the slides from the 2016 Data Reduction Workshop describing the scripted pipeline for a general overview; the CASA integrated pipeline performs the same steps, but includes more diagnostic plots than the scripted pipeline.

The VLA pipeline CASAguide provides an example run with a detailed explanations of how to run, interpret and modify the VLA pipeline and its products.

For many Stokes I continuum observations the VLA calibration pipeline will produce well-calibrated data that may only need minor further flagging.  In some circumstances more attention may be required, for example: S-band observations of fields close to the geostationary satellite belt. For more information about additional flagging, please see the CASA Integrated Pipeline Special Topics page.

For more detailed information about each task of the pipeline, you may examine the help file in CASA for each pipeline task. For example, in CASA, you could type:

help hifv_importdata

to see the CASA help file and available options for the import data step of the VLA pipeline. A complete list of tasks used by the VLA (and ALMA) pipeline may be found by typing tasklist at the CASA prompt and looking under the "User defined tasks" for all of the hifv_*, hifa_*, hsd_*, hif_*, and h_* tasks (not all of these are used by the VLA pipeline).

Please submit comments, questions, and suggestions via Pipeline Department of the NRAO Helpdesk.

Obtaining the Pipeline

Current version (CASA-Integrated):

The pipeline is now part of every other CASA release, starting with CASA 4.5.3: please see the Obtaining CASA webpage and download the desired VLA Pipeline version of CASA. The current version is CASA 4.7.2, which uses Pipeline Cycle4-R2-B. Once downloaded, please see the instructions below for running the pipeline.

Scripted pipeline:

This scripted pipeline version may continue to have updates in addition to the integrated versions for those who require greater flexibility to modify their reduction production procedures within the overall calibration framework developed by NRAO. If you are interested in obtaining the scripted pipeline, please see the Scripted Pipeline page.

Running the Pipeline

Stokes I Continuum

To run the CASA Integrated Pipeline on datasets that contain only continuum data (64 MHz or 128 MHz spectral windows):

import pipeline.recipes.hifv as hifv

and then:

hifv.hifv(['mySDM'])

Rerunning the pipeline

After running the pipeline, you may find that additional flagging may be required (resulting in the need to re-derive calibration) or that you may wish to rerun the pipeline again on the raw data with some different parameters for the pipeline. Instructions for both of these cases may be found on the special topics page and in the slides from the 2016 Data Reduction Workshop presentation describing extra flagging.

Special Cases

Incorrect scan intents

Features and limitations

This section will be updated as the pipeline continues to be developed. The comments below apply to the following pipeline versions: r39437 (Pipeline-Cycle4-R2-B).

• The VLA Calibration Pipeline produces a dataset that can be used as a starting point by the user for self-calibration and imaging of science targets.  Note that the pipeline does not do any self-calibration of fields that are not specified as calibrators.

• Although the CASA task "fluxscale" uses medians to do the flux density bootstrapping, which is fairly robust, in some cases (especially at high frequencies, for datasets that may have suffered from pointing or other problems) the default flux density scale may be uncertain.  For projects that may require very accurate flux density scales, it is advisable to run the scripted pipeline.
• The pipeline effectively calibrates each spectral window separately.  This means that the signal-to-noise ratio (s/n) on the calibrators needs to be sufficiently high that solutions can be obtained within the solution intervals specified in the pipeline.  For the delay calibrator, this means a s/n>3 per integration time, t(int); for the bandpass calibrator, a s/n>5 is required for solution times up to 10*t(int); for the gain calibrator a s/n>3 is required for solution times up to 10*t(int), and >5 for a scan average.  Following the guidelines for the strength of gain calibrators as a function of observing bandwidth in the VLA Observational Status Summary for the high frequency end of Q-band will be reasonably safe for all frequencies, although very narrow spectral channels may be problematic.  In these cases the data may need to be calibrated by hand, using polynomial bandpass fitting instead.
• The pipeline is not compatible with P band observations (even if the SB is P band only) due to the lack of models in CASA for P band. For this case, the pipeline would assume a 1 Jy, flat spectral index model making all flux densities derived by the pipeline arbitrary.
• The CASA pipeline is not designed to handle spectral line observations, polarization calibration, or OTF mosaicking, though we are working on adding these capabilities in the future.

Applying Pipeline Calibration to Raw Data

To apply the calibration and flag tables produced by the pipeline, we recommend using the same version of CASA used by the pipeline as well as the same version of the pipeline. The version of both the pipeline and CASA used may be confirmed via the Observation Overview Page, which should be included in the weblog, which is available for download as part of the pipeline products. We recommend starting with a fresh SDM-BDF, though a fresh MS should work, too. There may be small differences in the final result or statistics if online flags were applied when requesting the MS. There is a known issue restoring pipeline calibration with CASA 4.5.3: for more information, please see our known issues page.

In order for the pipeline to work properly, please follow the steps below to prepare your directory and calibration files for application. The pipeline products you should have ready are the raw SDM-BDF, unknown.session_1.caltables.tgz, *.ms.flagversions.tgz, *.ms.calapply.txt, and casa_piperestorescript.py. To ensure everything is correctly placed for the script to apply calibration, please follow these steps:

1. Create a directory where you will work, call it something like "restoration".

mkdir restoration

1. Go into your restoration directory and create three new directories named exactly as follows:

mkdir rawdata
mkdir working
mkdir products

1. Place the raw SDM-BDF into the "rawdata" directory.

   mv /path/to/fresh/data/SDM-BDF rawdata/

2. Place unknown.session_1.caltables.tgz, mySDM.ms.flagversions.tgz, and mySDM.ms.calapply.txt into the "products" directory

   mv *.tgz products/
   mv *.txt products/
   

3. Place the casa_piperestorescript.py file into the "working" directory

mv casa_piperestorescript.py working/

1. Edit the casa_piperestorescript.py file to include "../rawdata/" before the name of the SDM-BDF (mySDM) in the call to hif_restoredata:
2. From the "working" directory, start CASA

   casa-pipe

3. Start the restore script from the CASA prompt:
   

   execfile('casa_piperestorescript.py')

4. Enjoy calibrated data once the process finishes