VLA Calibration Pipeline 2023.1.0.124; CASA version 6.5.4

General Description

The VLA calibration pipeline performs basic flagging and calibration using CASA. It is currently designed to work for Stokes I continuum data (except P-band and 4-band), but can work in other circumstances as well. Each Scheduling Block (SB) observed by the VLA is automatically processed through the pipeline. The VLA calibration pipeline performs the following steps:

Loads the data in from the archival format (Science Data Model-Binary Data Format [SDM-BDF]) and converts to a CASA MeasurementSet (MS), applies Hanning smoothing (when the data were not frequency averaged by the VLA correlator), and obtains information about the observing set-up from the MS;
Applies online flags (generated during observing) and other deterministic flags (shadowed data, edge channels of sub-bands, etc.);
Prepares models for primary flux density calibrators;
Derives pre-determined calibrations (antenna position corrections, gain curves, atmospheric opacity corrections, requantizer gains, etc.);
Calculates level of gain compression that is present.
Iteratively determines initial delay and bandpass calibrations, including flagging of RFI (Radio Frequency Interference) and some automated identification of system problems;
Derives initial gain calibration, and derives the spectral index of the bandpass calibrator;
RFI flagging is done on data with the initial calibration applied;
Derives final delay, bandpass, and gain/phase calibrations, and applies them to the data;
Runs the RFI flagging algorithm on the target data;
Calculates data weights based on the inverse square RMS noise of the MS;
Creates diagnostic images of calibrators.

For more information about pipeline heuristics, please see the pipeline presentations from the 2021 VLA Data Reduction workshop.

The scripted pipeline is no longer being updated, but is still available with older CASA versions; the most recent version is only guaranteed to work with CASA 5.3. If you are interested in obtaining the scripted pipeline, please see the Scripted Pipeline page.

Please see the Release Notes and Known issues below; the pipeline reference manual is also available. For questions, suggestions, and other issues encountered with the pipeline, please submit a ticket to the Pipeline Department of the NRAO Helpdesk.

VLA Imaging Pipeline

An imaging pipeline has been developed for VLA continuum data, building on the imaging pipeline developed for ALMA. The imaging pipeline will produce single continuum images of each target field per observed band. For instructions of how to run the imaging pipeline please see the VLA Imaging Pipeline webpage.

VLA Pipeline 2023.1 (CASA 6.5.4) Release Notes

More fine-grained control of the application of gain compression correction is available via the hifv_syspower task is possible. The correction can now be turned on and off on a per-band basis, essential for supporting multi-band setups.
Plots are now produced in hifv_syspower for K, Ka, and Q-band. The normalized plots of Pdiff can give an indication of how far off the flux density scale might be for a given target. The syspower corrections should not be applied to 3-bit data.
Antenna position corrections are now applied when Bx=By=0 and Bz!=0.
Addition of hif_selfcal task to the VLA imaging pipeline for automated science target self-calibration
See also the ALMA pipeline documentation for general, non-VLA specific infrastructure changes.

Obtaining the Pipeline

The pipeline is released and included with CASA. The Obtaining CASA webpage has links to the CASA version with the most recent VLA pipeline and provides access to older versions. Follow these links to obtain CASA 6.5.4 with the pipeline for Linux systems and MacOS 11.0 systems.

Please note that pipeline operation on MacOS systems is not officially supported and may not produce reliable results and/or results consistent with runs performed in Linux.

Pipeline Requirements

The VLA calibration pipeline runs on each completed SB (typically a single SDM-BDF) separately; there is currently no provision for it running on collections of SBs.
The pipeline relies on correctly set scan intents. We therefore recommend that every observer ensures that the scan intents are correctly specified in the Observation Preparation Tool (OPT) during the preparation of the SB (see OPT manual for details). Typical intents are:
- CALIBRATE_FLUX (required): flux density scale calibration scans (toward one of the standard VLA calibrators 3C48, 3C138, 3C147, or 3C286), the pipeline will use the first field with this intent. If this intent is not present, the pipeline will fail
- CALIBRATE_AMPLI and CALIBRATE_PHASE (required): temporal complex gain/phase calibration; if these intents are not present, the pipeline will fail
- CALIBRATE_BANDPASS (optional, but highly recommended): scan that is used to obtain the bandpass calibration (only the first instance of CALIBRATE_BANDPASS is used regardless of the band; multi-band scheduling blocks cannot use different bandpass calibrators for different bands); if not present, the first field with scan intent CALIBRATE_FLUX will be used for bandpass calibration
- CALIBRATE_DELAY (optional): delay calibrator scan; if not present the first scan with a CALIBRATE_BANDPASS intent is used for delay calibration, and if that one is not available delays are calculated using the first CALIBRATE_FLUX scan.
The pipeline also currently requires a signal-to-noise of >~3 for each spectral window of a calibrator per integration (for each channel of the bandpass).

Automatic Processing for Science Observations

Starting with Semester 2013A D-configuration, a version of the calibration pipeline has been run automatically at the completion of all astronomical scheduling blocks, except for P-band and 4-band observations, with the resulting calibration tables and flags archived for future use. At NRAO we execute the standard pipeline optimized for Stokes I continuum processing, independent of the actual science goal. As such, it may be necessary to re-run the pipeline with modifications depending on the science goal.

Investigators are notified when the calibrated data are ready for download; detailed quality assurance checks are performed on a subset of data via the SRDP project (see next section) or can be performed by NRAO staff upon request. Users will request their calibrated data from the NRAO archive, which will automatically produce the calibrated MS by applying the saved calibration and flag tables. See the section "Restore calibration from archived products" for details on the restoration procedures. Questions regarding the availability of pipeline-processed data should be submitted to Pipeline Department of the NRAO Helpdesk.

Science Ready Data Products (SRDP) Processing

Starting in June 2019, a subset of VLA projects will undergo a more rigorous quality assurance process as part of the NRAO SRDP Initiative. As of September 2023, SRDP processing is being done on most continuum-only projects observing at C-band and higher frequencies (single and multi-band data). Spectral line projects and those that used 3C138 and 3C48 as flux density calibrators are excluded from SRDP processing. Such projects are still run through the pipeline, but do not receive detailed quality checks.

SRDP deliveries may to expand to include S-band and possibly L-band now that gain compression correction is available in the standard pipeline. Users will be delivered calibrated data that may have additional flagging applied and are expected to be ready for imaging by the user.

Running the Pipeline

The pipeline can take a few hours to a few days to complete depending on the specifics of the observation; ensure that your process can run for the full duration without interruption. Also, make sure that there is enough space in your directory as the data volume will increase by about a factor of four. A display environment must be set as well: use xvfb or some other virtual display environment if you will run the pipeline as a batch job. There are several ways to run the pipeline and, in most cases, we recommend starting with the raw data (an SDM-BDF) that can be requested from the NRAO archive. Place your data (preferably the SDM-BDF, but an MS is also acceptable) in its own directory for processing. For example:

#In a Terminal
mkdir myVLAdata
mv mySDM myVLAdata/
cd myVLAdata

Next, start CASA from the same directory where you placed your SDM-BDF. Note: do not try to run the pipeline from a different directory by giving it the full path to a dataset, as some of the CASA tasks require the MS to be co-located with its associated gain tables. And do not try to run the pipeline from inside the SDM-BDF or MS directories themselves. While SDM-BDFs and MSs are directories, they should always be treated as single entities that are accessed from the outside. It is also important that a fresh instance of CASA is started from the directory that will contain the SDM-BDF or MS, rather than using an existing instance of CASA and using "cd" to move to a new directory from within CASA, as the output plots will then end up in the wrong place and potentially overwrite your previous pipeline results.

Starting CASA with Pipeline Tasks

To start CASA with the pipeline from your own installation type:

#In a Terminal
casa --pipeline

Note that starting CASA without the --pipeline option will start CASA without any of the pipeline-specific tasks. (In turn, if you try to use 'casa --pipeline' for manual data reduction, plots that are using matplotlib, like plotants will not work; use 'casa' without --pipeline for manual data reduction.)

If you are at the Domenici Science Operations Center (DSOC) or using NRAO computing resources at the DSOC, we provide a shortcut to the latest CASA version that includes the pipeline. To start this version, type:

#In a Terminal
casa-pipe

To list other versions of CASA with the pipeline that are available at the DSOC, type:

#In a Terminal
casa-pipe -ls

To start a particular version, type:

#In a Terminal
casa-pipe -r <full text from version list>

Now that CASA is open, there are several ways to start the pipeline.

The CASA homepage has more information on using CASA at NRAO.

Method 1: casa_pipescript.py

The pipeline can be run using a 'casa_pipescript.py' file, the provided example (see example below) may be used, or a script from any previous pipeline run using the same CASA version. The casa_pipescript.py will need to be edited to add the intended SDM-BDF (or MS) name and executed in CASA. For this to work, the 'casa_pipescript.py' must be in the same directory where you start CASA with the pipeline. Once CASA is started (same steps as above) type:

#In CASA
execfile('casa_pipescript.py')

Also see the Imaging pipeline section for running the Calibration pipeline with Target Imaging.

Method 2: Recipe Reducer

The pipeline comes with several predefined sets of instructions, called recipes, to accommodate for different needs: one for VLA data, one for ALMA data, one for VLASS, etc.

# In CASA
import pipeline.recipereducer
pipeline.recipereducer.reduce(vis=['mySDM'],procedure='procedure_hifv.xml',loglevel='summary')

The upside to this method is that an exact reference to your CASA path and pipeline directory are not needed. But the down side is that this method will store the pipeline weblog in a directory called pipeline-procedure_hifv unlike the other methods described above and below that will create a unique directory name based on the date and time. Thus, one has to be cautious against running this procedure multiple times in the same directory. This method also includes a call to hifv_exportdata, which will package the calibration products into a 'products' directory one level up; this may not be desired by all users.

Method 3: Python Pipeline recipes

Deprecation Warning: The Python recipes may be removed in a future pipeline release.

The pipeline comes with several predefined sets of instructions, called recipes, to accommodate for different needs: one for VLA data, one for ALMA data, one for VLASS, etc. Once started, the pipeline does not require any further input from the user and should run to completion automatically. The default VLA recipe for Stokes I continuum is called hifv which stands for 'heuristics, interferometry, VLA'. To use this VLA data recipe, you first need to import it into CASA via the following command from the CASA prompt. Note that how to do this has changed in CASA 6 and the steps used with previous CASA versions will no longer work. While 'execfile' does not formally exist in Python 3 its functionality has been implemented in CASA 6.

# In CASA
execfile("/my/path/casa-6.5.4-9-pipeline-2023.1.0.124/pipeline/pipeline/recipes/hifv.py")

and then start the pipeline via:

# In CASA
hifv(['mySDM'])

where mySDM is the name of the SDM-BDF (or MS) from your observations.

Method 4: One Stage at a Time

You may notice that the 'casa_pipescript.py' is a list of specific CASA pipeline tasks being called in order to form the default pipeline. If desired, one could run each of these tasks one at a time in CASA to inspect intermediate pipeline products as an example.

If you need to exit CASA between stages, you can restart the pipeline where you left off. However, in order for this to work, none of the files can be moved to other directories.

First, use the CASA pipeline task h_resume after starting CASA again. This will set up the environment again for the pipeline to work. Type:

# In CASA
h_resume()

Now, you may start the next task in your list.

What you get: Pipeline Products

VLA pipeline output includes data products such as calibrated visibilities, a weblog, and all calibration tables. Note that the automated execution at NRAO will also run an additional data packaging step (hifv_exportdata) that moves most of the files to an upper level '../products' directory. This step is omitted in the manual execution and all products remain within the 'root' directory where the pipeline was executed.

The most important pipeline products include (mySDM is a placeholder for the SDM-BDF name):

A MeasurementSet (MS) 'mySDM.ms' with applied flags and calibrated visibilities in the CORRECTED_DATA column that can be used for subsequent imaging (root directory).
A weblog that is supplied as a compressed tarball weblog.tgz. When extracted, it has the form pipeline-YYYYMMDDTHHMMSSS/html/index.html, where the YYYYMMDDTHHMMSSS stands for the pipeline execution time stamp (multiple pipeline executions will result in multiple weblogs). The weblog contains information on the pipeline processing steps with diagnostic plots and statistics. An example is given in the VLA pipeline CASA guide. During execution of the pipeline, this same directory will be present within the working directory. If the pipeline was run using the recipe reducer method, the weblog directory will have the form pipeline-procedure_hifv/html/index.html.
Calibrator images for each band (files start with 'oussid*' in the root directory).
All calibration tables and the 'mySDM.ms.flagversions' directory that contains various flag backups made at various stages of the pipeline run (see section Calibration Tables).
The casa-YYYYMMDD-HHMMSS.log CASA logger messages (in pipeline-YYYYMMDDTHHMMSSS/html/).
'casa_pipescript.py' (in pipeline-YYYYMMDDTHHMMSSS/html/), the script with the actual executed pipeline heuristic sequence and parameters. This file can be used to modify and re-execute the pipeline (see section The casa_pipescript.py file).
'casa_commands.log' (in pipeline-YYYYMMDDTHHMMSSS/html/), which contains the actual CASA commands that were generated by the pipeline heuristics (see section The casa_commands.log file).
The output from CASA's task listobs is available at 'pipeline-YYYYMMDDTHHMMSSS/html/sessionSession_default/mySDM.ms/listobs.txt' and contains the characteristics of the observations (scans, source fields, spectral setup, antenna positions, and general information).
As previously mentioned in Automatic Processing for Science Observations, calibrated MSs for data since late-2016 can be generated automatically using the NRAO archive. For older data, the archived calibration needs to be re-applied to the raw data that have to be downloaded in the SDM-BDF format from the archive (more ideally, older data should be recalibrated using the current pipeline). To prepare for this restoring procedure, NRAO adds the hifv_exportdata pipeline task as a last step. This task packages calibration tables in 'unknown.session_1.caltables.tgz' and flag backups in 'mySDM.ms.flagversions.tgz'. An additional text file, 'mySDM.ms.calapply.txt', is also produced by hifv_exportdata that CASA task applycal uses when restoring the calibration. The restoring process itself is performed by a script called 'casa_piperestorescript.py'. See the section Restore calibration from archived products for details on the restoration procedures.

The Pipeline Weblog

Information on the pipeline run can be inspected through a weblog that is launched by pointing a web browser to file:///<path to your working directory>/pipeline-YYYYMMDDTHHMMSSS/html/index.html. The weblog contains statistics and diagnostic plots for the SDM-BDF as a whole and for each stage of the pipeline. The weblog is the first place to check if a pipeline run was successful and to assess the quality of the calibration.

An example walkthrough of a pipeline weblog is provided in the VLA Pipeline CASA guide.

Note that we regularly test the weblog on Firefox. Other browsers may not display all items correctly.

Calibration Tables

The final calibration tables of the pipeline are (where mySDM is a placeholder for the SDM-BDF name):

mySDM.ms.hifv_priorcals.s5_3.gc.tbl : Gaincurve
mySDM.ms.hifv_priorcals.s5_4.opac.tbl : Opacity
mySDM.ms.hifv_priorcals.s5_5.rq.tbl : Requantizer gains
mySDM.ms.hifv_priorcals.s5_6.ants.tbl : Antenna positions (if created)
mySDM.ms.hifv_finalcals.s13_2.finaldelay.tbl : Delay
mySDM.ms.hifv_finalcals.s13_4.finalBPcal.tbl : Bandpass
mySDM.ms.hifv_finalcals.s13_5.averagephasegain.tbl : Average Phase offsets
mySDM.ms.hifv_finalcals.s13_7.finalampgaincal.tbl : Flux calibrated Temporal Gains 
mySDM.ms.hifv_finalcals.s13_8.finalphasegaincal.tbl : Temporal Phases

The casa_pipescript.py File

The pipeline sequence of the pipeline heuristic steps are listed in the 'casa_pipescript.py' script that is located in the pipeline-YYYYMMDDTHHMMSSS/html (where YYYYMMDDTHHMMSSS is the timestamp of the execution) directory. A typical 'casa_pipescript.py' has the following structure (where mySDM is again a placeholder for the name of the SDM-BDF raw data file and will have the name of the one that was processed):

# This CASA pipescript is meant for use with CASA 6.5.4 and pipeline 2023.1.0.124
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
try:
    hifv_importdata(vis=['mySDM'])
    hifv_hanning()
    hifv_flagdata(hm_tbuff='1.5int', fracspw=0.01, intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*')
    hifv_vlasetjy()
    hifv_priorcals()
    hifv_syspower()
    hifv_testBPdcals()
    hifv_checkflag(checkflagmode='bpd-vla')
    hifv_semiFinalBPdcals()
    hifv_checkflag(checkflagmode='allcals-vla')
    hifv_solint()
    hifv_fluxboot()
    hifv_finalcals()
    hifv_applycals()
    hifv_checkflag(checkflagmode='target-vla')
    hifv_statwt()
    hifv_plotsummary()
    hif_makeimlist(intent='PHASE,BANDPASS', specmode='cont')
    hif_makeimages(hm_masking='centralregion')
    #hifv_exportdata()
finally:
    h_save()

(Note that executions at NRAO may show small differences, e.g., an additional final hifv_exportdata (commented out in example above) step that packages the products to be stored in the NRAO archive.)

The above is, in fact, a standard user 'casa_pipescript.py' file for the current CASA and pipeline version (download to edit and run yourself) that can be used for general pipeline processing after inserting the correct mySDM filename in hifv_importdata.

The pipeline run can be modified by adapting this script to comment out individual steps, or by providing different parameters (see the CASA help for the parameters of each task). The script can then be (re-)executed via:

# In CASA
execfile('casa_pipescript.py')

We will use this method later for an example where we modify the script for spectral line processing.

General modifications to the script include 'h_init(weblog=False)' for faster processing without any weblog or plotting.

The casa_commands.log File

casa_commands.log is another useful file in pipeline-YYYYMMDDTHHMMSSS/html (where YYYYMMDDTHHMMSSS is the timestamp of the pipeline execution) that lists all the individual CASA commands that the pipeline heuristics (hifv) tasks produced. Note that 'casa_commands.log' is not executable itself, but contains all the CASA tasks and associated parameters to trace back the individual data reduction steps.

Restore calibration from archived products

To apply the calibration and flag tables produced by the pipeline, we recommend using the same version of CASA used by the pipeline as well as the same version of the pipeline. The version of both the pipeline and CASA used may be confirmed via the main "Observation Overview Page", the home page of the weblog. We recommend starting with a fresh SDM-BDF, although a fresh MS should work, too. There may be small differences in the final result or statistics if online flags were applied when requesting the MS.

In order for the pipeline to work properly, please follow the steps below to prepare your directory and calibration files for application. In addition to the raw SDM-BDF you will need the following pipeline products that are included in the calibration tarball from the NRAO archive: 'unknown.session_1.caltables.tgz', 'mySDM.ms.flagversions.tgz', 'mySDM.ms.calapply.txt', 'unknown.pipeline_manifest.xml', and 'casa_piperestorescript.py' (where mySDM is a placeholder for the SDM-BDF name). To ensure everything is correctly placed for the script to apply calibration, please follow these steps:

Create a directory where you will work, call it something like "restoration".
```
mkdir restoration
```
Go into your restoration directory and create three new directories named exactly as follows:
```
mkdir rawdata working products
```
Place the raw SDM-BDF into the "rawdata" directory.
```
mv /path/to/fresh/data/mySDM rawdata/
```
Place 'unknown.session_1.caltables.tgz', 'mySDM.ms.flagversions.tgz', 'unknown.pipeline_manifest.xml', and 'mySDM.ms.calapply.txt' into the "products" directory:
```
mv *.tgz products/
mv *.xml products/
mv *.txt products/
```
Place the 'casa_piperestorescript.py' file into the "working" directory:
```
mv casa_piperestorescript.py working/
```
The 'casa_piperestorescript.py' looks similar to the 'casa_pipescript.py', but runs a special hifv_restoredata pipeline task to apply flags and calibration tables, followed by hifv_statwt. Edit the hifv_restoredata call to include "../rawdata/" in front of the name of the SDM-BDF (mySDM), e.g.:
```
__rethrow_casa_exceptions = True
h_init()
try:
  hifv_restoredata (vis=['../rawdata/mySDM'], session=['session_1'],\
                   ocorr_mode='co',gainmap=False)
  hifv_statwt()
finally:
  h_save()
```
From the "working" directory, start CASA:
```
casa --pipeline
```
or if you use computers at the DSOC:
```
casa-pipe
```

Start the restore script from the CASA prompt:

# In CASA
execfile('casa_piperestorescript.py')

Enjoy calibrated data once the process finishes.

Note that if the original pipeline was run in parallel mode, a multi-MS (MMS) is created. The restore will only work for MMSs if the data has the same shape. More specifically, the restoration of the flags can only be achieved if the MSs have identical structure. All production pipeline runs at NRAO are serial.

Quality (QA) Assessment Scores

Each pipeline stage has a quality assessment (QA) score assigned to it. The values range from 0 to 1 where

0.9-1.0 Standard/Good (green color)
0.66-0.90 Below-Standard (blue color; also shown as a question mark symbol)
0.33-0.66 Warning (amber color; cross symbol)
0.00-0.33 Error (red color; cross symbol)

We recommend that all pipeline stages and the relevant products are checked. Below-standard and Warning scores should receive extra scrutiny. These scores often refer to large fractions of flagging that are applied to the data, but may also point out other issues where in fact additional flagging may be required. The QA section at the bottom of the weblog of each stage will provide more information about the particular origin of each score. Errors are usually very serious issues with the data or processing and should be resolved in any case.

Examples for QA scores are provided in the Pipeline CASAguide.

Further flagging

Using a flagging template

Although the pipeline attempts to remove most RFI, there are still many cases where additional flagging is required. The pipeline will then be re-started with the additional flags pre-applied.

The best way to do so is to inspect the data and to record all the required flags in a flagging template. See the CASA flagdata task help for the details on the format. Here is an example for a template:

mode='manual' scan='1~3' reason='bad_data'             #flags scans 1 to 3
mode='clip' clipminmax=[0,10] reason='flag_outliers'   #flags data outside an amplitude range
                                                        #here all amplitudes larger than 10 Jy
# this line will be ignored
mode='quack' scan='1~3,10~12' quackinterval=1.0 reason='remove_starting_integrations'  #removes first second 
                                                                                       #from scans 1-3 and 10-12

The most important modes are manual to flag given time ranges, antennas, spws, scans, fields, etc., and clip to flag data exceeding threshold amplitude levels.

Flagging templates can be saved in text files with any given name, e.g. 'myflags.txt'. In 'casa_pipescript.py' modify the parameters of the hifv_flagdata task; change template=True and add filetemplate='myflags.txt'. It is important to provide the reason field, otherwise the flags may not be applied. The reason field cannot have spaces, but underscores are ok.

 hifv_flagdata(hm_tbuff='1.5int', flagbackup=False, scan=True, fracspw=0.01, \
  intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, \
  *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*',\
  template=True, filetemplate='myflags.txt')

The default for filetemplate is 'mySDM.flagtemplate.txt'. Therefore, flag files with this name would not require filetemplate to be specified.

Interactive Flagging

For some data it is not straightforward to derive flagging commands that can be placed in a template. In that case, one may use the interactive plotms or viewer/msview CASA GUIs to flag data directly in the MS. Re-execution of the pipeline (via the casa_pipescript.py file) will be possible, but a few steps require attention:

Since Hanning smoothing was likely performed in the initial pipeline run, one should turn off Hanning smoothing for all re-executions. Otherwise the frequency resolution degrades more and more and flags will be extended to neighboring channels by smoothing an already flagged MS. To do so, comment out hifv_hanning in the 'casa_pipescript.py' file (see the section on "Spectral Line" for a similar example).
By default, the pipeline will always revert back all flags to their original state that are saved in the 'mySDM.ms.flagversions' file. It will thus ignore all modifications that were made afterwards. To avoid resetting all flags, one should manually flag the MS and place it in a new directory. Do NOT copy over the related 'mySDM.ms.flagversions' directory. Then run the pipeline with the flagged MS as input to hifv_importdata via the modified 'casa_pipescript.py' file (it is possible to also run hifv.hifv(['MeasurementSet']) but remember that this will repeat Hanning smoothing). With this procedure the pipeline will not be able to recover original flags and will proceed with the manual, interactive flags that the user has applied directly to the MS.

Flag the Flux Calibration Gain Table

Sometimes the gain solutions of the flux calibrator are not good for all antennas, even with the rejection performed by hifv_fluxboot. It is possible to flag the solutions directly in the 'fluxgaincal.g' calibration table. The CASA task plotms can be used for the flagging. The pipeline can use the flagged table, say 'fluxgaincal_edited.g' for the flux calibration by modifying the hifv_fluxboot call in the 'casa_pipescript.py' as follows:

hifv_fluxboot(caltable='fluxgaincal_edited.g')

Avoiding a Specific Reference Antenna

In some cases, the pipeline may choose a reference antenna that is not ideal for calibration purposes. The pipeline algorithm picks an antenna (actually a ranked list of antennas) that is not heavily flagged and that is close to the center of the array. Other factors, e.g. phase jumps, or bad deformattors that are not caught in the hifv_testBPdcals stage, may still be present on the reference antenna and will then be reflected on all solutions. When this happens, it is advisable to tell the pipeline not to use a specific antenna as a reference antenna. This can be achieved by the parameter refantignore that is available in some hifv tasks. E.g. if we want to avoid that the pipeline uses antenna 'ea28' as a reference antenna, the casa_pipescript.py can be modified as follows (changes are shown in red in hifv_testBPdcals, hifv_semiFinalBPdcals, hifv_solint, hifv_fluxboot, hifv_finalcals):


# This CASA pipescript is meant for use with CASA 6.5.4 and pipeline 2023.1.0.124
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
try:
 hifv_importdata(vis=['mySDM'])
 hifv_hanning()
 hifv_flagdata(hm_tbuff='1.5int', fracspw=0.01, intents='*POINTING*,*FOCUS*,\
               *ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, *SYSTEM_CONFIGURATION*,\
               *UNSPECIFIED#UNSPECIFIED*')
 hifv_vlasetjy()
 hifv_priorcals()
 hifv_syspower()
 hifv_testBPdcals(refantignore='ea28')
 hifv_checkflag(checkflagmode='bpd-vla')
 hifv_semiFinalBPdcals(refantignore='ea28')
 hifv_checkflag(checkflagmode='allcals-vla')
 hifv_solint(refantignore='ea28')
 hifv_fluxboot(refantignore='ea28')
 hifv_finalcals(refantignore='ea28')
 hifv_applycals()
 hifv_checkflag(checkflagmode='target-vla')
 hifv_statwt()
 hifv_plotsummary()
 hif_makeimlist(intent='PHASE,BANDPASS', specmode='cont')
 hif_makeimages(hm_masking='centralregion')
finally:
 h_save()

Gain Compression Correction

The pipeline now allows for gain compression correction to be applied for VLA bands using the hifv_syspower task. This task was originally developed for the VLA Sky Survey and has been adapted to work with general VLA observations. The corrections should absolutely not be applied to data taken with the 3-bit samplers. The corrections are not recommended for K, Ka, and Q-band data. The plots created for 3-bit data and K, Ka, and Q-bands are informational only, and can inform whether the flux scale may be significantly inaccurate (variable power differences (Pdiff) throughout observations or systematic differences between the flux calibrator and the complex gain calibrator). The Pdiff measurements are normalized relative to the measurements taken on the flux density calibrator.

Receiver gain compression occurs when the input power of an amplifier is increased to a level that reduces the gain of the amplifier and causes a nonlinear increase in output power. This can typically occur when strong RFI is present and the receiver will record the observed flux densities as being lower than they actually are. The task makes use of the switched power information available from the telescopes to determine if the receiver gains are being compressed. The gain compression correction will apply a scaling to the data in order to mitigate the effect of compression. By default the gain compression correction is not applied, but all the plots are made to assess whether the correction should be applied.

The correction is performed for power differences (Pdiffs) that are between 0.7 and 1.2 by default, values outside those ranges will have their values (and subsequently data) flagged. Expanding the range of Pdiffs corrected is not expected to result in a more accurate correction.

To apply the correction, one needs to re-run the pipeline and set the argument apply=True for hifv_syspower.

hifv_syspower(apply=True)

There are known instances where the switched power data and hence the Pdiff data are bad for a given antenna, this can be characterized by the Pdiff plots looking relatively constant or with slow variations in time but having values > 1 or < 1 or extremly noisy relative to other antennas. When switched power data are bad there is the antexclude parameter that can be used in conjuction the usemedian parameter. The format for antexclude is a Python dictionary whose format is {'Band' : {'eaXX' : {'usemedian': True or False}}}. This enables usemedian to be specified on a per-antenna and per-band basis. If usemedian=True, the median Pdiff from all antennas will be used for the antenna specified in antexclude. If usemedian=False, then the Pdiff values will be set to 1.0 for the specified antenna.

If there are bands where the syspower corrections are not desired, the do_not_apply parameter is available; specify the bands where syspower corrections should not be applied in a comma-separated string. See example below:

# Single Band Example
hifv_syspower(clip_sp_template=[0.8,1.3], 
              antexclude={'S':{'ea02':{'usemedian':True}, 
                               'ea03':{'usemedian':False}}}, 
              apply=True)

# Multi-band Example (S, C, X and K bands)
hifv_syspower(clip_sp_template=[0.7,1.2], 
              antexclude={'S':{'ea02':{'usemedian':True}, 
                               'ea03':{'usemedian':False}}, 
                          'C':{'ea02':{'usemedian':True}, 
                               'ea12':{'usemedian':False}}}, 
              apply=True, do_not_apply='K,X')

Total Electron Content (TEC) Application

The TEC data format changed and datasets observed after August 7, 2023 cannot fetch TEC data using the CASA helper script used by the pipeline. A future version of CASA and the pipeline will restore this functionality.

The VLA pipeline now plots the TEC values at the time of observation, if the TEC data are available when the pipeline is executed, this is controlled by the show_tec_maps parameter whose default is True. The TEC data are not applied by default, but can be applied using the parameter apply_tec_correction, whose default is False, but can be set to True in a pipeline re-run.

hifv_priorcals(apply_tec_correction=True)

The application of TEC corrections is experimental and should be used with caution. TEC corrections are likely to primarily benefit L-band and possibly some S-band data if the ionosphere is very active. TEC data generally become available within ~2 days after an observation.

Note that the underlying tool in CASA used the uncompress program which is part of the ncompress package in typical Linux installations. The ncompress package has been deprecated in RHEL8 and may not be included in some standard installations. If problems are encountered with the hifv_priorcals task and the issue is the lack of uncompress, either request to have the package installed or run hifv_priorcals with show_tec_maps = False if you do not require the TEC correction.

Modifying the Pipeline for non-Stokes I Continuum Data

The pipeline is developed for the Stokes I continuum case. But it is possible to modify the 'casa_pipescript.py' and run the pipeline for other use cases:

Spectral Line

The pipeline is not optimized for calibrating spectral line data. Some pipeline steps of the regular VLA pipeline may be detrimental for spectral line setups and need to be modified or turned off. Calibrators also require enough signal-to-noise to reliably derive bandpasses, gains, phases, etc. for the typically more narrow spectral line spectral windows (spws) and channels. The pipeline will also flag edge channels for each spw. If the spectral line happens to be located on an spw edge, additional modifications to the script may be necessary.

The 'cont.dat' file

The easiest way to run the pipeline on data with spectral lines is to prepare a file, 'cont.dat', that specifies the frequencies only containing continuum (no spectral line). This will protect some procedures of the pipeline from being applied to spectral lines (see below).

The 'cont.dat' file has the following format:

Field: FIELDNAME1

SpectralWindow: SPWID1
freqrange1 LSRK (in GHz, LSRK) 
freqrange2 LSKR (in GHz, LSRK) 

SpectralWindow: SPWID2
freqrange1 LSRK (in GHz, LSRK) 
freqrange2 LSRK (in GHz, LSRK) 
... 


Field: FIELDNAME2

SpectralWindow: SPWID1
freqrange1 LSRK (in GHz, LSRK) 
freqrange2 LSRK (in GHz, LSRK) 
...

where FIELDNAMEx is the field name for each source. This provides the flexibility to define different continuum ranges for different targets. SPWIDn stands for the spw ID. Field names and spw ids can be found in the listobs output. An example with fields M82 and NGC3077 may look like:

Field: M82

SpectralWindow: 19
37.104~38.29GHz LSRK
38.30~39.104GHz LSRK

SpectralWindow: 37
31.360~32.123GHz LSRK
32.130~33.360GHz LSRK

Field: NGC3077

SpectralWindow: 37
31.360~32.123GHz LSRK
32.130~33.360GHz LSRK

For the field M82 this file defines spw 19 frequency ranges 37.104 – 38.290 GHz and 38.300 – 39.104 GHz as containing only continuum. This would be a setup where the line is found in the 10 MHz between 38.290 – 38.300 GHz. It also treats frequencies below 37.104 GHz and above 39.104 GHz the same as spectral lines. This can be used, for example, to exclude edge channels from being part of the autoflagging and weight calculations. Analogously, a spectral line falling in the 7 MHz range between 32.123 – 32.130GHz (between the two continuum ranges in spw 37) will be protected by the specification for spw 37 for both the M82 and NGC3077 fields.

'cont.dat' shall be placed in the root directory where the SDM-BDF resides and where the pipeline is executed. The pipeline will automatically pick up the file, there is no need to explicitly provide the file name in 'casa_pipescript.py'.

Note that if you use the cont.dat file, only the fields and spws that appear in the cont.dat will have RFI flagging and statwt run on them. So, if you have 64 spws, and only one is specified in the cont.dat, then the remaining 63 spws will not have RFI flagging applied or their weights recalculated. This behavior may change in future pipeline releases.

Pipeline Modifications for Spectral Line Data

In addition to creating a 'cont.dat' file, we advise to modify the following tasks in 'casa_pipescript.py' for spectral line data:

hifv_hanning: Hanning smoothing lessens the Gibbs ringing from strong spectral features, usually strong, narrow RFI, or very strong spectral lines such as masers. Hanning smoothing, however, reduces the spectral resolution. Therefore, depending on the data and the science case, one may or may not choose to apply Hanning smoothing. To disable the application of Hanning-smoothing in the pipeline, simply comment out hifv_hanning or remove the step from 'casa_pipescript.py'. Also, a single edge-channel is always flagged by the hanning smoothing task procedure.
hifv_flagata: The pipeline, by default, flags 1% of the data on each spw edge as well as the first and last 20MHz of the lowest and highest frequency (assuming that those are the baseband edges). In some cases, for example spectral surveys, lines may fall right on such frequencies. The edgespw, fracspw, and baseband parameters in hifv_flagata can be adjusteded to flag different percentages of the edges. Note that if edgespw=True, it will always flag at least one edge channel, no matter how small fracspw is.
hifv_checkflag(checkflagmode='bpd-vla'): While it is advisable to always run the RFI flaggers on the bandpass calibrators, users should be aware that a flag growth used in this mode will cause extra edge channels to be flagged if edge-channel flagging is used.
hifv_checkflag(checkflagmode='target-vla'): Flagging prior to this step was only applied to the calibrator scans, which should be line-free. But hifv_checkflag attempts to auto-flag all fields including target fields. The rflag mode in CASA's flagdata is designed to remove outliers that deviate from a mean level. Strong spectral lines can fulfill this criterion and be flagged. The 'cont.dat' file will ensure that rflag will only be applied to the continuum frequency ranges specified in it. We recommend manual flagging for the spectral line frequency ranges after the pipeline has finished processing.
hifv_statwt: A similar argument applies to the hifv_statwt step, where the visibilities are weighted by the square of the inverse of their RMS noise. Strong spectral lines will increase the RMS and will therefore be down-weighted. The cont.dat file will restrict statwt to only use the continuum frequency ranges for the rms and weight calculations and thus prevent the inclusion of spectral features. Alternatively, hifv_statwt can be excluded from the pipeline altogether and the CASA task statwt can be executed manually after the pipeline has finished, where statwt's parameter fitspw should be set to continuum channels only.

Given the above, we recommend to carefully prepare 'cont.dat' and modify 'casa_pipescript.py' as follows. The SDM-BDF name (here: mySDM) and maybe other parameters will have to be adapted for the run:

# This CASA pipescript is meant for use with CASA 6.5.4 and pipeline 2023.1.0.124
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
try:
 hifv_importdata(vis=['mySDM'])
 # Hanning smoothing is turned off in the following step. 
 # In the case of extreme RFI or very bright masers, Hanning smoothing 
 # may still be required.
 #hifv_hanning()
 hifv_flagdata(fracspw=0.01, \
  intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, \
  *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*', hm_tbuff='1.5int')
 hifv_vlasetjy()
 hifv_priorcals()
 hifv_syspower()
 hifv_testBPdcals()
 hifv_checkflag(checkflagmode='bpd-vla')
 hifv_semiFinalBPdcals()
 hifv_checkflag(checkflagmode='allcals-vla')
 hifv_solint()
 hifv_fluxboot()
 hifv_finalcals()
 hifv_applycals()
 # Keep the following two steps in the script if cont.dat exists. 
 # Otherwise we recommend to comment out the next two task.
 hifv_checkflag(checkflagmode='target-vla')
 hifv_statwt()
 hifv_plotsummary()
 hif_makeimlist(intent='PHASE,BANDPASS',specmode='cont')
 hif_makeimages(hm_masking='centralregion')
finally:
 h_save()

If a spectral line happens to be close to edge channels, one can decide to turn off edge channel flagging by setting the parameter edgespw=False in hifv_flagdata (if the line falls on the edge of a baseband, one may also consider to set baseband=False to avoid flagging the outer 10 baseband edges):

hifv_flagdata(intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*,\
                       *UNKNOWN*,*SYSTEM_CONFIGURATION*,\
                       *UNSPECIFIED#UNSPECIFIED*', 
                       fracspw=0.01, baseband=False, edgespw=False,\
                       hm_tbuff='1.5int')

or one can choose to reduce the fraction of edge channels being flagged. In the example below, we reduce the number to 1% on each end of each spw (note that this is now the pipeline default, the default was previously 5%):

hifv_flagdata(intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*,\
                   *UNKNOWN*, *SYSTEM_CONFIGURATION*,\
                   *UNSPECIFIED#UNSPECIFIED*',\
                   fracspw=0.01, baseband=True, edgespw=True,\
  hm_tbuff='1.5int',)

But note that including edge channels in the calibration may introduce uncertainties given that spw edges have low signal-to-noise and may contain correlator artifacts. Inspect the data to ensure the spw edges are usable.

Once all modifications are made, run the pipeline as:

# In CASA

execfile('casa_pipescript.py')

After the calibration has been obtained, the run can be followed up with CASA's flagdata mode='rflag' and statwt commands if required (e.g. if 'cont.dat' was not used).

Polarization Calibration

We are currently developing polarization calibration heuristics for the VLA pipeline. At this stage, however, the VLA pipeline does not derive or apply polarization calibration. If the scan intents included proper polarization intents (CALIBRATE_POL_LEAKAGE and CALIBRATE_POL_ANGLE), the pipeline will run with refantmode='strict', which is desireable for polarization calibration. The user may perform polarization calibration steps after the pipeline was run by using the pipeline calibration tables for pre-calibration as required.

Polarization calibration steps are explained in the 3C75 CASA guide (in particular, crosshand delay, the D-term (leakage), and polarization angle will be required). We also refer to the corresponding chapter in CASAdocs.

Mixed setups

If data were obtained in mixed correlator modes (e.g., spectral line and continuum, or different frequency bands), it may be advisable to separate the data into spectral line and continuum portions first. Then the individual parts can be calibrated via the default pipeline or by executing modified 'casa_pipescript.py' scripts. We recommend separating the data by flagging using the flagdata task and do not recommend using the split task, as described further below.

To start, we recommend importing the SDM to an MS and applying online flags at the same time. The corresponding CASA importasdm command would look like:

# In CASA
importasdm(asdm='mySDM', vis='mySDM.ms', ocorr_mode='co',\
           applyflags=True, savecmds=True, tbuff=1.5,\
           outfile='mySDM.flagonline.txt')

Note that we apply the online flags via applyflags=True but still save the flag commands in an outfile in case one would like to inspect those. We set tbuff=1.5 to extend the flags to 1.5 times the integration time (which the pipeline would do in hifv_flagdata).

After that step, if calibrating the continuum and line portions separately, you may want to make a full copy of the MS such that there is one to use for the continuum portion and another for the line portion. Then, we recommend flagging the spws to be excluded from a continuum or line reduction using the CASA task flagdata. We recommend flagging rather than splitting because running split can sometimes alter the MS metadata in ways that then prevent the pipeline from running. Modify 'casa_pipescript.py' and use each new separate MS as input in hifv_importdata.

Weak Calibrators

The VLA pipeline requires a minimum signal-to-noise of ~3 for each spw (each channel for the bandpass) and calibrator scan. Failure to integrate sufficiently on calibrator sources will lead to sub-optimal reductions with the pipeline. Users are advised to closely follow the Guide to Observing with the VLA to avoid issues due to faint calibrators whether using the pipeline or manual reduction.

Partially Frequency-Averaged Data

The pipeline invokes Hanning smoothing when frequency averaging at the VLA correlator is turned off (note that online frequency averaging is no longer offered). In that case, Hanning smoothing can fix the Gibbs ringing for strong narrow spectral features (e.g., RFI or maser lines). When the online frequency averaging is turned on, however, adjacent channels do not exhibit the typical Gibbs zig-zag pattern anymore and Hanning smoothing should not be applied for further processing. The pipeline currently detects if any spectral window was pre-averaged in frequency on the VLA correlator. In that case, it turns the Hanning smoothing off for all data. If one has an SDM where some spws are frequency averaged and some are not, the pipeline should be stopped after hifv_importasdm. The frequency averaged and non-averaged spws should then be separated into individual MSes using the CASA task 'split'. The pipeline can be started again on the individual MSes.

Incorrect scan intents

As mentioned in the "Pipeline Requirements", scan intents tell the pipeline which scans and fields are used for flux, delay, bandpass, gain and phase calibration. Scan intents should be set up correctly in the OPT before submitting the schedule block for observation.

When incorrect scan intents are identified after observations, one can still change the SDM-BDF with updated scan intents, although some care will be required.

The SDM-BDF metadata is structured in the form of XML files that can be edited. We provide a small Scan Intent Editing Perl Script to do this. The script is pretty self-explanatory and can add and delete scan intents to any scan.

Alternatively, the SDM can also be manually edited. Great care, however, should be taken not to corrupt the structure of the SDM-BDF/xml. We therefore advise not to edit the SDM-BDF/xml manually but to use the Perl script instead.

However, to edit the xml manually, cd into the SDM and edit the file 'Scan.xml'. We strongly recommend creating a backup copy of the 'Scan.xml' file in case the edits corrupt the metadata.

'Scan.xml' is divided into individual <row></row> blocks that identify each scan.

An example of a scan with a single scan intent (here: OBSERVE_TARGET) may look like:

   <row>
       <scanNumber>1</scanNumber>
       <startTime>4870732142800000000</startTime>
       <endTime>4870732322300000256</endTime>
       <numIntent>1</numIntent>
       <numSubscan>1</numSubscan>
       <scanIntent>1 1 OBSERVE_TARGET</scanIntent>
       <calDataType>1 1 NONE</calDataType>
       <calibrationOnLine>1 1 false</calibrationOnLine>
       <sourceName>J1041+0610</sourceName>
       <flagRow>false</flagRow>
       <execBlockId>ExecBlock_0</execBlockId>
   </row>

We can now change the scan intent, e.g., from OBSERVE_TARGET to CALIBRATE_AMPLI by simply updating the <scanIntent> tag:

   <row>
       <scanNumber>1</scanNumber>
       <startTime>4870732142800000000</startTime>
       <endTime>4870732322300000256</endTime>
       <numIntent>1</numIntent>
       <numSubscan>1</numSubscan>
       <scanIntent>1 1 CALIBRATE_AMPLI</scanIntent>
       <calDataType>1 1 NONE</calDataType>
       <calibrationOnLine>1 1 false</calibrationOnLine>
       <sourceName>J1041+0610</sourceName>
       <flagRow>false</flagRow>
       <execBlockId>ExecBlock_0</execBlockId>
   </row>

If we want to add a second intent, we will have to make additional changes. Let's add CALIBRATE_PHASE:

   <row>
       <scanNumber>1</scanNumber>
       <startTime>4870732142800000000</startTime>
       <endTime>4870732322300000256</endTime>
       <numIntent>2</numIntent>
       <numSubscan>1</numSubscan>
       <scanIntent>1 2 CALIBRATE_AMPLI CALIBRATE_PHASE</scanIntent>
       <calDataType>1 2 NONE NONE</calDataType>
       <calibrationOnLine>1 2 false false</calibrationOnLine>
       <sourceName>J1041+0610</sourceName>
       <flagRow>false</flagRow>
       <execBlockId>ExecBlock_0</execBlockId>
   </row>

Inside <scanIntent> we added the second intent, but also increased the second number from 1 to 2. In addition, we specified <numIntent> to be 2, and added a second entry to <calDataType> and <calibrationOnLine>. For the latter two, we also updated the second number from 1 to 2.

Analoguously, if we now add a third intent, CALIBRATE_BANPDASS, to the same scan, the <row> will look like:

   <row>
       <scanNumber>1</scanNumber>
       <startTime>4870732142800000000</startTime>
       <endTime>4870732322300000256</endTime>
       <numIntent>3</numIntent>
       <numSubscan>1</numSubscan>
       <scanIntent>1 3 CALIBRATE_AMPLI CALIBRATE_PHASE CALIBRATE_BANDPASS</scanIntent>
       <calDataType>1 3 NONE NONE NONE</calDataType>
       <calibrationOnLine>1 3 false false false</calibrationOnLine>
       <sourceName>J1041+0610</sourceName>
       <flagRow>false</flagRow>
       <execBlockId>ExecBlock_0</execBlockId>
   </row>

Check with CASA's listobs on the imported MS (after importing the data to an MS via importasdm) if the scan intents are now displayed as desired. Revert back to the original 'Scan.xml' if the above was not successful and contact the NRAO helpdesk for advice.

Allowed Intents

CALIBRATE_AMPLI : Amplitude calibration scan
CALIBRATE_PHASE : Phase calibration scan
CALIBRATE_BANDPASS : Bandpass calibration scan
CALIBRATE_DELAY : Delay calibration scan
CALIBRATE_FLUX : flux measurement scan.
CALIBRATE_POINTING : Pointing calibration scan
CALIBRATE_POLARIZATION : Polarization calibration scan
CALIBRATE_POL_LEAKAGE : Polarization Leakage calibration scan
CALIBRATE_POL_ANGLE : Polarizaiton Angle calibration scan
OBSERVE_TARGET : Target source scan
CALIBRATE_ATMOSPHERE : Atmosphere calibration scan
CALIBRATE_FOCUS : Focus calibration scan. Z coordinate to be derived
CALIBRATE_FOCUS X : Focus calibration scan; X focus coordinate to be derived
CALIBRATE_FOCUS Y : Focus calibration scan; Y focus coordinate to be derived
CALIBRATE_SIDEBAND_RATIO : measure relative gains of sidebands.
CALIBRATE_WVR : Data from the water vapor radiometers (and correlation data) are used to derive their calibration parameters.
DO_SKYDIP : Skydip calibration scan
MAP_ANTENNA_SURFACE : Holography calibration scan
MAP_PRIMARY_BEAM : Data on a celestial calibration source are used to derive a map of the primary beam.
TEST : used for development.
UNSPECIFIED : Unspecified scan intent
CALIBRATE_ANTENNA_POSITION : Requested by EVLA.
CALIBRATE_ANTENNA_PHASE : Requested by EVLA.
MEASURE_RFI : Requested by EVLA.
CALIBRATE_ANTENNA_POINTING_MODEL : Requested by EVLA.
SYSTEM_CONFIGURATION : Requested by EVLA.
CALIBRATE_APPPHASE_ACTIVE : Calculate and apply phasing solutions. Applicable at ALMA.
CALIBRATE APPPHASE PASSIVE : Apply previously obtained phasing solutions. Applicable at ALMA.
OBSERVE_CHECK_SOURCE

Known Issues

This section will be updated as the pipeline continues to be developed. The comments below are general to most pipeline versions, including the current production pipeline used for VLA data. The current production version is CASA 6.5.4.

The TEC data format changed and datasets observed after August 7, 2023 cannot fetch TEC data using the CASA helper script used by the pipeline. A future version of CASA and the pipeline will restore this functionality.
The calibration pipeline should not be run in an mpicasa environment; the pipeline may fail or produce inaccurate results.
The spatial models for the flux density calibrator 3C48 has problems at K-band and higher.

Although the hifv_fluxboot task performs clipping of the gain solutions used for flux density boot strapping, in some cases (especially at high frequencies, for datasets that may have suffered from pointing or other problems) the default flux density scale may be uncertain if outlier rejection was not sufficient. For projects that may require very accurate flux density scales, a user may want to flag the flux gain table by hand.
While the pipeline supports multi-band data, the same bandpass and flux density calibrators need to be used for all bands. If not, the data for one of the bands will be flagged.
The pipeline effectively calibrates each spectral window separately. This means that the signal-to-noise ratio (S/N) on the calibrators needs to be sufficiently high that solutions can be obtained within the solution intervals specified in the pipeline. For the delay calibrator, this means a S/N>3 per integration time, t(int); for the bandpass calibrator, a S/N>5 is required for solution times up to 10*t(int); for the gain calibrator a S/N>3 is required for solution times up to 10*t(int), and >5 for a scan average. The calibrator strength guidelines as a function of observing bandwidth in the Guide to Observing with the VLA for the high frequency end of Q-band; following these guidelines will be safe for all frequencies, although very narrow spectral channels may be problematic. In these cases the data may need to be calibrated by hand, using polynomial bandpass fitting instead.
In some instances, the post-RFI flagging plots look aesthetically worse than the pre-RFI flagging plots. This is due to a poorly performing antenna (higher noise than others) that is getting heavily flagged in the RFI flagging. The post-RFI flagging plot then has less data to average together resulting in a worse looking plot. This is not a problem and the outcomes from imaging with and without the flagging of these poorly performing antennas are not scientifically different. In the case of these noisy data not getting flagged (as in the previous pipeline version), they are strongly downweighted by statwt so they do not contribute much to the final images anyway.
Edge channel flagging reported in hifv_flagdata will appear as zero for most continuum modes. This is because the pipeline by default is only flagging 1% of edge channels, which results in 1 edge channel being selected for flagging in most continuum spws; however, Hanning smoothing has already resulted in the flagging of the first and last channels in a spectral window, so there are no more edge channels needing to be flagged.
The VLA pipeline is not currently designed to automatically handle spectral line observations; a modified recipe can be used along with additional information in the cont.dat file.
The pipeline does not currently perform polarization calibration; however, if the polarization intents were set properly, the output of the calibration pipeline may be used as the starting point for polarization calibration following the Stokes I calibration. See the 3C75 CASA guide.
The calibration pipeline supports calibration of OTF mosaic data for the VLA Sky Survey (VLASS) using a specific calibration recipe that is only validated for S-band.
The pipeline currently fails when a CALIBRATE_PHASE or CALIBRATE_AMPLI intent is added to a scan that already has a CALIBRATE_FLUX or CALIBRATE_BANDPASS scan intent.
CASA 6.X now pre-pends the paths listed in the system PYTHONPATH environment variable to its own PYTHONPATH. This is opposite of the CASA 5.X behavior that appended the system PYTHONPATH. This can cause issues with CASA importing Python libraries from the system Python installation rather than its internal Python libraries.
The pipeline results can be changed subtly depending on the the environment variable OMP_NUM_THREADS. The NRAO processing uses OMP_NUM_THREADS=1 and if this is not used, there may be minor differences (not scientifically relevant, however) in the results obtained.
The pipeline does not currently support partially frequency averaged data, see section on this subject.
The pipeline has not been validated for P-band observations and is not likely to succeed on P-band data.
For information about know issues affecting previous release, please see the Known Issues page.

Previous Pipeline Releases:

CASA 6.4.1 Pipeline web page