Special Topics on Pipeline Reduction

Additional Flagging

There are occasions when the standard VLA Calibration Pipeline does not flag enough data to sufficiently calibrate your data, so some additional flagging will be required. If you only need to add more flags to your science target(s), no further recalibration of your data is required and you may accomplish this in CASA as usual. However, if any of your calibrator sources require additional flagging, you will want to re-derive the calibration with your additional flags. To re-calibrate your SB with the VLA Calibration Pipeline and also include the extra flags, there are two likely starting points: from the pipeline calibrated MS just produced or from the raw SDM-BDF. Instructions for both cases are below.

Starting with a Pipeline Calibrated MS

1. Carefully inspect the calibrated MS and flag as desired.

2. Create a new directory and copy ONLY the calibrated and fully flagged MS to this new directory. No other files, such as MSname.flagversions or other calibration tables should be copied.

3. From the directory with your MS, start CASA. In CASA, you will now clear the current pipeline calibration (but not the flagging!) by using the CASA task clearcal with addmodel=False. This is done so your rerun of the pipeline won't result in incorrect calibration. We recommend the following parameters for clearcal.

default('clearcal')
vis='MSname'
field=''
spw=''
intent=''
addmodel=False
async=False

# You are now ready to run clearcal by typing the following:
clearcal()

4. Next, run the clearstat task in CASA.

clearstat()

5. If the MS had Hanning smoothing applied before, Hanning smoothing should NOT be done again. To avoid this, you will need to enter "n" when prompted to select Hanning smoothing or not when you start the pipeline by typing the following in to the CASA prompt:

execfile('/path/on/your/computer/to/scripts/EVLA_pipeline.py')

Please note: the flagging statistics (eg., zero flags, etc.) reported by the rerun will no longer be correct since extra flags were incorporated into the MS before the pipeline rerun.

Starting from a SDM-BDF

If you have only the calibration tables and flag tables from a pipeline processing,  you may apply the pipeline calibration to the raw SDM-BDF as described here, then return to this page and follow the above instructions for starting with a calibrated MS. This should also work if you create your own MS and apply flags before running the pipeline.

Removing all Calibration and Flagging

If you wish to rerun the pipeline, or run a modified version of it, on a given scheduling block that has already been through the pipeline (and you do NOT want to keep the previous pipeline calibration!), we recommend starting with the original SDM-BDF. However, if no SDM-BDF is available, the pipeline processed MS may be reset to work correctly with the next pipeline processing attempt. To do this, please use the steps below:

To clear the pipeline calibration from a MS:

1. Run clearcal with addmodel=F. This clears the corrected data column by replacing it with a copy of the data column.
2. Run flagdata with mode='unflag'. This unflags all data, including any online flags already applied.
3. The VLA CASA Calibration pipeline, as currently implemented, runs the task statwt at the very end to down-weight bad data on per subband basis (e.g., due to noisy antennas, or due to RFI). To reset the weights, run the initweights task in CASA.
4. Hanning smoothing considerations

• You cannot undo Hanning smoothing if it was applied to a MS.
• If you wish to rerun the pipeline on a MS that has been Hanning smoothed, and you do NOT want any Hanning smoothing, a raw SMD is required Note that Hanning smoothing is applied to all data automatically processed through the VLA CASA Calibration pipeline immediately after observation.
• If Hanning smoothing is desired, and you want to re-run the pipeline, AND the MS you are using has had Hanning smoothing applied previously, DO NOT select Hanning smoothing again or you will lose extra edge channels and reduce the spectral resolution further.