Facilities > VLA > Data Processing > Pipeline > CIPL Special Topics on Pipeline Reduction

CIPL Special Topics on Pipeline Reduction

Additional Flagging

There are occasions when the standard VLA Calibration Pipeline does not flag enough data to sufficiently calibrate your data, so some additional flagging will often be required. If you only need to add more flags to your science target(s), no further recalibration of your data is required and you may accomplish this in CASA as usual. However, if any of your calibrator sources require additional flagging, you will want to re-derive the calibration with your additional flags. To re-calibrate your SB with the VLA Calibration Pipeline and also include the extra flags, we suggest using the flagging template that the pipeline creates (or, that you may create before starting the pipeline if you know in advance what flags are needed). This template, named SDMName_flagtemmplate.txt, where "SDMName" is the full File Set ID as shown in the archive) will allow one to record any additional manual flagging in one file which the pipeline will use to flag as recorded.

Using the flag template

If you have already run the pipeline once, this flagging template has already been created: however, it is empty except for some syntax hints that are commented out and ignored by the pipeline. You may add any manual flagging you wish by editing this file directly and populating it with your CASA formatted flagging commands. The CASA task flagdata will currently NOT allow one to append this file!

1.) Move the SDM-BDF and the SDMName_flagtemplate.txt file into a new directory.

2.) From this directory, start CASA and follow these steps to run the pipeline. The pipeline should detect your flag template text file and apply the flags there as well in the hifv_flagdata stage of the pipeline and this should show up as a checked "Flagging Template" flagging agent and give the percentage of data flagged from these commands.

 

CASA 4.5.3 and earlier

If you are using CASA 4.5.3 or earlier versions of the Integrated pipeline, use the following method since the flagging template will not work: these CASA versions will clear any template you have and your entries will not be saved!

1. Carefully inspect the calibrated MS and flag as desired.

2. Create a new directory and copy ONLY the calibrated and fully flagged MS to this new directory. No other files, such as MSname.flagversions or other calibration tables should be copied, otherwise the the flagging you have done will be reset.

3. From the directory with your MS, start CASA. In CASA, you will now clear the current pipeline calibration (but not the flagging!) by using the CASA task clearcal with addmodel=False. This is done so your rerun of the pipeline won't result in incorrect calibration. We recommend the following parameters for clearcal.

default('clearcal')
vis='MSname'
field=''
spw=''
intent=''
addmodel=False

You are now ready to run clearcal by typing the following:
clearcal()

4. Next, run the clearstat task in CASA.

clearstat()

5. Those working from an NRAO machine, please skip to step 6! For other installations, you will need to create a pipeline recipe that does not call Hanning smoothing. To do this, open a new terminal and navigate to your CASA installation.

cd /path/to/your/installation/casa/pipeline/pipeline/recipes/

Make a copy of hifv.py and call it hifv_nohanning.py and then edit this file to comment out the following lines:


#from hifv_hanning_cli import hifv_hanning_cli as hifv_hanning

#hifv_hanning (pipelinemode=pipelinemode)

Save your edit and move to step 6.

6. Since this MS has already been through the pipeline once, Hanning smoothing has been done once and should NOT be done again. To avoid this, you will need to create a pipeline "recipe" that will perform the pipeline calibration other than Hanning smoothing. This recipe already exists for those using an NRAO machine in Socorro, and this version of the pipeline should be called from the CASA prompt

import pipeline.recipes.hifv_nohanning as hifv_nohanning

and then

hifv_nohanning.hifv(['MSname'])

Please note: the flagging statistics (eg., zero flags, etc.) reported by the rerun may no longer be correct since extra flags were incorporated into the MS before the pipeline rerun.

 

Removing all Calibration and Flagging

If you wish to rerun the pipeline, or run a modified version of it, on a given scheduling block that has already been through the pipeline (and you do NOT want to keep the previous pipeline calibration!), we recommend starting with the original SDM-BDF. However, if no SDM-BDF is available, the pipeline processed MS may be reset to work correctly with the next pipeline processing attempt. To do this, please use the steps below:

To clear the pipeline calibration from a MS:

  1. Run clearcal with addmodel=F. This clears the corrected data column by replacing it with a copy of the data column.
  2. Run flagdata with mode='unflag'. This unflags all data, including any online flags already applied. If you want to keep online flags, one could use flagmanager's restore option (flagversion='original')
  3. The VLA CASA Calibration pipeline, as currently implemented, runs the task statwt at the very end to down-weight bad data on per subband basis (e.g., due to noisy antennas, or due to RFI). To reset the weights, use the initweights task in CASA.
  4. Hanning smoothing considerations
  • You cannot undo Hanning smoothing if it was applied to a MS.
  • If you wish to rerun the pipeline on a MS that has been Hanning smoothed, and you do NOT want any Hanning smoothing, a raw SDM is required (downloadable from the archive). Note that Hanning smoothing is applied to all data automatically processed through the VLA CASA Calibration pipeline.
  • If Hanning smoothing is desired, and you want to re-run the pipeline, AND the MS you are using has had Hanning smoothing applied previously, DO NOT select Hanning smoothing again or you will lose extra edge channels and reduce the spectral resolution further.

Staff | Policies | Diversity