Running the VLASS Pipelines

Instructions on running the VLASS pipelines for users outside of NRAO

VLASS Data

VLA Sky Survey (VLASS) data  is available from the NRAO archive and can be found when searching for the 'VLASS' keyword. The visible sky is observed three times in the individual epochs and each epoch is further sub-divided into two portions covering half of the sky. For example, a given source appears in Epochs 1.2, 2.2 and 3.2. A special case is epoch 1.1. Due to technical issues at the start of the survey, the Epoch 1.1. data is available but shows pointing errors that need extra processing. For a smoother dataset, VLASS will therefore observe an epoch 4.1 to replace the 1.1. data until further technical development has been commissioned. In total, data is available for epochs (1.1), 1.2, 2.1, 2.2, 3.1, 3.2, and (after observing) 4.1.  In the archive, e.g. each for 'VLASS1.2', or for data at a given position. 

VLASS Pipelined Data

Most users will retrieve the data products that NRAO has processed. Again, these files are available from the NRAO archive. Calibration tables are attached to the raw visibilities for download ('cals' column). Images are available by selecting the 'Images' tab in the archive search. The images can be viewed in the archive interface via CARTA or downloaded as fits files. 

Restoring VLASS visibilities

If a user wants to use the NRAO calibration to restore the calibrated visibilities, they can download the calibration tables and follow the
restoration instructions for the general VLA pipeline. 

VLASS Pipelines

The following VLASS/CASA pipelines are used for the processing at NRAO. It is always advisable to use the latest version.

casa-6.5.4-9-pipeline-2023.1.1.8-py3.8.tar.xz

casa-6.4.1-12-pipeline-2022.1.1.5-py3.6.tar.xz

casa-6.1.3-3-pipeline-2021.1.1.32.tar.xz

casa-pipeline-release-5.6.3-19.el7.tar.gz

Calibration

In the following, we describe the pipeline that we use at  NRAO to calibrate VLASS data. VLASS has its own pipeline tasks but also uses tasks common to the VLA and ALMA pipelines. For further documentation, we therefore refer to the help features of the pipeline tasks, as well as the VLA and ALMA pipeline documentations.
To perform the VLASS calibration, use one of the VLASS CASA and pipeline versions mentioned above (best to use the latest version).   As a first step, download the SDM from the archive to a working directory. Note, that the final data of a pipeline run will be stored in ../products. So it is advisable to run the pipeline in a directory structure that is two levels deep to avoid overwriting data in the ../products directory when multiple datasets are processed. For calibration, we use the file 'casa_pipescript.py' as below that is executed within the CASA pipeline versions. In the following 'casa_pipescript.py' file, replace the 'MyVLASS-SDM' in hif_importdata with the actual name as obtained from the archive: 
 Contents of 'casa_pipescript.py' for VLASS calibration:
context = h_init()
context.set_state('ProjectSummary', 'proposal_code', 'VLASS')
context.set_state('ProjectSummary', 'proposal_title', 'unknown')
context.set_state('ProjectSummary', 'piname', 'unknown')
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
context.set_state('ProjectStructure', 'ppr_file', 'PPR.xml')
try:
 hifv_importdata(vis=['MyVLASS-SDM'], session=['session_1'])
 hifv_hanning(pipelinemode="automatic")
 hifv_flagdata(intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*', flagbackup=False, baseband=False, clip=True, autocorr=True, template=True, hm_tbuff='manual', online=True, tbuff=0.225, shadow=True, edgespw=False, fracspw=0.0, quack=False)
 hifv_vlasetjy(pipelinemode="automatic")
 hifv_priorcals(show_tec_maps=True, apply_tec_correction=False, swpow_spw='6,14')
 hifv_syspower(apply=True)
 hifv_testBPdcals(pipelinemode="automatic")
 hifv_checkflag(checkflagmode='bpd-vlass')
 hifv_semiFinalBPdcals(pipelinemode="automatic")
 hifv_checkflag(checkflagmode='allcals-vlass')
 hifv_solint(limit_short_solint=0.45)
 hifv_fluxboot(fitorder=2)
 hifv_finalcals(pipelinemode="automatic")
 hifv_circfeedpolcal(pipelinemode="automatic")
 hifv_flagcal(pipelinemode="automatic")
 hifv_applycals(flagsum=False, flagdetailedsum=False, gainmap=True)
 hifv_checkflag(checkflagmode='target-vlass')
 hifv_statwt(pipelinemode="automatic")
 hifv_plotsummary(pipelinemode="automatic")
 hifv_fixpointing(pipelinemode="automatic")
 hifv_exportdata(gainmap=True, exportmses=False, exportcalprods=True)
finally:
 h_save()
It is possible to modify the standard parameters of this script and also to insert other CASA and python functions and tasks. More information on how to run the pipeline is provided  on the VLA pipeline pages, which also discusses the output files, e.g. the weblog. 
We also would like to refer to VLASS Memo #20 (Carlson et al.) that describes user defined VLASS processing (also see Erik Carlson's GitHub account). 

Imaging

VLASS has three main imaging pipelines 1) Quick Look (QL) for fast continuum processing 2) Single Epoch (SE) for advanced imaging procedures, including selfcal and better masking and 3) Coarse Cubes (CC) which consist of an image per spectral window (i.e. up to 16 images) including polarization IQU products. A more detailed description is provided in Basic Data Products.
VLASS visibilities are observed on-the-fly with very short dump times, which leads to a very large number of  pointings on the sky. The pipeline will image only selected poitings in a, typically, 1 degree2 area (imaging first a larger area for cleaning and anti-aliasing reasons, then cropping to 1x1 degree). For a pre-selection of pointings, NRAO decimates the visibilities to PIMS, i.e. 'per image MSs'. A user can do this, too, using the script appended in  VLASS Memo #20 (Carlson et al..), also available on GitHub. PIMS, however, are not needed and the calibrated MS can be used as a whole.
 

Quick Look

Analogous to the calibration, a 'casa_pipescript.py' script is used to run the QL imaging. Change the 'myCalibratedMS' in hif_importdata to the calibrated MS name.  
 Contents of 'casa_pipescript.py' for VLASS QL imaging:
context = h_init()
context.set_state('ProjectSummary', 'proposal_code', 'VLASS')
context.set_state('ProjectSummary', 'proposal_title', 'unknown')
context.set_state('ProjectSummary', 'piname', 'unknown')
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
context.set_state('ProjectStructure', 'ppr_file', 'PPR.xml')
try:
 hifv_importdata(vis=['myCalibratedMS'], nocopy=True, session=['session_1'])
 hif_editimlist(parameter_file='parameter.list')
 hif_transformimagedata(datacolumn='data', clear_pointing=True, modify_weights=False)
 hif_makeimages(hm_cleaning='manual', hm_masking='none')
 hifv_pbcor(pipelinemode="automatic")
 hif_makermsimages(pipelinemode="automatic")
 hif_makecutoutimages(pipelinemode="automatic")
 hifv_flagdata(autocorr=False, scan=False, quack=False, clip=False, baseband=False, edgespw=False, online=False, template=False, hm_tbuff='manual')
 hifv_exportvlassdata(pipelinemode="automatic")
finally:
 h_save()
The hif_editimlist task also calls for a file 'parameter.list' that contains some of the imaging parameters that are used in multiple steps in the pipeline. For VLASS operations, we currently use the following 'parameter.list' file:
editmode='add'
imagename='VLASS2.1.ql.T10t31.J200200-033000'
phasecenter="J2000 20:02:0.165 -03.30.0.0000"
imsize=[7290,7290]
cycleniter=200
Parameters can be edited and added (see alsosetion 'Script Modifications' below). If it is desired to use the same phase center as the production VLASS images, an archive image search will reveal those. 

Single Epoch 

The SE imaging is similar, with its own 'casa_pipescript.py' as listed below. Here, in this example, the parameter file is called 'SEIP_parameter.list', which also contains instructions on the imaging mode that is used. For production, we use the following  'SEIP_parameter.list' file:
imagename='VLASS2.2.se.T202t32.J153226-353000'
phasecenter="J2000 15:32:26.493 -35.30.0.0000"
imaging_mode='VLASS-SE-CONT-MOSAIC'
Of course, the imagename and phasecenter parameters differ for each run. 
The corresponding 'casa_pipescript.py' file, that is executed within CASA, calls the 'SEIP_parameters.list' file and also contains the MS (in this example 'myCalibratedMS')
__rethrow_casa_exceptions = True
context = h_init()
context.set_state('ProjectSummary', 'proposal_code', 'VLASS')
context.set_state('ProjectSummary', 'proposal_title', 'unknown')
context.set_state('ProjectSummary', 'piname', 'unknown')
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
context.set_state('ProjectStructure', 'ppr_file', 'PPR.xml')
context.set_state('ProjectStructure', 'recipe_name', 'hifv_vlassSEIP')
try:
 hifv_importdata(nocopy=True, vis=['myCalibratedMS'], session=['session_1'])
 hif_editimlist(parameter_file='SEIP_parameter.list')
 hif_transformimagedata(datacolumn='data', clear_pointing=False, modify_weights=True, wtmode='nyq')
 hifv_vlassmasking(maskingmode='vlass-se-tier-1', vlass_ql_database='/home/vlass/packages/VLASS1Q.fits')
 hif_makeimages(hm_masking='manual')
 hifv_checkflag(checkflagmode='vlass-imaging')
 hifv_statwt(statwtmode='VLASS-SE', datacolumn='residual_data')
 hifv_selfcal(selfcalmode='VLASS-SE')
 hif_editimlist(parameter_file='SEIP_parameter.list')
 hif_makeimages(hm_masking='manual')
 hif_editimlist(parameter_file='SEIP_parameter.list')
 hifv_vlassmasking(maskingmode='vlass-se-tier-2')
 hif_makeimages(hm_masking='manual')
 hifv_pbcor(pipelinemode="automatic") 
 hif_makermsimages(pipelinemode="automatic")
 hif_makecutoutimages(pipelinemode="automatic")
 hif_analyzealpha(pipelinemode="automatic")
 hifv_exportvlassdata(pipelinemode="automatic")
finally:
 h_save()

Coarse Cube

Finally, coarse cube images are also delivered for VLASS. They are created using prior information. In particular, an SE run has to be performed first. Some of the parameters determined in SE, e.g. masks, are then re-used in CC processing. The SE processing creates a file 'reimaging_resources.tgz' in the ../products/ directory. This file needs to be copied into the CC processing directory, as well as the 'SEIP_parameters.list' that was used for the SE run.
In addition, a 'CCIP_parameters.list' file is needed to control the CC processing. e.g.:
imagename='VLASS2.2.se.T202t32.J153226-353000'
phasecenter="J2000 15:32:26.493 -35.30.0.0000"
imaging_mode='VLASS-SE-CUBE'
spw=['2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17']
and like in previous steps, the parameter files and the calibrated MS are called within a 'casa_pipescript.py' file. For CC this looks like:
context = h_init()
context.set_state('ProjectSummary', 'proposal_code', 'VLASS')
context.set_state('ProjectSummary', 'proposal_title', 'unknown')
context.set_state('ProjectSummary', 'piname', 'unknown')
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
context.set_state('ProjectStructure', 'ppr_file', 'PPR.xml')
context.set_state('ProjectStructure', 'recipe_name', 'hifv_vlassSEIP')
try:
 hifv_importdata(nocopy=True, vis=['myCalibratedMS'], session=['session_1'])
 hif_editimlist(parameter_file='SEIP_parameter.list')
 hif_transformimagedata(datacolumn='data', clear_pointing=False, modify_weights=True, wtmode='nyq')
 hifv_restorepims(pipelinemode="automatic")
 hifv_flagtargetsdata(pipelinemode="automatic")
 hif_editimlist(parameter_file='CCIP_parameter.list', editmode='replace')
 hif_makeimages(hm_masking='manual')
 hifv_pbcor(pipelinemode="automatic")
 hif_makermsimages(pipelinemode="automatic")
 hif_makecutoutimages(pipelinemode="automatic")
 hifv_analyzestokescubes(pipelinemode="automatic")
 hifv_exportvlassdata(pipelinemode="automatic")
finally:
 h_save()
This script is shorter than the SE processing, since some of the products and parameters are inherited from SE to CC via the 'reimaging_resources.tgz' file.

Script Modifications

As mentioned earlier, modifications to the standard scripts can be done, either in the pipeline tasks themselves or by adding CASA tasks to the 'casa_pipescript.py' script. The 'help' function for the tasks are useful for their descriptions. However, it is frequently more useful to make the changes in the 'SEIP_parameters.list' or 'CCIP_parameters.list' files. When modified there, the parameters will be used in multiple pipeline tasks. Examples are, e.g. imsize=[10280,'10280], cell=['1arcsec','1arcsec'], or uvtaper=['2arcsec','2arcsec','10deg'], that can be inserted into the *_parameters.list files.   

Resources

For any questions, please contact the NRAO helpdesk.

Connect with NRAO

The NSF National Radio Astronomy Observatory and NSF Green Bank Observatory are facilities of the U.S. National Science Foundation operated under cooperative agreement by Associated Universities, Inc.

Founded in 1956, the NSF NRAO provides state-of-the-art radio telescope facilities for use by the international scientific community. NSF NRAO telescopes are open to all astronomers regardless of institutional or national affiliation. Observing time on NSF NRAO telescopes is available on a competitive basis to qualified scientists after evaluation of research proposals on the basis of scientific merit, the capability of the instruments to do the work, and the availability of the telescope during the requested time. NSF NRAO also provides both formal and informal programs in education and public outreach for teachers, students, the general public, and the media.

The U.S. National Science Foundation (U.S. NSF) is an independent federal agency created by Congress in 1950 "to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense..."

AUI collaborates with the scientific community and research sponsors to plan, build, and operate cutting-edge facilities. We cultivate excellence, deliver value, enhance education, and engage the public.