Rubin Alert Pipeline
General Description
NRAO is currently performing limited service observing on the JVLA for alerts generated by the Vera C. Rubin Observatory, carried out under project code ALRT01. Observations use a standard X-band (8-10GHz), 8-bit continuum setup and are typically short in duration (~12.5 mins), consisting of a single 2 minute scan centered on the alert coordinates, bracketed by scans on a nearby complex gain calibrator. For information about this observing program and instructions on how to submit an alert observation proposal, please see the Rubin Observing Page. In order to minimize observation length and allow for fast turnaround, the observations do not include a standard flux density calibrator, instead using the power from a noise diode switched into the signal path to calibrate the data. The resulting uncertainty in the flux density scale is ~20%, which is somewhat worse than can be achieved using more rigorous calibration techniques.
A special pipeline has been developed for the Rubin alerts using switched power calibration, further details of which are described below. It is based on a modified version of the standard JVLA pipeline. For speed, no self-calibration is performed by the Rubin alert pipeline, and results also do not undergo any quality assurance checks before being ingested into the NRAO archive. For more information about general pipeline heuristics, please see the pipeline presentations from the 2021 JVLA Data Reduction workshop.
Accessing Rubin Alert Data and Images
Once an observation from project code ALRT01 has been observed, the data are immediately ingested into the NRAO archive and are publicly available when fully ingested. Searching for project code ALRT01 in the main search text field of the archive should bring up all available results from Rubin alert observations. Observations are processed through the calibration and imaging pipeline after the raw data have been ingested, and once the calibration and imaging pipeline has completed, the pipeline results are also ingested into the NRAO archive and will be publicly available as soon as ingestion is completed.
Data Products Available
The following data and formats are available directly from the NRAO archive:
- Raw visibility data (Science Data Model-Binary Data Format [SDM-BDF]) - raw, uncalibrated visibility data from the JVLA.
- Basic Measurement Set (MS) - raw, uncalibrated data in CASA Measurement Set format, with or without online flags applied.
- Calibrated Measurement Set (CMS) - pipeline-calibrated CASA Measurement Set, with calibration and flagging applied.
- Calibration Tar file - A tar file with the pipeline products, including the final calibration tables, final flag version, weblog, and related files.
- Images - FITS files for available images produced by the pipeline. Both non-primary-beam corrected (flat-noise) and primary-beam corrected (pbcor) images are available via download.
For more information about what the standard JVLA CASA pipeline produces in general, please see this section of our main VLA CASA Calibration web page.
Download Instructions
Images
FITS file images may be requested from the NRAO archive for download or viewing remotely via CARTA on NRAO servers. At this time, only the primary beam corrected image is available to view remotely using CARTA remotely via the archive, the image must be downloaded to get the non-primary beam corrected (flat noise) image.
To view the image in CARTA on NRAO servers:
If you know the Alert ID used:
- Enter only the first 14 characters of the Alert ID into the archive's main text field and search (return key). The last part of the observation name, MJD.fractional day, may also be entered if known: e.g. "61231.92840232639" (see example below)
- Click on the "View Images" tab below the search field: if an image is found,
- Click the small CARTA icon to the left of the page, in-line with the image search result row (this should generate a pop-up window; mouse hover-over should show "View in CARTA")
- Enter a valid email address in the text field for this in the pop-up window
- Click Submit request. This should automatically take you to the CARTA server once ready. If a bad gateway message is displayed, try refreshing the page. This should generate an email confirming submission as well as an email when CARTA and the requested data are ready to view, along with a link to the CARTA session. This process should take only a minute or two to launch CARTA if the system is behaving normally. The CARTA session will only last one hour.
To see all available Rubin Alert images in the archive, enter ALRT01 into the Archive's main search text field and search (return key), then click the "View Images" tab below the search input area.
Multiple images may be selected to view in one session, if available.
To download the images:
If you know the Alert ID used:
- Enter only the first 14 characters of the Alert ID into the archive's main text field and search (return key).
- Click on the "View Images" tab below the search field: if an image is found,
- Click the small clipboard icon to the left of the page, in-line with the image search result row (mouse-hover over icon should show "Select for Download")
- Click the "Download" button to the left of the screen, above the search results (this should generate a pop-up window)
- Enter a valid email address in the text field for this in the pop-up window
- Click Submit request. This should generate an email confirming submission as well as another email when the request is complete and available to download.
To see all available Rubin Alert images in the archive, enter ALRT01 into the Archive's main search text field and search (return key), then click the "View Images" tab below the search input area.
Multiple images may be selected to download in one request, if available.
Image products to expect when requesting an image download:
- *_sci.X_band.cont.regcal.I.tt0.fits - non-primary beam corrected (flat-noise) continuum image
- *_sci.X_band.cont.regcal.I.tt0.pbcor.fits - primary beam corrected continuum image
- *_sci.X_band.cont.regcal.I.mask.fits - deconvolution mask file used by tclean
- *_sci.X_band.cont.regcal.I.pb.tt0.fits - The primary beam used when creating the primary beam corrected image
- *_sci.X_band.cont.regcal.I.alpha.fits - The spectral index map created by tclean (NB: includes the spectral index of the primary beam, so is only good close to the peak of the primary beam)
- *_sci.X_band.cont.regcal.I.alpha.error.fits - The estimated error in spectral index map
- *_sci.X_band.cont.regcal.I.tt1.fits - 1st order Taylor coefficients of per-pixel polynomials representing the sky multiplied by the primary beam
- *_sci.X_band.cont.regcal.I.tt1.pbcor.fits - Primary beam corrected images of 1st order Taylor coefficients of polynomials representing the sky multiplied by the primary beam
Calibrated MS
Search for project code ALRT01 in the main text entry field on the NRAO archive or use the Archive raw science data (SDM) identifier found on the Rubin Alerts Results Page. All observations that have been calibrated and have pipeline products available in the archive will show an entry in the "Cals" column to the right of the search results. To download a pipeline-calibrated MS:
- Either click the plus sign (+) button to the left of the returned search result row for ALRT01, or click the "View Observations" tab above the search results
- Select a single observation you want to download (only one observation may be selected at a time when requesting a Calibrated MS!) by clicking the clipboard icon to the left of that observation's row
- Click the "Download" button above the search results displayed (this should generate a pop-up window)
- Enter a valid email into the email text entry
- Ensure that "Calibrated Measurement Set" is selected in the "Choose download data format" section
- Click the Submit Request button when ready. This should generate an email confirming submission as well as another email when the request is complete and the data are available to download (links in complete email). This process should usually take less than 20 minutes for the short duration ALRT01 observations.
When requesting a Calibrated MS from the archive, the delivered products will include two MS files, with an example pair below:
ALRT01.sb51672064.eb51673735.61347.068516666666.ms - This contains all calibrated data, calibrators and the science target. Expect ~8GB in size.
ALRT01.sb51672064.eb51673735.61347.068516666666_targets_cont.ms - This contains only the science target scan. Expect ~700MB to 1GB in size.
Raw Data
To download the raw, uncalibrated data, one may request the SDM-BDF or Basic MS option. Search for project code ALRT01 in the main text entry field on the NRAO archive or use the Archive raw science data (SDM) identifier found on the Rubin Alerts Results Page. All observations that have been observed under ALRT01 should show up in the search results.
To download the raw, uncalibrated MS:
- Either click the plus sign (+) button to the left of the returned search result row for ALRT01, or click the "View Observations" tab above the search results
- Select a single observation you want to download (more than one observation in Basic MS format may be requested at once) by clicking the clipboard icon to the left of that observation's row
- Click the "Download" button above the search results displayed (this should generate a pop-up window)
- Enter a valid email into the email text entry
- Select "Basic Measurement Set" in the "Choose download data format" section
- Ensure that the "Apply telescope flags" box is checked
- Click the Submit Request button when ready. This should generate an email confirming submission as well as another email when the request is complete and the data are available to download (links in complete email). This process should usually take less than 20 minutes for the short duration ALRT01 observations.
To download the raw, uncalibrated SDM-BDF:
- Either click the plus sign (+) button to the left of the returned search result row for ALRT01, or click the "View Observations" tab above the search results
- Select the observations you want to download (more than one observation in SDM-BDF format may be requested at once) by clicking the clipboard icon to the left of that observation's row
- Click the "Download" button above the search results displayed (this should generate a pop-up window)
- Enter a valid email into the email text entry
- Select "SDM-BDF dataset (metadata + visibilities) " in the "Choose download data format" section
- Click the Submit Request button when ready. This should generate an email confirming submission as well as anther email when the request is complete and the data are available to download (links in complete email). This process should usually take less than 20 minutes for the short duration ALRT01 observations.
Naming Conventions
The raw visibility data, basic MS and Calibrated MS files generally will have file names such as:
SDM-BDF (raw visibility data as delivered by the JVLA)
ALRT01.sb51672064.eb51673735.61347.068516666666
- ALRT01 - project code
- sb51672064 - Scheduling Block ID (SB, think of this as the script that is run on the JVLA)
- eb51673735 - Execution Block ID (EB, think of this as a particular run of a specific SB)
- 61347 - Modified Julian date of observation
- 068516666666 - Fractional day: the time of the start of the observation on a given UTC day.
Measurement Set
Measurement Sets will usually match the SDM-BDF name as above and end with ".ms" to indicate this is a measurement set. Calibrated MS requests from the NRAO archive will include a second MS ending with _targets_cont.ms.
Images
FITS images from the NRAO archive will generally have a file name structure as follows:
ALRT01.MJD61347.068516666666._17114118770602__sci.X_band.cont.regcal.I.tt0.pbcor.fits
- ALRT01 - The first part is the project code, ALRT01
- MJD661347 - Modified Julian Day
- 068516666666 - Fractional day -
- _17114118770602_ - Science target field name as entered into the Observing Preparation Tool by staff, limited to the first 14 characters.
- _sci - indicating this is a science target image (there are some calibrator images also available in the archive for other projects)
- X_band - Indicates this image uses the X band receivers on the JVLA
- cont - indicates a continuum image
- regcal - indicates the image was made from standard calibration results, not self-calibration
- I.tt0 - indicates the image is Stokes I image using the mtmfs mode (Multi-term (Multi Scale) Multi-Frequency Synthesis-imaging) in tclean
- pbcor - Indicates this is the primary beam corrected image
- Image names that do not have pbcor at the end indicate this is the flat-noise image (non-primary beam corrected)
For questions, suggestions, and other issues encountered with this pipeline or results obtained using it, please submit a ticket to the Rubin Alerts Department of the NRAO Helpdesk. If you encounter issues retrieving data from the archive, please include your request ID, if applicable, in your Helpdesk query.
Pipeline Script
The Rubin alert pipeline is available in CASA/pipeline versions 6.7.4 and following. General instructions for how to run the JVLA calibration and imaging pipelines are available at the standard JVLA pipeline webpage. In order to execute the Rubin alert pipeline, which uses switched power to provide the flux density calibration, a special casa_pipescript.py recipe is needed. The following script includes includes all of the changes needed and should give the same results on ALRT01 data as the automated runs done shortly after observation:
context = h_init()
context.set_state('ProjectSummary', 'proposal_code', 'VLA/ALRT01')
context.set_state('ProjectSummary', 'proposal_title', 'unknown')
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
context.set_state('ProjectStructure', 'recipe_name', 'switchedpower_nofluxcal')
mysdm=''
myms=mysdm+'.ms'
try:
hifv_importdata(vis=[mysdm], session=['session_1'], specline_spws='none')
hifv_hanning()
hifv_flagdata(intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*', edgespw=False, baseband=False, hm_tbuff='1.5int')
hifv_priorcals(apply_swpowcal=True,show_tec_maps=False)
hifv_testBPdcals()
hifv_solint()
hifv_fluxboot()
hifv_finalcals(use_flux_cal=False)
hifv_applycals()
hifv_flagdata(autocorr=False,baseband=True,edgespw=True,fracspw=0.01,hm_tbuff='manual',tbuff=[0.0],shadow=False,quack=False,template=False,online=False,clip=False)
hifv_checkflag(checkflagmode='bpd-vla')
hifv_checkflag(checkflagmode='target-vla')
hifv_statwt()
hifv_plotsummary()
hifv_mstransform()
hif_makeimlist(specmode='cont', intent='PHASE')
hif_makeimages(hm_masking='auto',hm_cyclefactor=2.0)
hif_checkproductsize(maximsize=16384)
hif_makeimlist(specmode='cont', intent='TARGET')
hif_makeimages(hm_masking='auto',hm_cyclefactor=2.0)
hifv_exportdata()
finally:
h_save()
Modifications to the standard pipeline
For experts, this section describes in more detail the differences between the standard JVLA pipeline and the Rubin alert pipeline. The Rubin alert calibration and imaging pipeline performs calibration and automated flagging and imaging using CASA. Each Scheduling Block (SB) observed under ALRT01 on the JVLA is automatically processed through the Rubin alert pipeline with results being available to the public via the NRAO Archive or via the Rubin Alert Results website. Changes were found to be necessary within already-existing pipeline tasks, as well as a general calibration recipe change.
Removing Stages for Faster Processing
Since the observing setup and strategy are fixed (beyond choice of calibrator and science target), some stages of the pipeline can be skipped to allow faster processing times. These include:
- hifv_setjy → This stage is not needed because one of the four JVLA standard flux density scale calibrators (3C48, 3C138, 3C147, & 3C286) is not observed.
- hifv_syspower → No gain compression corrections are needed at X-band with the 8-bit system.
- hifv_checkflag → All RFI flagging is moved to after hifv_applycals, so that the data used for deriving the amplitude calibration in the hifv_finalcals step is as similar to the data used for deriving the online switched power calibration corrections as possible.
- hifv_semiFinalBPdcals → The complex gain calibrator is used as the bandpass calibrator, so this step is not needed.
- hif_exportdata → Only the final hif_exportdata after the imaging is needed.
- hif_checkproductsize → Only one hif_checkproductsize is needed for continuum-only imaging.
- hif_findcont, hif_uvcontsub, hif_makeimlist(cube), hif_makeimages (cube) → Not needed for continuum-only imaging.
- hif_selfcal → Removed out of speed considerations, as this stage can take a considerable amount of time, especially for A configuration data.
Turn off baseband and spw edge channel flagging in hifv_flagdata
The switched power system of the JVLA is sensitive to the entire frequency range observed within the subband, and flagging the baseband and spw edges in the initial hifv_flagdata stage of the pipeline, as in the standard recipe, breaks some of the assumptions associated with using the switched power to calibrate the amplitudes. The modified call to hifv_flagdata is
hifv_flagdata(intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*', edgespw=False, baseband=False, hm_tbuff='1.5int')
Turn on Switched Power Corrections
The pipeline has the ability to use the switched power data as a prior calibration, by setting apply_swpowcal=True in hifv_priorcals. It is not normally used in the standard JVLA calibration pipeline, because it does not work well for 3-bit data. However, the Rubin alert observations use only 8-bit data. Applying the switched power corrections puts the raw data into units of antenna temperature, Kelvin.
hifv_priorcals(apply_swpowcal=True)
Add Antenna Efficiencies
In order to convert from Kelvin to something similar to Janskys the antenna gain corrections need to be applied. This is achieved using caltype='gceff' in the CASA task gencal, within hifv_priorcals. This change was introduced for CASA/pipeline versions 6.7.4 and following (prior versions use caltype='gc', which corrects only for a normalized elevation-dependent gain). This is now implemented for both standard JVLA pipeline runs and the Rubin alert pipeline.
Derive Final Calibration Tables with solnorm=True
Since the absolute flux scale is determined from the switched power calibration, the calibration tasks affecting amplitudes in hifv_finalcals, namely bandpass and gaincal, needed to be changed to be executed with solnorm=True for the Rubin alert pipeline. This is selected by setting use_flux_cal=False in hifv_finalcals, where gaincal also uses normtype='median'
hifv_finalcals(use_flux_cal=False)
Add Steps to Flag Data
Flagging of baseband and spw edge channels, as well as RFI flagging of the calibrator and the target, are performed after applying all the calibrations in the Rubin alert pipeline.
hifv_flagdata(autocorr=False,baseband=True,edgespw=True,fracspw=0.01,hm_tbuff='manual',tbuff=[0.0],shadow=False,quack=False,template=False,online=False,clip=False)
hifv_checkflag(checkflagmode='bpd-vla')
hifv_checkflag(checkflagmode='target-vla')
Imaging
Once calibration is done, images are made of the science target and calibrator source. Auto-masking is used, the image maximum image size of 16384 pixels on a side from the standard JVLA imaging pipeline is retained, and the hm_cyclefactor is set to 2.0 to help prevent cases where high side-lobes stop masks from being generated in snapshot images.
hif_makeimlist(specmode='cont', intent='PHASE')
hif_makeimages(hm_masking='auto',hm_cyclefactor=2.0)
hif_checkproductsize(maximsize=16384)
hif_makeimlist(specmode='cont', intent='TARGET')
hif_makeimages(hm_masking='auto',hm_cyclefactor=2.0)
Summary of Changes to Existing Pipeline Tasks
The following changes to already-existing pipeline tasks were made to support the Rubin alert pipeline and are available with CASA/pipeline versions 6.7.4 and following:
- The call to gencal in hifv_priorcals was changed from using caltype='gc' to caltype='gceff'.
- A parameter 'use_flux_cal' (default=True for the standard JVLA pipeline) was introduced in hifv_finalcals that, when set to False, has the effect of setting solnorm=True in the call to bandpass (for finalBPcal.tbl), and setting solnorm=True and normtype='median' in gaincal (for finalampgaincal.tbl).
- A new recipe is included with the above changes.
For questions, suggestions, and other issues encountered with this pipeline or results obtained using it, please submit a ticket to the Rubin Alerts Department of the NRAO Helpdesk. If you encounter issues retrieving data from the archive, please include your request ID, if applicable, in your Helpdesk query.

Connect with NRAO