SRDP > Features

Features

by Davis Murphy last modified Jun 19, 2019 by John Tobin

Quality Assured calibration for the VLA

The VLA calibration pipeline has been run on data immediately after collection since 2013 and operates on all data that are setup properly to run through the VLA pipeline. Data that are properly setup include: a Bandpass calibrator, a flux density calibrator, complex gain calibrators visited at recommended cycle times, and properly set scan intents in the Observation Preparation Tool (OPT). For bands that require reference pointing, the pointing is conducted at the recommended once per hour time interval. Pulsar data and Solar data are not processed in the VLA pipeline; the pipeline also does not currently support P-band data.

Data that are successfully processed by the pipeline for selected projects will then be reviewed by NRAO Data Analysts (DAs) and Scientists following a Quality Assurance (QA) process. As part of the QA process, the DAs and/or Scientists may perform additional flagging on calibrators and targets, and re-run the pipeline with additional flagging, if necessary. The goal is to provide the users with a calibrated dataset that is ready for imaging. The final calibration and flagging will be ingested into the archive. Raw data will still be available to the PIs and Co-Is immediately.

The calibration pipeline will still be run on L, S, C, and X-band data during pilot operations, but these data may not go through the QA process.

Spectral line is not currently be supported by the QA process. The continuum portion of spectral line datasets will still be QAed. Users requiring spectral line observations should still follow the instructions on https://science.nrao.edu/facilities/vla/data-processing/pipeline#section-22 to calibrate their VLA spectral line data using the pipeline.

ALMA Calibrated Measurement set download

PIs and Delegees will be able to download calibrated measurement sets for their proprietary ALMA data once ingested into the ALMA archive and mirrored in the NRAO archive. Archive users can also download calibrated measurement sets for any non-proprietary data. Calibrated measurement sets can currently only be downloaded for datasets that were fully calibrated by the ALMA pipeline; manual reductions are not supported. Only Cycle 5 data and later from the 12m array and ACA can be restored using this process due to the way the data were archived in earlier cycles.

The calibrated measurement sets are generated on-the-fly when requested. As such, there will be a time lag between when a request is submitted and when the data are ready to be downloaded. The time for the calibrated data to be ready for download depends strongly on the size of the dataset. The restoration process reapplies the flagging done by the pipeline and apply the calibration tables produced and archived by the ALMA pipeline. The manual reductions follow a different workflow and are not compatible with being restored to a calibrated state in the same manner.

Time until dataset ready for different raw data sizes:

10 GB  -  ~1 hr

100 GB - ~10 hr

1 TB - ~60 hr

VLA Calibrated Measurement set download

PIs and Co-Is will be able to download calibrated measurement sets for their proprietary VLA data once calibration and QA have been performed. Archive users can also download calibrated measurement sets for any non-proprietary data. Calibrated measurement sets can currently only be downloaded for datasets that were fully calibrated by the VLA pipeline from mid-August 2016 and later (CASA 4.7.1 and later) Pulsar data, Solar datam and P-band  are not supported in the VLA pipeline.

The calibrated measurement sets generated on-the-fly when requested. As such, there will be a time lag between when a request is submitted and when the data are ready to be downloaded. The time for a calibrated data to be ready for download depends strongly on the size of the dataset. The restoration process applies hanning smoothing, reapplies the flagging done by the pipeline, applies the calibration tables produced and archived by the VLA Pipeline, and runs statistical weighting.

Updated Archive Interface

The archive frontend has been completely revamped to feature an updated interface to enable data discovery. The archives serve both Jansky VLA and ALMA data initially and will soon serve VLBA, GBT, and Legacy VLA data as well.

Images from the VLASS survey are also being served by the archive, but only for epoch 1.1 at present. VLASS Pilot and 1.2 will soon follow.

Users can search for data in a variety of ways and initiate restorations of datasets to a calibrated state in the archive.

ALMA Cube Imaging

User-defined data cubes can be produced for proprietary ALMA data (by the PI or delegee) and non-proprietary data. Users can specify the spectral width and channel width in frequency or velocity units to produce a data cube that may better fit their science needs. Continuum imaging is also produced as part of this process in case it is needed for interpretation of the spectral line data. The cube imaging process makes use of the ability to restore a calibrated measurement set and the ALMA imaging pipeline. This feature has the same limitations as the calibrated measurement set download in that it can only operate on pipeline-reduced Cycle 5 data and beyond.

The imaging pipeline will operate using the current version of the ALMA imaging pipeline to take advantage of the most recent improvements to continuum finding and automasking. Thus, imaging products may be different and likely superior to those delivered in Cycle 5.

The cube imaging process functions as follows:

  1. The selected dataset is restored to a calibrated state using the same process for download of a calibrated measurement set
  2. Imported into the ALMA imaging pipeline
  3. Custom cube(s) and continuum images are produced using special imaging pipeline recipe