Automated Self-calibration

The SRDP project is pleased to announce the release of ALMA and VLA imaging pipelines that include a new task to perform self-calibration of the science targets (hif_selfcal). We expect that the imaging pipeline will function properly in most circumstances. This task is validated and used for automated data processing by ALMA and the VLA and also included in the ALMA User-Defined Imaging (AUDI) capability.

This implementation of automated self-calibration heuristics is specifically intended for single pointing observations; mosaics and spectral scans will be supported in a future release (see Standalone tools). The heuristics are expected to perform well on datasets that have modest to high signal-to-noise (S/N), dependent upon the distribution of flux as a function of uv-distance. Long-baseline data can be successfully self-calibrated, depending on how much flux is present at the longest baselines. Heuristics to better optimize self-calibration of long baselines will be addressed in a future release.

Quick Access to AUDI in the NRAO Archive: https://data.nrao.edu

Link to the CASA+Pipeline tarball including selfcal: casa-6.5.4-9-pipeline-2023.1.0.124

Example Results from Automated Self-calibration

Without Self-calibrationWith Self-calibration

ALMA Band 6 images of the NGC 2071 IR star forming region from ALMA project 2018.1.01038.S (Cheng et al. 2022). The left image is without self-calibration, while the right image is from the automated self-calibration within the pipeline.

ALMA User-Defined Imaging Workflow

AUDI requests will attempt phase-only self-calibration by default, and if self-calibration is successful, both the non-self-calibrated and the self-calibrated images will be delivered to users. If self-calibration is unsuccessful, only the non-self-calibrated images are delivered. By design the AUDI workflows operate on only a single source and produce a single data cube per request. The self-calibration solutions are applied to both the continuum and spectral line data. In many cases, self-calibration primarily benefits the continuum data and the impact to the line data is negligible because the line emission is generally too faint to be dynamic range limited.

Self-calibration solutions are applied using applycal(applymode='calflag') which means that flagged gain solutions will result in data being flagged. After each self-calibration solution interval the S/N is verified to ensure that there is a net gain from that self-calibration interval and it is ensured that the beam has not increased by more than 5% in beam area from the pre-self-calibration image. If S/N decreases or the beam changes by > 5%, that solution interval is regarded as 'failed' and the last successful solution interval is restored.

When self-calibration is conducted with applycal(applymode='calflag'), the line data cube can have a marginally higher RMS due to the flagging applied. Users will have the ability choose whether the cube with or without self-calibration applied better suits their needs.

Interactive Pipeline Usage

Users can make use of this task in the interactive use of the ALMA and VLA imaging pipelines. In such a usage mode, self-calibration supports:

    • Multi-source data
    • Multi-band data (VLA only)
    • Parameters to make self-calibration more permissive/restrictive

Example ALMA imaging pipeline script:

# This CASA pipescript is meant for use with CASA 6.5.4 and pipeline 2023.1.0.124
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Atacama Large Millimeter/submillimeter Array')
context.set_state('ProjectSummary', 'telescope', 'ALMA')
try:
    hifa_importdata(vis=['myCaldMS_1.ms','myCaldMS_2.ms'], session=['session_1','session_2'], datacolumns={'data': 'raw','corrected': 'regcal_contline_all'})
    hif_mstransform()
    #hifa_imageprecheck()
    hif_makeimlist(specmode='mfs')
    hif_findcont()
    hif_uvcontsub()
    hif_makeimlist(specmode='cont',datatype='regcal')
    hif_makeimages()
    hif_makeimlist(specmode='cube',datatype='regcal')
    hif_makeimages()
    hif_selfcal()
    hif_makeimlist(specmode='cont', datatype='selfcal')
    hif_makeimages()
    hif_makeimlist(specmode='cube',datatype='selfcal')
    hif_makeimages()
    hifa_exportdata(imaging_products_only=True)
finally:
    h_save()

Example VLA imaging pipeline script:

# This CASA pipescript is meant for use with CASA 6.5.4 and pipeline 2023.1.0.124
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Karl G. Jansky Very Large Array')
context.set_state('ProjectSummary', 'telescope', 'EVLA')
try:
    hifv_importdata(vis=['myCaldMS.ms'], datacolumns={'data': 'raw','corrected': 'regcal_contline_all'})
    hifv_flagtargetsdata()
    hif_mstransform()
    hif_checkproductsize(maximsize=16384)
    hif_makeimlist(specmode='cont',datatype='regcal')
    hif_makeimages(hm_cyclefactor=3.0)
    hif_selfcal()
    hif_makeimlist(specmode='cont',datatype='selfcal')
    hif_makeimages(hm_cyclefactor=3.0)
    hifv_pbcor()
    #hifv_exportdata(imaging_products_only=True)
finally:
    h_save()

Task Parameters

We list the most relevant parameters here that users may want to experiment with, the less relevant parameters are listed below the line.

  • field (string) - field names to self-calibrate e.g., "HL_Tau"; default = "" which will self-calibrate all sources
  • apply_cal_mode_default (string) -  Apply mode to use for applycal task during self-calibration; default = 'calflag'; options: 'calflag' ,'calonly', 'calflagstrict'
  • amplitude_selfcal (boolean) - Attempt amplitude self-calibration following phase-only self-calibration; default = False
  • gaincal_minsnr (float) - Minimum S/N for a solution to not be flagged by gaincal; default = 2.0
  • minsnr_to_proceed (float) - Minimum estimated self-cal S/N computed on a per solution interval, per per antenna basis, used to determine whether to attempt self-calibration for a source at a given solution interval; default = 3.0
  • delta_beam_thresh (float) - Allowed fractional change in beam area for self-calibration to accept results of a solution interval; default = 0.05

Less relevant parameters to adjust:

  • rel_thresh_scaling (string) - Scaling type to determine how clean thresholds per solution interval should be determined going from the starting clean threshold to 3.0 * RMS for the final solution interva'; default='log10', options: 'linear', 'log10', or 'loge' (natural log)
  • dividing_factor (float) - Scaling factor to determine clean threshold for first self-calibration solution interval. Equivalent to (Peak S/N / dividing_factor) *RMS = First clean threshold;  however, if (Peak S/N / dividing_factor) *RMS is < 5.0; a value of 5.0 is used for the first clean threshold. default = 40.0 for < 8 GHz; 15.0 for > 8 GHz
  • check_all_spws (boolean) - The S/N of mfs images created on a per-spectral-window basis will be compared; default=False
  • inf_EB_gaincal_combine (boolean) -If True, the gaincal combine parameter will be set to 'scan,spw'; if False, the gaincal combine parameter will be set to 'scan'; default=False; the only applies to the first solution interval that is computed over the length of an entire Execution Block
  • spw (string)- spectral windows to self-calibrate; default = ""; all science spws will be self-calibrated
  • apply (boolean) - apply final selfcal solutions back to the input Measurement Sets; default = True

 

Standalone self-calibration tools

The ALMA/VLA imaging pipeline implementations of self-calibration are based on the standalone automated self-calibration implementations developed by John Tobin and Patrick Sheehan. Users interested in further investigation/customization of the detailed heuristics can access the code on github. Note that the tunable parameters for the standalone versions might have different names than those listed above. Also, there may be new parameters and some parameters that have been removed, see the documentation for the specific standalone versions.

Stable Version (heuristics used by current pipeline): https://github.com/jjtobin/auto_selfcal/

The development version has new heuristics improvements to better handle lower S/N long-baseline data, initial support for mosaics, and spectral scan support. The development version is the basis for pipeline improvements for the Fall 2024 release. Branches from the development version also host further heuristic development.

Development Version: https://github.com/psheehan/auto_selfcal

 

Acknowledgements:

We acknowledge the adaptation of some routines and methodology published by the DSHARP (Andrews et al. 2018) and eDisk (Ohashi et al. 2023) ALMA large programs in the development of the automated self-calibration procedures. Some code from these projects is reused in the standalone and pipeline implementations of self-calibration. The development and implementation of self-calibration was a joint effort between the NRAO SRDP Project, the ALMA Pipeline Working Group, and the ALMA and NRAO pipeline development teams.

 

Help with SRDP and self-calibration:

Please visit the NRAO helpdesk and submit a ticket under the ‘Data Products’ Topic.

Connect with NRAO

The National Radio Astronomy Observatory and Green Bank Observatory are facilities of the U.S. National Science Foundation operated under cooperative agreement by Associated Universities, Inc.

Founded in 1956, the NRAO provides state-of-the-art radio telescope facilities for use by the international scientific community. NRAO telescopes are open to all astronomers regardless of institutional or national affiliation. Observing time on NRAO telescopes is available on a competitive basis to qualified scientists after evaluation of research proposals on the basis of scientific merit, the capability of the instruments to do the work, and the availability of the telescope during the requested time. NRAO also provides both formal and informal programs in education and public outreach for teachers, students, the general public, and the media.

The National Science Foundation (NSF) is an independent federal agency created by Congress in 1950 "to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense..."

AUI collaborates with the scientific community and research sponsors to plan, build, and operate cutting-edge facilities. We cultivate excellence, deliver value, enhance education, and engage the public.