Letter from the Lead

Jeffrey Kern

It is my pleasure to welcome you to the first edition of the Common Astronomy Software Applications (CASA) Newsletter. This semi-annual publication is intended to provide the CASA user community an understanding regarding what the CASA team is working on and details about data processing using CASA. We will have recurring sections – such as a “Letter from the Lead” and an update from the CASA Helpdesk – as well as sections describing how CASA approaches various aspects of data processing.

The CASA team is hard at work on a number of projects. A major rework of the imaging capabilities designed to provide a more intuitive interface, and more flexible architecture, is being refined and tested. The transition to an optionally scratchless system is underway that will prevent the repeated need to write calibrated data to disk before the data can be inspected.  We anticipate that this will decrease the time spent waiting for tasks to complete and improve the user experience. Finally, we are in the midst of testing parallel processing through the CASA package. Details on all of these capabilities will be announced in subsequent editions of this newsletter.

David Wilner (Harvard-Smithsonian Center for Astrophysics) provided a short description of the CASA Users Committee activities for the January 2015 NRAO e-news. Based on the feedback at the September 2014 face-to-face meeting that the CASA project should seek to improve communication with the larger radio astronomy community, we have undertaken a number of initiatives, including this newsletter.

To receive future CASA News issues, be sure to subscribe by visiting the CASA Announcements list or click on the Subscribe button in the left column under this issue's table of contents.

In conjunction with the Helpdesk, we have created a webpage to provide feedback on tickets originally submitted to the Helpdesk that have made their way to the CASA development team to be addressed. On this page, you can see the Helpdesk ticket number, a short description, and the current status. Tickets marked as “scheduled” have been triaged to be addressed in one of the upcoming semi-annual CASA releases; those marked as “unscheduled” have not yet been assigned a release and are part of our regular planning process.

Finally, we are considering creating a new mailing list to provide a method for those working with CASA to interact and discuss their issues and developments. This is predominately designed as a peer-to-peer communication tool with little interaction from the CASA team. Of course, some topics are just too interesting not to comment on. In addition, the CASA team would use this as a communication path to announce pre-release versions of upcoming CASA releases for testing and notification of major infrastructure changes to the package that are not of interest to general users but may impact those working with CASA at a more technical level. Please indicate in the users survey (see below) if you would be interested in such a list.

ALMA Science Verification Imaging

Crystal Brogan

[click to enlarge]

This image was created in CASA from the ALMA science verification (SV) data acquired for the protoplanetary disk HL Tau during the 2014 ALMA Long Baseline Campaign (LBC). The image is a combination of the 1.3 and 0.87 mm (Bands 6 and 7) continuum data and was created accounting for a linear spectral index using multi-frequency synthesis and nterms=2. The angular resolution is 33.5 x 21.1 mas (P.A. +9 degrees). This image demonstrates both the transformational power of ALMA to resolve fine-scale structure in its most extended configurations and the innovative cleaning techniques available in CASA to create high fidelity images through the combination of high fractional bandwidth data.

The HL Tau SV data, including reference images, can be downloaded from the ALMA Science Portal and more information about the imaging used for the LBC SV data can be found on the following casaguide page. For more information on the CASA technique for accounting for spatial spectral index variations see Rau & Cornwell (2011, A&A, 532, A71).

2015 User Survey

The CASA team is conducting a user survey to better understand the CASA community.  The results of this survey will be used to prioritize developments in upcoming CASA releases and to better optimize our algorithms and architecture in line with the issues confronting our users. Even if you are not a frequent user of the CASA package, we would like to hear from you.

To participate in the CASA User Survey, please visit the Survey website no later than Wednesday, 18 March 2015.  The survey is anonymous and takes only about 10 minutes to complete.

User Support

Anand Crossley

CASA user support is provided through the NRAO and ALMA Helpdesks.  The NRAO Helpdesk is the proper choice for general CASA issues, including those originating at non-NRAO telescopes. The ALMA Helpdesk is the proper Helpdesk for CASA issues that pertain specifically to ALMA.

In 2014, NRAO received almost 500 new Helpdesk tickets from CASA users covering topics ranging from CASA installation to low-frequency calibration, to combining images and forming mosaics. We are glad to hear from you! Please visit our Helpdesks and let us know how we can better support your data reduction needs. 

Here are some tips on getting the most from your CASA Helpdesk experience, and a few words on new Helpdesk features available now and in the near future.

Helpdesk Tips
If you run into a problem while running a CASA command, attach the CASA log file to your Helpdesk ticket. Doing so will help us better understand your particular data processing situation and enables us to reply more promptly. Since CASA log files are generated in plain text, uploading them should not take long.

Please include the CASA version (e.g., Release 4.2.2) you are using when you submit your ticket. If you are not using the latest version of CASA, try upgrading first. The CASA developers are constantly addressing bugs and adding new features, and it is possible your issue has been addressed in a more recent CASA release.

Check to see if your operating system is officially supported by the CASA project. A list of officially supported operating systems is available on the Obtaining CASA web page. Note that while using unsupported operating systems may not cause problems in some cases, differences in system libraries can result in unexpected errors.  Try using a supported operating system if at all possible.

Recent and Upcoming Changes
If you request a CASA enhancement or report a bug, you can now monitor the status of your issue using the CASA Internal Ticket Monitor. Here you will find a list of CASA developer tickets associated with the Helpdesk. The list includes the JIRA tracking on development status, resolution status, and the CASA version in which the change will be released. You can locate your issue by drilling down the issue listing for the Helpdesk ticket number in which you submitted your concern.  If you do not remember the Helpdesk ticket number, just open the Helpdesk:  all your old tickets are saved there for future reference. Searches can identify key words, topic, reporter, assignee, as examples.

In 2015, we will be aligning both of our Helpdesks (NRAO and ALMA) to make navigation between the two easier. If you use CASA for processing ALMA data and/or data from another telescope, you will be able to manage your help desk inquiries with greater ease. You will also benefit from a consolidated “KnowledgeBase”, where articles from both Helpdesks have been collected.

Improved Helpdesk search capability is also on the horizon. Our improved search will make finding tickets, KnowledgeBase articles, and relevant web pages easier than ever.

We welcome your feedback on CASA user support.  Feel free to send us a ticket with questions or recommendations on how we can best support you.

Hardware Recommendations Update

James Robnett

The NRAO Data Management and Software Department periodically develops hardware recommendations. These recommendations are intended to help guide the user in deciding how to get the best performance for their money when purchasing a system for data reduction. In addition to concrete suggestions, they discuss the evolving balance between file I/O, memory, processing power, and the implications on experienced user performance.

New hardware recommendations for the 4.3 CASA release are now available. The updated recommendations primarily reflect predictions regarding the future parallelization of CASA and its effect and interaction with modern processors and I/O systems. This version also discusses the application of the new generation of larger and more affordable, solid-state disks to data reduction. 

Pipelines

Lindsey Davis, on behalf of the CASA Pipeline Team

Last year was a busy time for CASA pipeline development. The ALMA interferometry calibration pipeline achieved conditional acceptance in March 2014 and full acceptance in September 2014. The ALMA interferometry pipeline is now part of standard ALMA science operations. About 80% of current standard mode ALMA interferometry projects are, with supplemental manual flagging, being processed and delivered to the PI(s) via the pipeline.

After acceptance, the pipeline software was, for the first time, made available to the ALMA user community as an add-on to the standard CASA 4.2.2 release. Last year also saw the VLA scripted calibration pipeline successfully ported to the standard CASA pipeline framework used by the ALMA pipelines. During the same period, ALMA single dish pipeline development proceeded smoothly, including major heuristics and framework improvements. Late 2014 saw the start of the Observatory commissioning process for the VLA interferometry and the ALMA single dish calibration pipelines.

In January 2015, the next pipeline release was delivered to the ALMA Pipeline Working Group for testing and acceptance. This is the first of a planned set of ALMA Cycle 3 focused releases. New interferometry capabilities for this first release include: (a) low SNR bandpass calibrator handling heuristics; (b) flux value retrieval from the ALMA calibrator catalog; (c) new flagging heuristics; (d) improvements to the quality assessment metrics; (e) generation of low spectral resolution pilot science target image cubes; and (f) major upgrades to the pipeline web log infrastructure. ALMA single dish pipeline improvements include: (a) better flagging support; (b) improved automatic line detection for baseline subtraction; and (c) improved imaging heuristics. The VLA pipeline includes a number of web log and plotting improvements, as well as generation of diagnostic calibrator source images.

Future releases in the Cycle 3 ALMA development cycle will focus on: (a) continued improvements to the low SNR calibrator heuristics; (b) development of the first continuum and full cube science target imaging capabilities; and (c) increasing the number of observing modes that the pipeline can successfully process. Candidates for new observing mode support are bandwidth switching and band-to-band phase transfer projects. Major improvements to the web log handling of pipeline runs with multiple sessions and ALMA Science Data Models are planned.

The current ALMA pipeline user guides and documentation for Cycle 2 can be found at the
ALMA Science Portal under the "ALMA Science Pipeline" section. Updates for ALMA Cycle 3 will appear in March.

Single Dish

Kanako Sugimoto, on behalf of the CASA Single Dish Team

CASA is capable of processing data taken by single dish (SD) telescopes. CASA is used to process total power data taken by ALMA both in manual data reduction and pipeline processing. CASA also officially supports other single dish telescopes, i.e., the Nobeyama 45m telescope and Atacama Submillimeter Telescope Experiment (ASTE). CASA 4.3 has twenty tasks to process single dish data. SD task names in CASA begin with “sd,” e.g., sdimaging for imaging. These SD tasks cover the full processing lifecycle of single dish data: data import, calibration, and imaging.

Data import and calibration functionality in CASA currently depend on the Australian Telescope National Facility Spectral Analysis Package (ASAP). ASAP was originally developed at the Commonwealth Scientific and Industrial Research Organization and has been extended at National Astronomical Observatory of Japan (NAOJ) for use by the ALMA and Nobeyama 45m telescope. In the past several years, the single dish team has been developing and improving tasks and functionality to support ALMA as well as improving the performance of the ASAP-based tasks. It is important to realize that ASAP is based on the Scantable data format, which is different from the MeasurementSet (MS) used to store visibilities in CASA, although they share a common lineage from the CASA Core table system.

We are working on allowing the SD tasks to work directly with the MS. This will prevent potential confusion stemming from having two different data formats in CASA. In addition, this will allow reuse of much of the infrastructure, and a more common look and feel throughout the package. We have started the migration during our current development cycle. This is a major initiative that requires re-implementation of all SD processing tasks. We expect the full migration to take 1.5 to 2 years. Both new and old SD tasks will be supported until the full migration is completed.

The new SD tasks will take advantage of the Sakura library, a high-performance array-processing library developed by NAOJ including contributions from their CASA development team. The Sakura library takes advantage of the vector processing instruction sets available on new generations of processors to provide more optimized processing capabilities throughout the SD package. Our intention is to deliver the Sakura-based system on the same timescale as the completion of the support for MS-based processing.

Visibility Weights and Their Calibration

George Moellenbrock

Like any scientific observational data, interferometric visibility measurements, such as are produced by synthesis arrays like ALMA and the VLA, are subject to finite statistical measurement errors. Insofar as astrophysical conclusions are to be inferred from specialized averages of the visibilities – e.g., images, models fit to the data, etc. – it is important to be aware of how the data are ultimately weighted in these averages, especially considering the calibration requirements for visibility data, and the weight adjustment schemes used in the imaging process to optimize deconvolution properties, effective angular resolution, and surface brightness sensitivity. In this brief note, the basic properties of visibility weights and their calibration are described, first generically, and then for the specific cases of ALMA and the VLA.

Fundamentals 
The noise on the real or imaginary part of a discrete visibility measurement on a baseline formed from antennas i and j is expressed by the well-known formula^[1]:

,

where T_i and T_j are the total system temperatures (total measured powers, including contributions from the astronomical source, atmosphere and electronics) for each antenna expressed in the units desired for σ, dt is the integration time, and dν is the bandwidth. An additional factor describing the (digital) efficiency of the measurement (η ≤ 1.0) has been suppressed in the denominator; this has done for simplicity and has no bearing on the following discussion. Note that the errors on the real and imaginary parts are equal in magnitude and statistically independent. When forming a real image from complex visibilities, however, the noise contributions from real and imaginary add in quadrature, and therefore do not double the net weight of the measurement.

For unitless, normalized (by effective total power) visibilities, the T_i and T_j factors are omitted from the above equation, and the noise will be a function only of dt and dν. That this should be true should become clear as calibration and normalization are discussed below.

Formally, the statistical weight, w, of the visibility follows from textbook statistics^[2]: w = 1/σ².  These are the “natural” data weights, and this definition ensures that properly weighted averages of such data will yield net measurements with weights proportional to total integration time and total bandwidth. It is useful to think of the weights as describing the nominal information content of a measurement, and this leads to our usual understanding of the Sampling Theorem, wherein the 2dtdν factor as expresses the number of samples required to describe a normalized signal with no information loss. Furthermore, averages using non-natural weights will necessarily be less sensitive aggregations in net. This can be shown quantitatively by applying standard error analysis to the average of a collection of weighted data, with one sample over- or under-weighted by some factor. The net weight will be less in both cases, and this result can be generalized to any convention for adjusted weights. Note that if all weights are scaled by the same factor, there is no net sensitivity loss. This property was often leveraged by 20th century synthesis instruments that observed with uniform integration times and bandwidths as a way to avoid the necessity of accounting for other obscure, yet usually uniform factors, such as digital efficiency. It was not uncommon to initialize the weights uniformly to 1.0 in such cases.

Initialization
For modern instruments, where integrations times and bandwidths may vary among spectral windows during an observation, it is desirable to initialize the weights accordingly. This is how the CASA Science Data Model fillers (importasdm and importevla) initialize the weights in the MeasurementSet. Since the channel bandwidth is currently uniform within spectral windows, the initialized weights are not channelized. Except for the VLA switched power calibration described below, CASA assumes that the weights have been initialized to reflect per-channel, per-integration sensitivity. Note that the weight initialization does not include factors related to system temperature, nor to nominal relative collecting area among antennas in inhomogeneous arrays. These effects will be accounted for in subsequent calibration, either explicitly or implicitly. To support direct weight initialization, the initweights task has been added to CASA version 4.3. This task includes provisions for initializing channelized weights. This is desirable for ALMA (see below), and other cases where the weights may be significantly channel dependent. The statwt task can be used (after calibration) to estimate per-spectral window weights directly from the apparent noise in the visibilities. New statwt modes enabling higher spectral resolution estimates of the weights are currently under development, e.g., to support narrow-band RFI conditions at low frequencies at the VLA.

Weight calibration
When the visibilities are calibrated in a manner that changes their effective amplitude scale, the scaling of the weights must follow accordingly. Visibility calibration accounts for systematic effects in the data, and comes in many forms, including phase-like geometry-related (delay) calibrations, which nominally have no impact on the weights. If the required phase-like calibration implies loss of coherence within the integration time and/or channel bandwidth, a corresponding baseline-based weight adjustment reflecting sensitivity loss is formally required; CASA does not yet support such corrections, and assumes the phase-like calibration restores coherence faithfully. 

All amplitude calibrations, however, do require corresponding corrections to the weights. The principle is simple: if a visibility amplitude, A, is to be corrected by dividing by a factor, c, then the associated σ value should also be divided by c. The weights are therefore calibrated by multiplying by c². Thus, for antenna-based gain calibration, where a complex voltage scale factor, g_i, is calculated for each spectral window and antenna as a function of time – usually by solving on a source with visibilities known a priori – the visibility and weight calibration take the following form:

Unless the data have already been calibrated to uniform flux density units (Jy), the gains, g_i, will have antenna-based scale reflecting relative collecting area, including both surface accuracy and nominal collecting area. As such, the observed visibilities will reflect the available relative collecting area on each baseline, with baselines among smaller antennas showing smaller correlations. Note that the gain calibration can be factored into a sequence factors without loss of generality. Since the net voltage gain factor is generally antenna-dependent, application of gain calibration will correct the otherwise uniform initial weights to a more interesting distribution as a function of baseline and time, reflecting the relative information content arising from the net efficiency of each antenna's contribution to the measurements. Another way of looking at this is that the visibilities have been converted from uniform normalized units to astrophysically meaningful units (e.g., Jy), in which each baseline has a distinct sensitivity scale that may also be a function of time.

For ALMA, the calibration of the per-spectral window bandpass requires special consideration. Apparently, it is just a channelized version of the gain calibration, usually assumed stable with time. However, the bandpass shape is dominated by effects (e.g., filters, attenuators) realized well downstream of the first amplifier. Radiometry theory^[3] shows that the effective noise fraction is established at the first amplifier and, as such, the net bandpass filters the shape of both the desired signal and the noise. Only at the very edges of spectral windows, where throughput is least, the effective filter shape dominates the total noise contribution, and these channels are routinely flagged. Therefore, to a very good approximation, the bandpass shape conserves SNR across the spectrum, and the weight calibration should not be channelized, if the initialized weights are uniform in channel. In effect, the bandpass restores spectral uniformity in both the signal and the noise. In CASA 4.2.2 and later versions, the bandpass calibration takes the following form (subscript k denotes the frequency/channel axis):

The angle brackets denote channel averaging. In CASA prior to version 4.2.2, the bandpass weight calibration took a slightly different, but approximately consistent form, which was somewhat less stable to outliers in the bandpass shape.

Note that bandpass normalization in CASA is implemented in the power domain in a manner consistent with bandpass calibration:

This ensures that the application of a normalized bandpass does not change the weights. For unnormalized bandpasses, the putative normalization factor is the square root of weight calibration factor, and it is applied in a manner exactly consistent with ordinary gain calibration.

In applycal, CASA supports control of whether or not the weights are calibrated, via the calwt parameter. It is recommended that this always be turned on (though see below for special considerations for VLA data). This ensures that the nominal sensitivity is always recorded in the MeasurementSet. In the future, the calwt parameter may be removed, weights will nominally always be calibrated, and control of the use of the weights in downstream processing will be improved.

Instrument-specific considerations
Modern instruments like ALMA and the Jansky VLA obey the fundamentals described above, but the details of the online processes generating the delivered visibilities require some explanation.

ALMA delivers normalized visibilities, but unlike traditional instruments that would typically normalize by a single factor calculated – the geometric mean of lag zero auto-correlations, or equivalent – per spectral window, corresponding to a unique signal path through the instrument, ALMA normalizes per spectral channel within each spectral window. ALMA first does the traditional lag-zero auto-correlation normalization on both cross- and auto-correlations, then normalizes by the auto-correlation spectra in the frequency domain. This enables accurate normalization across the spectral window when atmospheric lines are present, which absorb astronomical signals and add excess power to the total system temperature.

The sole practical reason for spectral normalization is to yield uniform noise across the spectral window (and for all time, so long as integration and channel bandwidth remain fixed) so that the data may be archived with compression depending on a uniform scale factor. When filled into CASA, visibilities therefore have uniform weights over all times and baselines (per spectral resolution and integration time settings). The initial data processing steps include multiplicative calibration by system temperature spectra – nominally including atmospheric absorption effects – to restore physical units (K), effectively reversing the online normalization. For optimal accuracy, this demands that the weights be spectrally resolved, and CASA, as of version 4.3, supports channelized weight calibration by ALMA system temperature spectra. This is the only CASA calibration operation that requires and supports spectral weight calibration. The current default, however, is to calibrate the unchannelized weights by dividing by the mean of the spectral system temperature. 

The Jansky VLA does not deliver normalized visibilities. Instead, each baseline's correlation measurements are in units corresponding to the product of the engineering voltage units in which the incident radiation field is sampled and digitized. As such, the effective scale of the data is essentially arbitrary, varies with baseline, and can change with time, as input power levels vary (e.g., with elevation), although the net sampling process – including at each telescope and in the front-end of the correlator – is configured to deliver signals for correlation in a limited nominal range optimized for the correlation operation.

Since the effective noise on each baseline is a function of the power sampled at both antennas, and since this power varies with antenna, the effective raw data weights will be baseline-dependent. These weights are not calculated online. Data and weights filled into a CASA MeasurementSet therefore require initial calibration into more useful physical units, and this may be achieved by applying the “switched-power” calibration, a gain-like measurement obtained by taking, per antenna, the square root of a continuously monitored backend power contributed by a reference noise source injected synchronously at the frontend. In addition to supplying calibration to the units of the noise source (nominally K, but generally known only to an accuracy of ~20%), this calibration enables tracking and removing gain fluctuations (some of which may be deliberate at various stages) occurring in the relatively complicated analog systems of the antenna. More importantly, the switched power system also supplies a measurement of total power at each antenna (in the squared units of the engineering voltage sampling), and this information, along with the switched power gain, can be used to initially calibrate the weights appropriately.

Note that application of this calibration treats the visibilities and weights differently, so as to restore consistency in their units. At this writing, non-linear effects occurring in Jansky VLA switched power measurements – mild for 8-bit sampling modes, and considerably more significant for 3-bit sampling – are still under investigation, and the Jansky VLA pipeline is not yet applying switched power by default. Thus, Jansky VLA data weights are not scaled into units consistent with the visibilities, and weight calibration is therefore not performed for gain and bandpass either. Effectively, the Jansky VLA weights are considered uniform per bandwidth and integration time setting, a reasonable approximation. For more accurate results, use of the switched power calibration and full weight calibration should be considered. Post-calibration use of statwt, to detect appropriate weights directly from the data is suggested.

Conclusion
With weights properly initialized, refined (per instrument, as needed), and calibrated, the prospects for imaging sensitivity will be optimized, for any imaging-weighting scheme. The imaging weighting schemes are beyond the scope of this article and non-trivial, generally trading sensitivity (natural) for improved resolution and deconvolution accuracy (robust, uniform), and/or surface brightness sensitivity (taper). Also, the combination of data from disparate sources – different array configurations, single dish, etc. – will more likely be nominally optimal, if the visibility weights are properly calibrated in each dataset. Please consult the CASA Cookbook for more details on weight handling and calibration. Future developments in weight calibration will be described there and in future newsletter articles.

Notes

1. Thompson, Moran, and Swenson 2001, Interferometry and Synthesis in Radio Astronomy, 2nd edition (John Wiley & Sons, New York). Section 6.2 provides a comprehensive discussion of the interferometer noise response.
2. Bevington and Robinson 2003, Data Reduction and Error Analysis for the Physical Sciences, 3rd edition (McGraw-Hill, New York).
3. Rohlfs and Wilson 2006, Tools of Radio Astronomy, 4th edition (Springer, Berlin), section 4.4.

2015 Calendar of Events