Letter from the Lead

Jeff Kern (NRAO)

The CASA development team is putting the final finishes on the CASA 4.6 release. The 4.6 development cycle was spent predominantly on efforts to improve the reliability and robustness of the package. This included a rework of the packaging mechanism to decrease the likelihood of library mismatches, several refactoring initiatives, and the expansion of our testing subsystem. Efforts to decrease the amount of disk access required by CASA processing are ongoing – see the article on MSTransform in this newsletter – as well as developments targeted to support the ALMA and VLA pipelines.  This will be the final CASA release to support Red Hat Enterprise Linux 5. RHEL6 will continue to be supported, and we are planning to add support for RHEL7 with the CASA 4.7 release (Fall 2016). On the OS X side, CASA 4.6 and 4.7 will be released with support for OSX 10.10 and 10.11.

There is still space available at the 15th Synthesis Imaging Workshop. Held in Socorro, New Mexico, 1-8 June 2016, the workshop will be comprised of a week of lectures on aperture synthesis theory and techniques at a level appropriate for graduate students in astrophysics. The workshop also includes two days of practical tutorials demonstrating data collection, calibration, and imaging of various types of data, including new data from the Jansky VLA, ALMA and the VLBA.

In addition to the usual articles on the work of the CASA development team, this newsletter includes articles from several outside collaborations working to extend the capabilities of the CASA package, as well as a note from the CASA Users Committee.

CASA Users Committee

Thibault Cavalié (CUC Chair), on behalf of the Committee.

Abstract: The CASA Users Committee (CUC) was formed in 2014 on the initiative of the NRAO Data Management and Software Department. Its principal role is to provide advice on the user community perceptions of the Common Astronomy Software Applications (CASA) package, primarily for ALMA and VLA observations. General CUC information and reports are on-line. On behalf of the CASA development team, the CUC would like to encourage users to follow CASA-related news and background information using the new CASA Newsletter. We also invite users to partake in a continued User Survey meant to guide future development efforts. We encourage the CASA users to contact their CUC representatives to provide feedback regarding capabilities, usability, reliability and performance of the CASA software package.

The National Radio Astronomy Observatory (NRAO) Data Management and Software Department chartered the CASA Users Committee (CUC) in 2014. The Committee’s primary role is to provide advice on the user community perceptions of the Common Astronomy Software Applications (CASA) software package, primarily for observations acquired with the Atacama Large Millimeter/submillimeter Array (ALMA) and the Jansky Very Large Array (VLA).

The Committee comprises ten experts. Five are appointed by the NRAO Assistant Director for Science Support and Research, two are appointed by the East Asian ALMA Regional Center (ARC) manager at the National Astronomical Observatory of Japan (NAOJ), two are appointed by the European ARC manager at European Southern Observatory (ESO), and one is appointed by the head of the Department of Science operations at the Joint ALMA Observatory (JAO).

The current CUC members are listed below, which region they represent (NA = North America, EU = Europe, EA = East Asia), their position and institution, and their primary scientific and technical interests.

• Rachel Akeson (NA): Astronomer at the NASA Exoplanet Science Institute, Caltech. Millimeter interferometric observations of disks around young stars, especially using ALMA.
• Thibault Cavalié (EU, Chair): Planetary scientist at the Laboratory for Space Science and Astrophysical Instrumentation (LESIA, Paris Observatory, France). Sub-mm and mm single dish and interferometric observations (ALMA, SMA) of planetary atmospheres.
• Ximena Fernández (NA): NSF Postdoctoral Fellow at Rutgers, the State University of New Jersey. Frequent VLA user, observing HI in galaxies across cosmic time.
• Rachel Friesen (NA): Dunlap Fellow at the Dunlap Institute for Astronomy & Astrophysics, University of Toronto. Early stages of star formation, structure and composition of dense gas in star-forming molecular clouds through observations at multiple single-dish and interferometer telescope facilities.
• Lizette Guzmán (CL): ESO Fellow working for ALMA, European Southern Observatory. Multi-wavelength observations of planetary nebulae, with particular interest in the formation of dust and PAHs in these objects; Optical and infrared spectroscopy, radio interferometry and chemical modeling.
• Tomoya Hirota (EA): NAOJ, Mizusawa VLBI Observatory, assistant professor. VLBI astrometry, star-formation and astrochemistry with ALMA.
• Daniel C. Jacobs (NA, Deputy Chair): Arizona State University. Observing Reionization and Dark Ages with 21cm line. Using CASA to commission new radio arrays (PAPER, MWA, HERA), developing new algorithms and pipelines to address challenges posed by wide-field, large N arrays.
• Alexander Karim (EU): Staff member for observational cosmology at the Argelander-Institute for Astronomy (Bonn University, Germany). Frequent user of sub-mm/radio interferometry including ALMA and the VLA, large programs using full-band spectral scans and large-area continuum mosaics.
• Kazushi Sakamoto (EA): Astrophysicist at the Academia Sinica Institute of Astronomy and Astrophysics (Taiwan). Frequent user of radio interferometers (ALMA, SMA, VLA) for line and continuum observations of nearby galaxies.
• David Wilner (NA): Astrophysicist, Harvard-Smithsonian Center for Astrophysics. Radio-millimeter-submillimeter observations of star and planet formation.

The CUC holds quarterly telecons with Brian Glendenning (NRAO Assistant Director - Data Management & Software Department) and Jeff Kern (CASA Group Lead) and an annual face-to-face meeting with the broader CASA Team. The Committee then prepares a report presenting its recommendations to the CASA Team for the coming year.

The initial CUC meeting was held at NRAO – Socorro on 22-23 September 2014, and a meeting report was compiled under the supervision of David Wilner (CUC Chair, 2014-15). This document and other relevant materials are available at the CUC wiki page. The CASA Project Manager responded to this report on 9 December 2014.

Following CUC recommendations, a CASA Newsletter is distributed every semester. On behalf of the CASA development team, the CUC would like to encourage users to follow CASA-related news and background information by subscribing to the CASA Newsletter.

A CASA User Survey was conducted in March 2015. As recommended by the CUC, the CASA Group Lead described the survey results in the CASA Newsletter (Issue #2).

The CUC held its second face-to-face meeting at NRAO – Charlottesville on 14-15 October 2015. Former CUC member John Carpenter (NA) – now ALMA Observatory Scientist at JAO – had to leave the Committee in mid-2015 and was replaced by Ximena Fernández (NA). This nomination was particularly welcome as it followed one of the 2014 CUC recommendations to increase the VLA representation in the CUC.

The goal of the second CUC meeting was to: (a) present users feedback on CASA software capabilities, usability, reliability and performance; (b) review the actions taken by the CASA Project in response of the first CUC recommendations in 2014; and (c) comment on recent development initiatives. The 2015 CUC report was delivered 9 November 2015, and the CASA Project Manager responded 14 January 2016.

The CASA project will be re-opening the CASA User Survey. We invite users to partake in this User Survey to guide future development efforts.

Finally, since many of the original Committee members are likely to rotate off in the next year or two, community members interested in serving on the CUC should contact their relevant regional group.

In the meantime, we encourage CASA users to contact their CUC representatives to provide the Committee with feedback regarding capabilities, usability, reliability and performance of the CASA software package. This will help us prepare the next face-to-face meeting and relay the users priorities, concerns, needs and more.

Using the CASA Helpdesk

Anand Crossley

Need advice on the best way to process your data? 
Have you encountered a CASA error, but are unsure what to do?
Perhaps you have thought of an enhancement that would make CASA more useful? 
Submit a ticket to the helpdesk!

The helpdesk system is one of our primary tools for supporting CASA users. Simply put, our goal is to help you do great science. Over the past year, we received 600 tickets from CASA users! We are glad to hear from you and encourage all our users to make use of this resource. We have collected here some guidelines for effectively submitting CASA issues to the helpdesk.

• Send us all error messages.  If you encounter an error or suspect a problem with CASA, send us all the relevant information. Attach your CASA log file to the helpdesk ticket and copy-paste your terminal output into the helpdesk ticket. The terminal output and CASA log messages are not always the same and having both can help us solve your problems faster. If your error occurred while running a CASA task, please also attach the <task>.last file.
• What CASA version and operating system are you using?  Be sure to enter the CASA version and operating system in the custom fields at the top of the helpdesk ticket submission form. This information is essential for diagnosing problems and finding solutions.
• What kind of hardware are you using? How much RAM, how much free disk space, and are you running CASA on your local machine or on a remote server? If you are running remotely, are you using VNC or some other mechanism to access the remote machine? If you are using an observatory-provided visitor account, tell us your account username.
• What data are you using?  If you obtained your data from one of our archives, we need to know the unique identifier for the data set. For ALMA data, tell us the member OUS ID, or the project code and scheduling block name. For VLA data, tell us the file set ID (full filename listed in the archive) or the project code and scheduling block number. If you obtained your data somewhere else, please give us as much information as possible about the origin. If possible, attach the output of listobs to the helpdesk ticket so we can quickly assess the type of data you have. If your data are on disk at an observatory facility, tell us where we can find it, and make sure the file permissions allow us access.
• History is important, especially when it comes to CASA data. Measurement sets, in particular, can become corrupted under some circumstances, and when that happens no amount of expert advice will solve your problem. So tell us what you have done to your data since pulling it from the archive.  
• Tell us your end goal. There is often more than one way to do it in CASA. If you can explain your high-level goals before getting into details, we may be able to identify an easier route.
• Your experience with CASA and synthesis imaging will help us compose a response suitable to your skill level. Have you been using CASA regularly for years or did you download CASA for the first time last week? Are you new to synthesis imaging and radio astronomy or have you been working in this field for many years?
• What resources have you already explored? If you have already reviewed the CASA Cookbook, CASA Guides, or helpdesk knowledgebase, tell us exactly what sections you reviewed and whether or not they were applicable to your problem. This information is particularly useful for general questions about technique and procedure.

The NRAO helpdesk is the proper place for general CASA issues. The CASA user support staff available via the NRAO helpdesk tend to have more expertise with the VLA, but expertise in CASA data reduction for other instruments, such as the ATCA, is also available.

The ALMA helpdesk should be contacted for all CASA issues that pertain to ALMA and ALMA data processing. Each ALMA Regional Center – North America, Europe, and East Asia – has its own set of CASA staff with expertise supporting ALMA. For highly technical questions, all regional centers share resources to provide the best possible answers.

The NRAO and ALMA helpdesk staff aim to provide an equal level of user support. You can expect a response to your inquiries within two business days when using either helpdesk. All support staff communicate with the CASA software development team and can submit bug reports and enhancement requests.

We welcome your feedback!  Feel free to let us know how we are doing and how we can best support your data processing goals. You can provide feedback in your existing CASA ticket or open a new ticket in the “General Queries” helpdesk department.

ALMA Data Mining Toolkit

Lee G. Mundy (University of Maryland) and the ADMIT Team

Figure 1: Example ADMIT output showing line identification and moment maps.

[click to enlarge]

The ALMA Data Mining Toolkit (ADMIT) is a value-added Python software package that integrates with the ALMA archive and CASA to provide scientists with quick access to traditional science data products such as spectra, line identification, moment maps and PV diagrams, as well as providing a platform for new innovative tools for exploring data cubes and derived products. ADMIT is an ALMA Development Project led by University of Maryland, University of Illinois, and NRAO. 

The goals of the ADMIT package are to:

• make the scientific content of image cubes immediately available to ALMA Archive users
• enable scientists to easily, reproducibly, and systematically analyze datasets
• create an analysis infrastructure that allows users to build new tools
• provide new types of tools for mining the science in ALMA data
• increase science mining of the ALMA Archive by building xml data descriptors, and
• overview science products that can be utilized into future upgrades of the ALMA archive.

We are working with the North American ALMA Science Center (NAASC) and ALMA to implement the archive data products in concert with the rollout of the pipeline imaging products to users. The home version of ADMIT is currently being beta-tested and will become broadly available 30 April. The home version allows users to create and display their own data products and requires CASA 4.6.0 on a local machine.

Figure 2: Sample ADMIT output illustrating a position-velocity diagram for a region with a CO outflow.

[click to enlarge]

ADMIT Details
ADMIT provides a user-friendly environment for viewing, creating and customizing science data products from ALMA data image cubes (exclusively images, not (u,v) data). ADMIT is built around a data-flow model. In the implementation for the ALMA Archive this will be a fixed flow that produces a standard set of archive products that will be available along with ALMA data released to you. In the implementation for scientist use, the data flow and re-flow model allows incremental and iterative analysis of image cubes, which builds a final flow that can be run reproducibly and implemented over multiple set of similar data in a survey.

Figures-1 and -2 illustrates the types of science products and figures created automatically by the ADMIT pipeline processing. The science products are viewable in web browsers or file browsers such as Nautilus. Science products such as data cubes for individual lines and moment maps are written as CASA images as well.

An example of the standard set of Archive products are available on the ADMIT website. Note that the ADMIT browser view adopts the ALMA pipeline product viewer format. Two views of the data flow, the long file, and the Line-ID Editor are shown across the top. Down the page are details for each of the ADMIT Tasks, which can be viewed by clicking in the bar.

Become an ADMIT Beta-tester
We are seeking currently seeking beta-testers for desktop use who have ALMA, SMA, CARMA, VLA, or single dish FITS data cubes. To use ADMIT, you must have CASA and Python installed on your computer.

If you would like participate in beta testing, please contact the ADMIT Team.

Status of CASA-based Pipelines

Lindsey Davis, on behalf of the CASA Pipeline Team

Pipeline team activities over the past six months have focused on: (a) supporting the first two ALMA imaging pipeline releases through their testing and validation; (b) improving the flagging and calibration heuristics for all three operational pipelines; (c) supporting observatory pipeline operations; and (d) participating in efforts to coordinate pipeline development activities across observatory / project boundaries where appropriate.

The most recent public pipeline release is available with CASA 4.5.2. This release includes the ALMA Cycle 3 interferometry and single calibration pipelines, a new version of the Jansky Very Large Array (VLA) calibration pipeline currently undergoing testing and validation, and the ALMA Cycle 3 informative imaging pipeline for use at the JAO and the ARC(s). Highlights of this release include: (a) routine production of per source / per spw continuum images and lines cubes, including support for large cubes; (b) High Performance Computing (HPC) support for selected plotting, baseline fitting, and image cleaning tasks; (c) improved gain calibration based flagging; (d) improved support for antenna position corrections; and (e) many quality assessment scoring and web log enhancements.

Although the CASA 4.5.2 release of the ALMA informative imaging pipeline met the original requirements, more recent and focused testing on a more representative set of ALMA projects by the North American ALMA Science Center revealed a number of shortcomings in the adopted line finding, continuum subtraction, clean threshold, and number of iteration determination heuristics. To address these problems the pipeline working group generated a new set of imaging requirements that require changes to the pipeline heuristics and workflow. Development to address the new imaging requirements is already underway. In addition to addressing the imaging issues, we are also developing: improved low SNR calibrator heuristics for both the ALMA and VLA interferometry pipelines; robust end-to-end HPC operations; and more improvements to the flagging heuristics.

Most of the early pipeline development, particularly pipeline framework development, was driven by the ALMA project. The VLA pipeline subsequently adopted that framework. Since the acceptance of the VLA pipeline for operations last year, an effort has been made to more closely coordinate ALMA and VLA pipeline development, particularly heuristics development. This is benefiting both pipeline projects as well as feeding back into CASA development. The current pipeline team developers routinely contribute to both projects. Most recently the pipeline team began responding to issues raised by testing the VLA calibration pipeline on VLA Sky Survey test data.

The pipeline is also becoming more involved in outward facing observatory activities. The ALMA and VLA interferometry pipelines were demonstrated together at the last CASA Users Committee meeting. The pipeline team participated in the most recent ALMA long baseline campaign as the first Cycle 3 long baseline data came off the telescope. Recently, the CASA pipeline for the VLA was, for the first time, included in a VLA data reduction workshop.

Over the next year, pipeline work will continue to focus on imaging pipeline development, HPC operations, supporting new observing modes, and the production of science ready data products.

Processing Australia Telescope Compact Array data with CASA

Mark Wieringa, CSIRO Astronomy and Space Science

The use of CASA for data reduction and analysis of Australia Telescope Compact Array (ATCA) data is increasing and CSIRO now provides support for the use of this package, in addition to the MIRIAD package. In late 2014, a Memorandum of Understanding was signed between CSIRO and NRAO to facilitate user support and code development for ATCA data in CASA. Under this agreement, most NRAO helpdesk queries related to importing and processing of ATCA data will go to CSIRO first for diagnosis. In practice, this has meant that most ATCA-related queries are resolved without having to forward them to the CASA developers.

Two tasks in the CASA 4.6 release are available to import ATCA data into CASA: importmiriad and importatca. The importmiriad task is meant for users who do their data import and/or initial calibration in MIRIAD and would then like to transfer the visibility data to CASA for further imaging and analysis. The task can be used as in the following example:

   importmiriad(mirfile='ngc5921.uv',vis=ngc5921.ms',tsys=True)

This command will convert a MIRIAD file into a MeasurementSet using the Tsys data to set the weights. One thing to keep in mind is that importmiriad does not apply MIRIAD calibration tables during the conversion, so make sure you apply these before importing the data.

The importatca task can read files in the ATCA archive format (RPFITS) and convert them into a MeasurementSet for calibration and imaging in CASA. As an example, the command

     importatca(files='*.C1234',vis='c1234.ms',options='noac',edge=10)

will read all the RPFITS files ending in '.C1234', discarding the auto-correlations, and flagging the 10% of the channels at the bandpass edges. Both tasks can import CABB zoom data with as many as 34 simultaneous frequency bands, multiple sets of frequencies and multiple sources.

Wideband primary beam models for ATCA have been part of CASA since release 4.4. These models use interpolation in frequency between 1-D radial polynomial beam fits. To implement this in CASA required updates to the PBMath hierarchy of classes to allow frequency interpolation.

The ability to apply time and frequency dependent opacity corrections for high frequency ATCA observations has just been completed and will be available in the next release. The existing scheme for opacity correction – using the tasks plotweather and gencal – is somewhat specific to the Jansky Very Large Array. To emulate the ATCA scheme some calibration code was added to fully implement the TfOpac type gain calibration and populate it with values from either the built-in CASA ATM model (available in the at tool) or the ASAP/Miriad model (based on Liebe's atmosphere model).

A recipe for the processing of ATCA/CABB data is under development and will appear as a casaguide when ready. Feedback on all ATCA-related features is welcome via the helpdesk.

The Cube Analysis and Rendering Tool for Astronomy

Erik Rosolowsky, CARTA Project Scientist

A screen capture of the CARTA viewer illustrating the image rendering panel, navigation tools, and an image histogram displaying with the same color map as the image.

[click to enlarge]

Through the ALMA Development Program, the University of Alberta and the University of Calgary are collaborating with NRAO to create the Cube Analysis and Rendering Tool for Astronomy (CARTA). CARTA is a visualization project that will provide the functionality of the CASA viewer combined with the stability and performance that comes with a modern software application. Currently, the CARTA project has developed the core application and is rapidly implementing new features to deliver the tools that radio astronomy users need. The CARTA project will deliver software to the ALMA project on 1 September 2016, when it will become part of the CASA software package and join in the release process alongside the current CASA viewer.

Multiple Interfaces
The CARTA viewer uses several different front-ends to connect to the plotting application. Many users will interact with CARTA as a stand-alone desktop application or invoke it from within CASA. CARTA runs on Mac, Ubuntu, and CentOS/Red Hat operating systems. In addition to the desktop application, CARTA can run in a server mode, where the visualization engine runs within a data archive and users can connect to the visualization server using their web browser. Since CARTA uses a GUI built around JavaScript, the desktop application and the server-side application have a common front end. In the long run, we plan to deploy CARTA in the NRAO and ALMA archives.  This server-based deployment will provide a method to visualize very large data sets by running servers on high-performance visualization hardware. Finally, CARTA can be controlled from Python using a scripted interface for operation within a pipeline environment.

Plug-in-based Architecture
Most functionality for CARTA is developed using a plug-in approach, with a relatively small piece of software containing the application core. In addition to the plugins that provide core functionality, users are able to provide their own plug-ins written in C++, JavaScript or even Python. The core software tracks the state of the viewer application and can save the state to a file. Users can take advantage of this state by customizing their viewer layout, saving that configuration or even a particular visualization session.

Current Development State
We aspire to include all the functionality in the current CASA viewer, including many features requested by users. Our development has focused on creating the new architecture and the current state of the tool includes an image viewer and a "2 + 1" data cube viewer, which moves through the channels of a data cube using movie controls.  The viewer includes histograms, image statistics, contouring, color table, and full coordinate functionality. We are actively developing multiple layer functionality (for RGB composites), regions of interest, and profiles. Not all features of the current CASA viewer will be in place by 1 September, but development is proceeding rapidly and will continue within the CASA team.

Beta Testing
CARTA v0.6 is now available for community review. In this testing phase, the development team wants to gather feedback about how to improve the current tool and what features should be prioritized in the next phase of development. If you would like to participate in the CARTA beta testing, please visit the project webpage to download a beta version and contact the development team. A user-feedback session will be held in Charlottesville on 8 April 2016.  Please contact Erik Rosolowsky if you are interested in attending or providing feedback.

The MSTransform Framework

Justo González & Sandra Castro (ESO)

Design principle and major features
The idea behind the mstransform module is to perform a series of transformations on the data, without having to read and write the intermediate results (MSs) in between, thus reducing the I/O overhead (see Fig. 1).

Figure 1: Classic sequence of transformation with intermediate I/O steps (above) and mstransform approach w/o intermediate I/O steps (below)

[click to enlarge]

The first version of mstransform in CASA 4.1 included the classical split operations (time/channel average), plus the cvel functionality (spectral combination, re-gridding, change of reference, and doppler shift correction) and the Hanning smooth functionality. Shortly after we added a SPW separation feature to separate large SPWs into smaller chunks (particularly useful for analysis of low frequency Jansky VLA data), the capability to re-grid each SPW separately (w/o combination) and full support for single dish (SD) data.

The major steps forward since the first version have been the introduction of the weight spectrum formalism in CASA 4.3 [1] the development of a plugin interface for mstransform that can connect with applications like plotms (for instance providing on-the-fly averaging) in CASA 4.4, and the introduction of on-the-fly (OTF) calibration through a calibrating iterator using the callib syntax in CASA 4.5 [2].

Task mstransform also fully supports parallel processing of an MS. It is possible to create an output Multi-MS and apply all the transformations in parallel using the task interface, or a simpler interface in task_partition [3].

MSTransform task family
The mstransform framework has a direct interface via the task mstransform; and since CASA 4.5, a new series of tasks reproduce the existing simpler interfaces but run the mstransform framework underneath: cvel2, split, hanningsmooth and partition. For the case of split and hanningsmooth the mstransform-based tasks will become the default in CASA 4.6.

Performance
One example of the performance gain obtained by skipping I/O steps can be observed by comparing cvel versus cvel2. Both tasks can perform a pre-hanning smooth step (if hanning=True), a pre-channel averaging step (if width ≥2), SPW combination and re-gridding, but whereas the task cvel uses intermediate MSs for each of these steps, cvel2 performs all operations in memory without intermediate I/O steps. A conversion to LSRK of the ALMA M100 demonstration dataset (~12GB in size) using width=2 and hanning=True is up to twice as fast with cvel2 as compared to the cvel, depending, of course, on the underlying I/O system performance (x2 factor obtained in a system with 150MB/s I/O throughput).

Future developments
Currently mstransform is undergoing a major refactoring to offer a separated plugin interface for each transformation. This will facilitate the integration of the various transformations in other parts of CASA – like the pre-averaging feature already offered by flagdata in CASA 4.6 (see Fig. 2) – including modification of the MS as part of the processing. This will also facilitate the creation of custom transformation pipelines, without requiring data be written to disk.

Figure 2: Transform Visibility Iterator pipeline providing generalized pre-averaging in flagdata (CASA 4.6)

[click to enlarge]

References:

1. Visibility Weights and Their Calibration, G. Moellenbrock, CASA News, Issue 1
2. On-the-Fly Calibration in Plotms, P. Ford & G. Moellenbrock, CASA News, Issue 2
3. Parallel Processing in CASA, S. Castro, J. Gonzalez, J. Taylor, CASA New, Issue 1

2016 Calendar of Events