Observing > NRAO Archive Tool Enhancements

NRAO Archive Tool Enhancements

On May 2, 2022, the NRAO transitioned to the new Archive Access Tool (AAT) for the VLA, VLBA, Legacy VLA, and GBT data. This new tool also includes ALMA data in its search returns. While the capabilities of this new archive tool go above and beyond what was possible with the legacy archive tool, we recognize that the transition for our users has not been entirely smooth. The NRAO is committed to maintaining and expanding the new archive interface and its capabilities to improve the experience of our users.

We value feedback from our user community on improving the NRAO archive tool. If a user has a feature request, or feedback on how a current feature/implementation could work better, they can file a helpdesk ticket at help.nrao.edu via the department: ‘Archive Access Tool Feedback.’

In the most recent release of the AAT (v 4.3.0.0/1/2, 12/23/2023), the NRAO implemented the following features:

  • Scanlists have been refactored to include complete spectral window information
  • New VLBA/GMVA data is ingested into the AAT directly and no longer involves the legacy archive
  • Automated metadata reharvesting to populate scanlist for VLA and VLBA.
  • QA Notes now visibile for VLA calibrations.

The next AAT release (v 4.3.0.3, approx. 04/2024) will include:

  • Updated more complete frequency information in the scripted archive data returns.
  • Number of antennas in AAT and scripted search returns
  • Corrections to the display of image products.

In AAT release (v 4.2.3, June 2023), the NRAO implemented the following features:

  • Fixes for ALMA ASDM downloads
  • Addition of configuration to TAP (script interface) returns
  • Updates to support inclusion of self-calibrated data products for AUDI
  • Addition of wget2 download commands to user e-mails

 

In AAT release v4.2.2 (March 2022), the NRAO addressed the features:

  • Improve availability of ALMA data for restoration and reimaging
  • Fix intermittent issues with the TAP server and implement speed improvements
  • Removed file number limit
  • Batch download of calibration tarballs
  • Reorganized download directory structure for non-processing downloads

 

In AAT release v4.2.1 (December 2022), the NRAO addressed the features:

  • Position and radius searches should now return more accurate results
  • Fixes to 'Select All' functionality
  • Faster VLBA indexing
  • Handling of VLBA data with large numbers of correlations

 

In AAT release v4.2.0 (September 2022), the NRAO addressed the features:

  • Scriptability of the archive query interface via VO-compliant Table Access Protocol (TAP); see documentation!
  • Inclusion of wget commands to retrieve data in notification e-mails
  • Sortable scanlists


In AAT release 4.1.1 (June 2022), the NRAO addressed the features:

  • Data delivery for non-staff users in New Mexico (i.e., nm- visitor account holders) to their lustre disk areas.
  • Improved data delivery speed.
  • Organization of individual VLBA observations into their observation segments.


For the upcoming releases, the NRAO has prioritized the following areas for development:

  • Scanlist improvements (timestamps, total integration time, spectral setups) (released in 4.3.0.2)
  • Download directory improvements (partially completed in 4.2.2; additional improvements in 4.3.1; ests. Summer 2024)
  • On-demand imaging of VLA EBs (expected 4.3.2 (est. Fall 2024)
  • Search improvements.
  • Search based on integration time.
  • Scriptable downloads

VLA-specific

  • Metadata display improvements (i.e., scan list, sources, frequency setup(s), total integration times) (partially implemented in released in 4.3.0.2, needs to be brought into main interface and not just scanlists; est. Fall 2024)

VLBA-specific

  • Completion of VLBA data collection: Fixing corrupted Mark IV files, and adding a few missing GMVA, HSA, and Global VLBI data.
  • Downloading large numbers (>500) of correlation files for VLBA projects in bulk via the 'Select All' option. (Implemented in 4.2.2)
  • Fixing the missing or incomplete metadata of some VLBA+EVLA (i.e. +Y1, +Y27) projects, without which download is impossible. For the time being, please contact our helpdesk to gain access if you are PI or CoI of the affected projects.

 

Known Issues affecting data in the archive and archive use (which we are working on to address):

  • All data deliveries come in a nested directory structure where your file, MS, or SDM is within directories that have the same name as your file, MS, or SDM. The actual files or MS/SDMs of interest are two or three levels inside this directory structure. (This is still true for processing downloads [Basic MS and calibrated MS], but raw data and FITS downloads are reorganized now).
  • Frequency-dependent field of view searches are not currently supported.
  • Insufficient metadata are available for some EVLA and some VLBA data with regard to their frequency setup, sources, and times on source; all Historical VLA data are impacted. The EVLA and VLBA data are currently having their metadata reharvested display their frequency metadata, while Historical VLA are likely to be updated by Fall 2024.

 

Staff | Policies | Diversity