Facilities > VLA > Documentation > Manuals > Computing Resources > Data Storage and Retrieval

Data Storage and Retrieval

by Gustaaf Van Moorsel last modified May 19, 2017 by K Scott Rowe

Data Storage

Observer accounts reside on the Lustre in /lustre/aoc/observers/<user>.   This area is also where you should store all data products and scratch files.   Lustre is a shared resource amongst all staff and observers, we ask that everyone keep their usage as far below the 5TByte limit as possible.

Please see the following special instructions for retrieving data directly from the archive into your home area.  Put note about NGAS account file ownership

Data Retrieval

The NMASC supports the following methods for securely transporting data to remote facilities and has plans to support XSEDE's Globus Connect platform.  For the following examples, <user> is your nm-#### account name, your data area would be /lustre/aoc/observers/<user>.

SFTP

SFTP is an encrypted ftp protocol.  You can access your home area from remote facilities via: sftp sftp.aoc.nrao.edu and login with your NMASC account name.   From there sftp behaves much like any ftp client.

The example below would connect user nm-1234 to the sftp server.  The current directory would be nm-1234's home lustre area.

sftp nm-1234@sftp.aoc.nrao.edu

SCP

SCP is an encrypted copy that can transfer between remote hosts.  The format is scp user@remote machine:/<remote path> <local path>. From your  machine you would run scp <user>@ssh.aoc.nrao.edu:/lustre/aoc/observers/<user>/<path to files> <local path>

The example below would copy all files ('*') in nm-1234's data directory to the current directory ('.') on your machine.

scp nm-1234@ssh.aoc.nrao.edu:/lustre/aoc/observers/nm-1234/data/* .

LFTP

LFTP is a more sophisticated version of the classic ftp protocol which, among other things, uses multiple channels to speed performance.

lftp -u <user> sftp://sftp.aoc.nrao.edu

RSYNC

RSYNC is a versatile file-copying tool that only copies necessary files, that is the ones that are missing in your local copy.  This is useful if, for example, you have deleted some files from your local copy of your JobID and to copy just those missing files.

The example below would copy all the files in nm-1234's data directory to a local directory.  Without the trailing '/' rsync would copy the directory and its contents , with a trailing '/' it copies only the contents of the directory.  Adding '--delete' to the arguments list will keep the two areas exactly in sync by removing files on your system's copy if they've been removed from the NMASC copy.

rsync -vaz nm-1234@ssh.aoc.nrao.edu:/lustre/aoc/observers/nm-1234/data/ .

Browser Access

Go to https://archive.nrao.edu/observers/<user>. You will need to login using the observer user name and my.nrao.edu password. This will allow you to navigate through your filesystem on a browser to view and download files.

GlobusOnline

NMASC will be adding a Globus Connect portal in coming months.