Facilities > VLA > Documentation > Manuals > Computing Resources > Resource Limits and Data Retention

Resource Limits and Data Retention

by Gustaaf Van Moorsel last modified Jun 15, 2017 by James Robnett

Allocation and limits on processing resources applies to both NRAO staff and observers.

Quota Limits

Beginning June 22nd, 2017 quotas will be enforced via native Lustre filesystem quotas based on user's default group.

Users are limited to 5TB of space in their /lustre/aoc/observers/<username> data area on the shared Lustre filesystem.  User data in large project areas is accounted for via separate project quotas and does not count toward the users quota. 

You can view your current usage with lfs quota -g <group> /lustre/aoc where <group> is your account name; observer account names and their default group are the same string.

The following shows the quota for the account nm-4386.

# lfs quota -g nm-4386 /lustre/aoc
Disk quotas for group nm-4386 (gid 24386):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
    /lustre/aoc 5455723856* 5368709120 6442450944 6d22h49m58s  527602       0       0       -

To accommodate brief processing spikes Lustre incorporates a one week grace period where usage can increase by 20% to 6TB. Attempts to write above 6TB or failure to reduce back below 5TB after the grace period will trigger an I/O error. 

The above example shows the account is over the 5TB quota (trailing "*") but below the 6TB hard limit with 6 days and 22 hours remaining on the one week grace period.

Cluster Resource limits

For interactive sessions users are limited to one compute node. Interactive session nodes are assigned for 1 to 14 days and can only be accessed by the reserving account.

During periods of increased pressure, users may be asked to release idle nodes to allow broader community access.

For batch requests users are asked to limit in flight batch jobs to one node's worth of jobs.

Requests for increased access duration, storage space and compute nodes should be submitted as a ticket to https://help.nrao.edu (Data archive and Download department) and will be reviewed by designated NMASC or NAASC staff on a case by case basis.

Data Retention

External accounts, along with any data products, will be removed two weeks from the completion of a processing request.    For interactive sessions, the account and data products will be removed two weeks from the end of the session.  You will receive an email warning prior to account deletion.  In the event of multiple processing requests, the account expiration date will be triggered by the last request to complete.

Large Proposals

Observers associated with large proposals (greater than 200 hours) who plan to use NRAO computing resources are encouraged to request the creation of a project area on the Lustre filesystems and a Unix group for shared data access among proposal members.

Project area size limits will be negotiated with NRAO at the time of the request to match project data rates and imaging plans but will typically be 10TB in size.  Quotas for projects are enforced via the project group.