Observing > Prop Eval & Time Alloc > Documentation > SRP Review Instructions

SRP Review Instructions

by Dana Balser last modified Dec 03, 2018

1 NRAO Science Review Panels

Thank you for serving on an NRAO Science Review Panel (SRP). You have been invited to serve on an SRP because of your broad knowledge and expertise in critical areas of contemporary astronomy and astrophysics. The key function of the SRPs, of which there are eight, is to review proposals for use of NRAO’s telescopes based on their scientific merit. The SRPs therefore play a critical role in determining NRAO’s science program, reflecting the scientific community’s judgment of what investigations are important, timely, and influential.

This brief guide provides instructions for entering your individual and independent reviews of proposals using the Proposal Submission Tool (PST; Section 2). Review guidelines are provided in Section 3. NRAO policies on conflicts of interests and confidentiality are fully covered in Section 4.

Beginning in 18B SRP members no longer have to review all proposals.  Each proposal will be allocated one primary reviewer, one secondary reviewer, and two or more tertiary reviewers, all of which will provide an individual review (score + text).   The goal is that no SRP member will have to review more than 35 proposals.  The primary reviewer will draft the consensus review and lead the discussion during the telecon.  The secondary reviewer will back up the primary reviewer.   The tertiary review will provide a review (score + text).  All members who have read a proposal are welcome to participate in discussion of a proposal during the telecon.  The Chair will moderate the discussion and make any adjustment to the SRP score, weighting input from members appropriately taking into account their role (reviewer or otherwise) and any specialized knowledge they may bring. 

2 Entering Science Reviews Using the PST

I. Login through the NRAO portal at my.nrao.edu

You will enter the PST at the Dashboard, which allows you to perform a number of functions related to proposal preparation and review, access documentation, and modify your user profile. Please check that your user profile is up to date.

As an SRP member, seven tabs should be visible below the National Radio Astronomy Observatory banner:

  • Dashboard

  • Proposals

  • Reviews

  • Data Processing

  • Obs Prep

  • Helpdesk

  • Profile

II. Click on the Reviews tab. If the Reviews tab is not visible, please click the Helpdesk tab and submit a ticket immediately to the Proposal Review department.

III. Upon clicking the Reviews tab you will see a new page with the same banner and primary tabs across the top. It should have one secondary tab below the primary tabs labeled My Reviews. Brief instructions are provided to SRP members in the grey box. More detailed instructions are provided here.

The remainder of the page displays all of the proposals assigned to the Science Review Panel (SRP), of which you are a member.

  • Column 1 gives the PST proposal ID.

  • Column 2 gives the legacy ID.

  • Column 3 gives the proposal title.

  • Column 4 gives the proposal authors.

  • Column 5 indicates whether the SRP member is conflicted on a proposal.

  • Column 6 gives the reason for a given conflict.

The PST automatically captures one type of conflict:

  1. the SRP member is the PI or a co-I on a proposal under consideration by the SRP on which they serve.

If you believe any of the conflicts identified by the PST are in error, please promptly submit a Helpdesk ticket to the Proposal Review department.

Please review all other proposal titles and author lists for which the PST has not identified a conflict (unchecked box) and self-declare any conflicts. Examples of conflicts that the PST does not (and cannot) check include circumstances under which one or more of the following is true for the PI or a co-I on a given proposal:

  1. they are a spouse or other family member of the reviewer

  2. they are a close friend or active collaborator of the reviewer

  3. they are a former student or advisor of the reviewer

  4. any other reason that leads the reviewer to believe they cannot render a fair and impartial judgment on the scientific merit of the proposal

Consult Section 4 for full details. Once all conflicts have been identified click on Accept conflicts status in the upper right of the page. This takes you to the main review management page.

IV. The banner, primary, and secondary tabs on the review management page remain as before. Below the secondary My Reviews tab is a field and Search button on the left side that can be used to find proposals of interest. To the right are four buttons labeled Export Science Reviews, Import Science Reviews, Print, and Help. The Help button provides general help on the use of the User Portal for proposal preparation. The other three buttons are discussed below.

To the left of the page, below the Search field and button is an Options box that allows you to filter the proposals displayed on the page by telescope or by those that the proposers declare to be Joint Proposals. Additional filtering options may be added in future releases of the PST.

The Proposals List page contains 8 columns.

  • Column 1, labeled “All”, allows one to print all proposals that the reviewer is not conflicted on. To do so, click the “All” button. The check boxes of all proposals should automatically be checked. Then click the Print button to the upper right of the page. A window is opened that allows you to select which portions of the proposals to print: the cover sheet, the science justification, the student support application, and/or the student dissertation plan. After making your selections, click on “Print” in the window to print all proposals to a pdf file that can, in turn, be saved to a file on your computer or sent to your printer. One can also use the check boxes in Column 1 to manually check one or more proposals to print via the Print button.

  • Column 2 lists the PST proposal ID and also displays two icons. By clicking on the column’s heading, one can sort the proposals in numerical order. By clicking on the PST proposal ID in column 2, highlighted in blue, one can view the proposal. From that proposal’s page:

  1. One can use the Options column to the left to navigate to different sections of the proposal (e.g., general, authors, scientific justification, etc.),

  2. One can also enter the scientific reviews page by clicking on Reviews in the Options column.

  3. One can also directly view other (unconflicted) proposals by clicking on the ID number in the Options column. Clicking My Reviews at the top of the Options column returns you to the main proposal management page.

  4. A printer icon below the secondary My Reviews tab allows one to print the proposal in its entirety to a printer or to your computer’s hard disk.

  5. Clicking on the pen-and-paper icon in column 2 takes you to the science review page for that proposal (see below); the printer icon in column 2 allows you to send a pdf copy of the proposal to a printer or to your computer’s hard disk.

  • Column 3 shows the legacy ID.

  • Column 4 gives the proposal title.

  • Column 5 gives the name of the PI on the proposal.

  • Column 6 indicates the Review Type: None, Conflict, Primary, Secondary, or Tertiary.
  • Column 7 indicates whether the reviewer is conflicted on the proposal. NB: if the reviewer is conflicted, they should not be able to view, print, or review the proposal in question.

  • Column 8 indicates the status of the review, initially labeled View Proposal.

V. We now offer two ways to enter your proposal reviews. We describe each of these in Sections A and B.

A. One can use the PST to enter the reviews one by one. To enter a science review for a proposal on which you are not conflicted, you can click Enter Review in Column 7 or the pen-and-paper icon in column 2. A less direct path is to click on the proposal ID in column 2 and then click on Reviews under the Options bar from the proposal-viewing page.

The Scientific Reviews page summarizes the review for the proposal of interest.

  • Column 1 gives your name.

  • Column 2 shows the review comments.

  • Column 3 shows a numerical score.

  • Column 4 shows review status information.

Clicking on your name in column 1 allows you to enter your science review. Please type comments regarding the scientific merit of the proposal into the box in column 2. Please assign a numerical score of [0.1,9.9]. A low score is better than a high score! Guidelines regarding the science review are provided in Section 3.

NB: On rare occasions you may encounter a proposal for which you believe you are not qualified to review. If you sincerely believe that you are unable to perform a credible assessment of the scientific merit of the proposal, make no changes to the default review information for the proposal. With a default score of 0.0, that proposal will be excluded from the score normalization process. Also submit a Helpdesk ticket to the Proposal Review department, stating the proposal ID and that you believe you are unqualified to review the proposal. NRAO staff will then mark that review as Complete.

The fifth column on the review page allows you to Save or Cancel a review in progress. If you wish to defer completion of the review until later click on Save. If you wish to return the review to a pristine state or to its status following the previous Save, please click Cancel.

If the review is complete, please check the Completed box in column 4 and then click on Save in column 5. If you later find that you have made a mistake in checking the Completed box for a particular proposal, please submit a Helpdesk ticket to the Proposal Review department.

B. The second method to enter your reviews is to write them in a text file external to the PST, and then use the Import Science Reviews button to import them into the PST. This method is advantageous to those who wish to work offline. While preparation of the external text file is straightforward, it must nevertheless conform to some simple conventions. The easiest way to proceed, therefore, is to create an external reviews template by clicking the Export Science Reviews button, which produces a simple text file listing all proposals on which you are not conflicted. Here is an example of a template produced by clicking the Export Science Reviews button:

* ###
* Proposal ID%%%
* Review (text)%%%
* Score (0.1 - 9.9)%%%
* Completed Status (Yes/No)%%%
* ###



and so on.

This template can be saved to a file on your computer that you can edit without being logged into the PST.  Alternatively, one can create the external text file manually, making sure that it conforms to the simple conventions outlined above.

The first six lines are comments that start with a "*" symbol and list the four entry fields which are separated by "%%%".  The proposals are separated by "###".   This format allows you too use multiple lines for each field; the pound and percent symbols may be used in your text (assuming you do not have three consecutive symbols).   In the default file there are no comments to the "Review" field and thus there are two "%%%"  expressions back-to-back.

The first entry is the proposal ID (e.g., VLA/15B-345).  The second entry is the review.  Multiple lines may be used here.  The third entry is the numerical score. Please read the instructions regarding proposal scores under section V.A above. The last entry is a simple “Yes” or “No” to indicate whether you have completed the review. If you import a proposal with the entry set to “Yes”, it will not be possible to over-write or otherwise change the review of that particular proposal thereafter. If the entry is set to “No”, it will be possible to change or over-write.

Here is an example of an external text file after some editing has been done:

* ###
* Proposal ID%%%
* Review (text)%%%
* Score (0.1 - 9.9)%%%
* Completed Status (Yes/No)%%%
* ###
This is my first science review.
It is tremendously exciting.%%%
This is my second science review.
Still a lot of fun.%%%
This is my third science review.
I am now a seasoned veteran.%%%

To import the reviews, return to the PST reviews page and click on the Import Science Reviews button. At the top of the page, a text pane will appear. To the left of the text pane are three buttons: “Browse”, “Upload”, and “Cancel”. Clicking on the “Browse” button opens a browse window on your computer that allows you to identify the external text file with your science reviews. The directory path and name of the file then appear in the text pane. Click the “Upload” button to import the file into the PST. The PST will report the number of reviews successfully uploaded and whether any problems were encountered. Click “Cancel” if you decide not to import the reviews.

For those reviews that you indicated were incomplete (a “No” in the fourth field), you can either continue to revise them in the text file and upload again later; or you can complete them in the PST. It is also possible to re-export your reviews for further editing outside the PST, and then re-import them. Again, once you have indicated that you have completed a review with a “Yes” in the fourth entry field, no further changes to the review will be allowed by the PST. However, as noted in Section V.A, if you find that you have made a mistake by indicating that a particular proposal has been completed, please submit a Helpdesk ticket to the Proposal Review department.

VI. Once all (unconflicted) reviews are completed, either by manually indicating that this is the case proposal by proposal in the PST, or by setting the fourth field of all proposals to “Yes” in an external review field that is then imported, the PST will acknowledge the fact and thank you!


3 Guidelines for Science Reviews

The purpose of the proposal-selection process for NRAO telescopes is to prioritize and recommend the proposals that potentially are most valuable for the advancement of scientific knowledge. This does not necessarily mean recommending only those proposals that will provide sure results; it also includes a careful consideration of well-reasoned proposals that may be unconventional but provide opportunities for new discoveries. In the evaluation of proposals, we ask that reviewers think about how best to exploit the full capability of the unique scientific instruments that NRAO operates on behalf of the community. In this context, we ask the reviewers to take a constructive approach.

The proposals you will be reviewing were submitted in response to the Call for Proposals available at https://science.nrao.edu/observing/call-for-proposals

As noted in the Call, there are opportunities for joint observations with the Hubble Space Telescope, the Chandra X-ray Observatory, and the Swift Gamma-Ray Burst Mission. Requests for such joint observations should be justified both scientifically and technically; see the Call for specifics.

For VLA proposals, please be mindful of the configuration plans at


If any aspect of these guidelines is unclear to you, please click the Helpdesk tab and submit a ticket to the Proposal Review department.

Scientific Merit

We ask panelists to evaluate proposals, first and foremost, on the basis of their scientific merit. The Observatory seeks proposals that may have high scientific impact, not just "sure things" with modest impact. We note that with each call for proposals, several proposal types may be submitted:

  • Regular proposals are those that request less than 200 hrs total observing time on the GBT, VLA, and/or the VLBA/HSA.

  • Large proposals are those that request more than 200 hrs total observing time on the GBT, VLA, and/or VLBA/HSA.

  • Triggered proposals are those for pre-planned observations of transients whose event times are unknown a priori. Well-defined criteria are required to trigger an observation.

Therefore, in weighing the scientific merit of a given proposal, please be mindful of the differing resource requirements of each. Ultimately SRPs are responsible for producing a rank-ordered list of proposals, based on their scientific merit, for consideration by the TAC. The SRP must compare the relative merit of a diverse set of proposals and opinion will likely vary. Before finalizing its rankings, each SRP will meet via telecon to discuss the proposals and adjust the rankings to reflect consensus opinion. SRP activities during the telecon will be described in a separate set of guidelines.

Large Proposals

Recognizing that meeting large and influential scientific objectives often requires committing significant resources, the NRAO strongly encourages Large proposals, defined above. NRAO policy regarding large proposals may be found at


Proposals that request large commitments of resources must provide a strong scientific justification for doing so. The scientific merits of Large proposals should be weighed against those of Regular and Triggered proposals. They should also be placed within the wider scientific context: how will a particular Large proposal influence the wider scientific community? Is it a critical component to multi-wavelength studies of a problem of current interest or does it stand on its unique merits? Will the proposed observations be regarded as an important resource by the wider community?

A plan for data reduction and data release is a mandatory part of the justification for any Large proposal. The data management plan should be considered by the SRP when assigning a score.  The maximum proprietary period is one year, but the proposers are free to specify a shorter time interval. SRP reviewers are asked to assess the likelihood that useful scientific data products will be produced and made available to the wider community within a reasonable time period.

While the relevant SRP is expected to review Large proposals on the basis of criteria outlined herein, all TAC members are expected to be familiar with the Large proposals submitted each semester given the potential impact of large proposals on NRAO resources. Large proposals will be considered as a group by the TAC.

Other Review Criteria

While panelists should base their reviews above all on scientific merit, other factors may be considered.

Publication record. Due recognition should be given to teams who have an established publication record from past related proposals. In contrast, some observers may have considerable data that clearly has not been analyzed and published yet. Reviewers may take these positive and negative factors into account.

Possibility of acquiring more appropriate data. Reviewers also may wish to consider the possibility that other data may be more appropriate for reaching the scientific goals of the proposal. For instance, the relevant data might already exist in the VLBA archive, or in the Canadian Galactic Plane Survey.

Selection of resources. Are the appropriate resources being requested to meet science objectives? For the VLA and VLBA, is/are the right configuration(s) and/or telescopes requested? SRP members must judge whether the requested observation are an appropriate observational approach to the meeting the stated science objectives.

NB: Most proposals receive an independent technical review to assess their feasibility. The technical reviews will be available for consideration following the submission of independent scientific reviews, but in advance of the SRP telecon. The technical reviews may be considered as an additional factor by the SRP as part of their telecon discussion.

Resource requirements. Reviewers can and should take into account the amount of resources requested. For example, a particular proposal or scientific goal may be viewed as a valuable use of 4 hours of telescope time, but may not be as valuable if it requires 80 hours of time, considering the other proposals that it would displace.

Student status. We encourage the use of NRAO telescopes for student research, particularly for Ph.D. dissertations. In this case, it is most helpful if the proposal will say in some manner how the proposed observations will be used in the dissertation, and whether they are a sidelight or a main focus of the thesis. Reviewers are encouraged to comment on this topic, and may choose to support proposals more strongly if there appears to be a well-thought-out program of student research. If a dissertation plan is associated with a proposal, it can be accessed via a pdf view of the proposal.

Proposal length. The scientific justification must obey page limits. Regular and Triggered proposals are allowed a maximum of 4 one-sided pages with 11 point font (minimum) to present the scientific justification, including all figures, tables and references. Large proposals are allowed a maximum of 10 one-sided pages with 11 point font (minimum) to present the scientific justification, including all figures, tables and references. The cover information form is not counted as part of these page limits. The technical justification is included in the cover information form.

Numerical Scores

The NRAO proposal evaluation process uses a scale from 0.1 to 9.9, inclusive, with 0.1 being an outstanding proposal, and 9.9 being a very poor proposal. It is helpful if reviewers use the available dynamic range rather than giving every proposal a score between 2.9 and 3.1! Reviewers' score distributions each are shifted to have a common mean (equal to 5) and linearly scaled to have a standard deviation of 2; the scores, so normalized, are later averaged for each proposal. This process works best if there is adequate dispersion in the scores prior to normalization.

As noted in Section 2.V, if you sincerely believe that you are unable to assess the scientific merit of a proposal, please do not modify the default score of that proposal; i.e., leave it as 0.0. This will ensure that it is excluded from the score normalization process and from the score average.


Comments from individual reviewers should be brief and to the point, observing the norms of professional courtesy and providing constructive feedback as appropriate. Please be alert to the role of students in a given proposal when providing feedback.

Comments from individual reviewers have two important roles.

(1) It is not uncommon for different reviewers to come to different conclusions about a given proposal. In resolving these differences, it is important to the function of the SRP for the individual reviewers to comment about what led them to their score. The comments from individual reviewers thus serve to guide the SRP to its consensus view about a proposal.

(2) Comments from individual reviewers will not be fully transmitted to the proposers. However, portions of review comments will carry over into the SRP consensus comments, which will be fully transmitted to the proposers and should help the proposers to understand the disposition of their proposals. Specific comments can be extremely helpful here, especially for proposals that are unsuccessful and need to be improved and resubmitted. The comments from individual reviewers thus serve as a source of specific points to appear in the SRP consensus comments.

Specific guidelines to the SRPs for developing consensus comments to the proposers prior to and during the SRP telecon are provided in a separate document.


4 NRAO Conflicts of Interest and Confidentiality Polices

Peer-reviewed proposals to use NRAO facilities for scientific investigations are the bedrock on which observing time is allocated to the community. The NRAO proposal review and time allocation process is described in detail in the Guide to NRAO Proposal Evaluation and Time Allocation. The process relies on the scientific community to evaluate proposals based on their scientific merit and to make recommendations regarding time allocation. The process must be robust and free of real or perceived conflicts of interest and must maintain confidentiality. This document details AUI/NRAO conflicts of interest and confidentiality policies associated with the proposal evaluation and time allocation process.


Conflicts of Interest

For the purposes of the NRAO proposal evaluation and time allocation process, an individual is considered to have a potential conflict of interest if one or more of the following is true:

  1. They are a PI or a co-I on a proposal under consideration.

  2. They are affiliated with the same department as the PI or a co-I on a proposal under consideration.

  3. They are a spouse, partner or other family member of a proposal author.

  4. They are a current or recent collaborator of a proposal author.

  5. They are a former student or advisor of a proposal author.

  6. They have any other reason to believe they cannot render a fair and impartial judgment on the scientific merit of the proposal.

In order to ensure that all proposals are treated fairly and without bias, such potential conflicts must be identified and declared. The response to a conflict depends on the role of the participant. The following participants in the NRAO proposal evaluation and time allocation process must declare conflicts of interest:

  • Members of Science Review Panels (SRPs)

  • NRAO staff responsible for technical reviews of proposals

  • Members of the Time Allocation Committee (TAC)

  • NRAO staff that support SRP and TAC meetings

  • NRAO staff that participate in the Directors Review

Members of Science Review Panels. Science reviewers are primarily external to NRAO and are subject to all conflict criteria enumerated above. The NRAO automatically captures conflicts of type (1) with the software tools used to manage the proposal review process. Members of SRPs are expected to otherwise declare those proposals on which they believe themselves to be conflicted. Conflicts of type (4) and (5) are in force if a given reviewer has collaborated with the proposal author, or has been a student of the author, or the thesis advisor of the author, within the last five years. Conflicted panelists will not be allowed to access or review the proposal(s) in question. During the course of SRP discussions, conflicted panelists will not hear or participate in discussion of the relevant proposal(s).

Technical Reviewers. Technical reviews are performed by NRAO staff. Technical reviewers are subject to all conflict types except for those of type (2). In particular, conflicts of type (4) and (5) are, as for science reviewers, only in force if the reviewer has collaborated, been a student, or the thesis advisor of a proposal author within the last five years. Proposals on which a technical reviewer is conflicted are assigned to another reviewer.

Members of the Time Allocation Committee. Each TAC member is the Chair of one of the eight SRPs. As such, they are subject to the same conflicts of interest policy as other SRP members during the science review of proposals. During the face-to-face TAC meeting, rigorous enforcement of all conflict types is impractical. Nevertheless, conflicts of types (1) and (6) are always enforced. TAC members may otherwise collectively identify proposals on which a member is believed to be conflicted. A conflicted TAC member must remove himself or herself from the room when the proposal in question is discussed and recommendations made.

NRAO Support Staff. NRAO staff support the SRP and TAC meetings. They are subject to all conflict types with the exception of type (2). If an NRAO staff member is conflicted during the course of an SRP discussion, they are prevented from hearing discussion of the proposal in question. NRAO staff members do not, in any event, participate in the discussion of the scientific merits of proposals under evaluation. If an NRAO staff member is conflicted during the course of a TAC meeting, they are subject to the same conflict criteria as TAC members.

Directors Review. The Directors Review is intended to review the time allocation process each semester to ensure that all relevant NRAO policies and procedures have been followed, resulting in a fair and transparent allocation of telescope time. NRAO staff members participating in the Directors review are subject to all conflict types except for type (2). Any participant of the Directors review that has a potential conflict must withdraw from discussion of the proposal(s) in question. If such conflicts involve the Director, the Director, too, must withdraw from discussion of any proposal(s) in question. To ensure that appropriate consideration is given to Observatory programmatics and priorities, the Head of Science Support and Research (SSR) will lead the Directors Review in lieu of the Director in such cases. If the Directors Review results in changes to the science program recommended by the TAC, such changes must be recorded. An account of the Directors Review, a description of the changes made to the science program, and the justification for such changes will be prepared by the SSR Head and forwarded to the AUI President for review.



Both NRAO staff and external members of the scientific community participate in the NRAO proposal evaluation and time allocation process. They will regularly be exposed to confidential information and proprietary data and ideas. The information, data, and ideas must be held in confidence and not disclosed to others. However, several exceptions apply according to the role of the individuals in question.

  • Members of a given SRP may disclose and discuss the contents of any proposal assigned to them with one another, consistent with the conflicts of interest policy.

  • Members of the TAC may disclose and discuss the contents of any proposal under consideration with each other, consistent with conflicts of interest policy.

  • NRAO staff serving as technical reviewers may need to consult with each other to ensure a fair and complete assessment of technical elements of the proposal. Hence, technical details may be disclosed and discussed internally by relevant NRAO staff as needed.

  • NRAO staff members that participate in the Directors Review may disclose and discuss the contents of any proposal under consideration with each other, consistent with the conflicts of interest policy.

The following individuals must comply with NRAO policies on confidentiality:

  • Members of Science Review Panels

  • Members of the Time Allocation Committee

  • NRAO staff responsible for:

    • technical reviews of proposals

    • proposal management software tools and associated data bases

    • providing SRP and TAC support

    • scheduling approved proposals on NRAO telescopes

  • NRAO staff that participate in the Directors Review of TAC recommendations

Proposals printed out for technical or science review must be disposed of by SRP and TAC members in a manner consistent with NRAO policy on confidentiality. Proposals and associated information (source conflicts, proposal rankings, pressure plots, etc.) printed out and made available to TAC members during NRAO TAC meetings must remain at the NRAO. NRAO staff will dispose of these materials in a manner consistent with NRAO policy on confidentiality.



All participants in the NRAO proposal evaluation and time allocation will be asked to review the policies detailed herein and asked to confirm that they understand and agree to comply with them.