1 NRAO Science Review Panels

Thank you for serving as Chair of an NRAO Science Review Panel (SRP). You have been invited to serve in this capacity because of your broad knowledge and expertise in critical areas of contemporary astronomy and astrophysics. The key function of the SRPs is to review proposals for use of NRAO’s telescopes based on their scientific merit. The SRPs therefore play a critical role in determining NRAO’s science program, reflecting the scientific community’s judgment of what investigations are important, timely, and influential.

This brief guide provides instructions for assigning reviewers to each proposal using the Proposal Submission Tool (PST; Section 2) and entering your individual and independent reviews of proposals (Section 3). Review guidelines are provided in Section 4. NRAO policies on conflicts of interests and confidentiality are fully covered here.

Beginning in 18B SRP members no longer have to review all proposals.  Each proposal will be allocated one primary reviewer, one secondary reviewer, and two or more tertiary reviewers, all of which will provide an individual review (score + text).   The goal is that no SRP member will have to review more than 35 proposals.  The primary reviewer will draft the consensus review and lead the discussion during the telecon.  The secondary reviewer will back up the primary reviewer.   The tertiary review will provide a review (score + text).  All members who have read a proposal are welcome to participate in discussion of a proposal during the telecon.  The Chair will moderate the discussion and make any adjustment to the SRP score, weighting input from members appropriately taking into account their role (reviewer or otherwise) and any specialized knowledge they may bring. 

2 Assigning Reviewers Using the PST

As SRP Chair, you will need to assign reviewers to each proposal. It is important that you instruct your panel to declare their conflicts soon so that you can assign reviewers.  The review process cannot begin until you have assigned reviewers.  To see which panel members have completed their conflict of interest status click on the Reviews Summary tab.  In the Options column to the left click Show Summary to display the table.  An (N) next to a name indicates a panel member needs to complete their conflict status review.  Once all panel members have reviewed their conflict status you can start assigning reviews. You may find it useful poll your panel to assess how comfortable they are with reviewing each proposals (e.g., scale of 1-5 with 1 being easy and 5 being difficult).

The summary table is a matrix with the proposal ID on the left and the science reviewers on the top.  The text in each cell corresponds to the Review Type, whereas the color corresponds to the Review State.  The Review Types are:

• None: reviewer is expected to only read the proposal and provide feedback at the telecon.
• Conflict: the reviewer is conflicted and therefore cannot review or view the proposal.
• Primary: the reviewer is expected to review the proposal and lead the discussion at the telecon.
• Secondary: the reviewer is expected to review the proposal and back up the Primary reviewer at the telecon.
• Tertiary: the reviewer is expected to review the proposal and provide feedback at the telecon.

The Review States are:

• White: no action on the review.
• Gray: reviewer has edited the review (it is in progress).
• Green: reviewer has completed the review.
• Red: reviewer is conflicted.
• Blue: the review is closed.

Please assign a minimum of four reviewers per proposal; except for Large proposals where there should be at least five reviewers per proposal if possible.  The goal is for each SRP member to have no more than about 35 proposals to review; the SRP chair should have no more than half the load of an SRP member.  Unless there is a conflict the default Review Type is None.  There are two ways to assign reviewers. 

A. Click on a cell and select either None, Primary, Secondary, or Tertiary.  Then click Assign Type. 

B. Click on the Export Reviews Types button.  This will produce a csv formatted file that lists: proposal id, title, P.I., the requested time in hours, is it a thesis proposal, is it a joint proposal, is it a multi-configuration proposal (VLA only), and the review typel.  The reviewer is identified by their initials and a PST ID in parenthesis.  (See an example in the table below.)  Edit the Review Type either in an editor or spreadsheet and then import back into the PST using the Import Review Types button.

3 Entering Science Reviews Using the PST

I. Login through the NRAO portal at my.nrao.edu

You will enter the PST at the Dashboard, which allows you to perform a number of functions related to proposal preparation and review, access documentation, and modify your user profile. Please check that your user profile is up to date.

As an SRP member, seven tabs should be visible below the National Radio Astronomy Observatory banner:

• Dashboard

• Proposals

• Reviews

• Data Processing

• Obs Prep

• Helpdesk

• Profile

II. Click on the Reviews tab. If the Reviews tab is not visible, please click the Helpdesk tab and submit a ticket immediately to the Proposal Review department.

III. Upon clicking the Reviews tab you will see a new page that the same banner and primary tabs across the top. It should have two secondary tabs below the primary tabs labeled My Reviews and Reviews Summary. Brief instructions are provided to SRP members in the grey box. More detailed instructions are provided here.

The remainder of the page displays all of the proposals assigned to the Science Review Panel (SRP), of which you are a member and Chair.

• Column 1 gives the PST proposal ID.

• Column 2 gives the legacy ID.

• Column 3 gives the proposal title.

• Column 4 gives the proposal authors.

• Column 5 indicates whether you are conflicted on a proposal.

• Column 6 gives the reason for a given conflict.

The PST automatically captures one type of conflict:

1. the SRP member is the PI or a co-I on a proposal under consideration by the SRP on which they serve.

If you believe any of the conflicts identified by the PST are in error, please promptly submit a Helpdesk ticket to the Proposal Review department.

Please review all other proposal titles and author lists for which the PST has not identified a conflict (unchecked box) and self-declare any conflicts. Examples of conflicts that the PST does not (and cannot) check include circumstances under which one or more of the following is true for the PI or a co-I on a given proposal:

1. they are a spouse or other family member of the reviewer

2. they are a close friend or active collaborator of the reviewer

3. they are a former student or advisor of the reviewer

4. any other reason that leads the reviewer to believe they cannot render a fair and impartial judgment on the scientific merit of the proposal

Consult here for full details. Once all conflicts have been identified click on Accept conflicts status in the upper right of the page. This takes you to the main review management page.

IV. The banner, primary, and secondary tabs on the review management page remain as before. Below the secondary My Reviews tab is a field and Search button on the left side that can be used to find proposals of interest. To the right are four buttons labeled Export Science Reviews, Import Science Reviews, Complete All Reviews, Print, and Help. The Help button provides general help on the use of the User Portal for proposal preparation. The other three buttons are discussed below.

To the left of the page, below the Search field and button is an Options box that allows you to filter the proposals displayed on the page by telescope or by those that the proposers declare to be Joint Proposals. Additional filtering options may be added in future releases of the PST.

The Proposals List page contains 9 columns.

• Column 1, labeled “All”, allows one to print all proposals that the reviewer is not conflicted on. To do so, click the “All” button. The check boxes of all proposals should automatically be checked. Then click the Print button to the upper right of the page. A window is opened that allows you to select which portions of the proposals to print: the cover sheet, the science justification, the student support application, and/or the student dissertation plan. After making your selections, click on “Print” in the window to print all proposals to a pdf file that can, in turn, be saved to a file on your computer or sent to your printer. One can also use the check boxes in Column 1 to manually check one or more proposals to print via the Print button.

• Column 2 lists the PST proposal ID as a link and also displays two icons. By clicking on the PST proposal ID in column 2 you are able to view the proposal. From that proposal’s page:

1. One can use the Options column to the left to navigate to different sections of the proposal (e.g., general, authors, scientific justification, etc.),

2. One can also enter the scientific reviews page by clicking on Reviews in the Options column.

3. One can also directly view other (unconflicted) proposals by clicking on the ID number in the Options column. Clicking My Reviews at the top of the Options column returns you to the main proposal management page.

4. A printer icon below the secondary My Reviews tab allows one to print the proposal in its entirety to a printer or to your computer’s hard disk.

5. Clicking on the pen-and-paper icon in column 2 takes you to the science review page for that proposal (see below); the printer icon in column 2 allows you to send a pdf copy of the proposal to a printer or to your computer’s hard disk.

• Column 3 shows the legacy ID.

• Column 4 gives the proposal title.

• Column 5 gives the name of the PI on the proposal.

• Column 6 indicates the Review Type: None, Conflict, Primary, Secondary, or Tertiary.

• Column 7 indicates whether a panel member is conflicted on the proposal. NB: if the reviewer is conflicted, they should not be able to view, print, or review the proposal in question.

• Column 8 indicates whether the Chair is conflicted on the proposal.

• Column 9 indicates the status of the review, initially labeled View Proposal.

V. The SRP Chair only enters reviews for those proposals with a Review Type of Pimary, Secondary, or Tertiary.  If the Review Type is None then you cannot enter a review but are expected to read the proposal and provide feedback during the SRP telecon.  If the Review Type is Conflict then you will not be able to access the proposal.   We offer two ways to enter your proposal reviews. We describe each of these in Sections A and B.

A. One can use the PST to enter the reviews one by one. To enter a science review for a proposal on which you are not conflicted, you can click Enter Review in Column 9 or the pen-and-paper icon in column 2. A less direct path is to click on the proposal ID in column 2 and then click on Reviews under the Options bar from the proposal-viewing page.

The Scientific Reviews page summarizes the review for the proposal of interest.

• Column 1 gives your name.

• Column 2 shows the review comments (initially blank).

• Column 3 shows a numerical score (initially 0.0).

• Column 4 shows review status information (initially blank).

Clicking on your name in column 1 allows you to enter your science review. Please type comments regarding the scientific merit of the proposal into the box in column 2. Please assign a numerical score of [0.1,9.9]. A low score is better than a high score! Guidelines regarding the science review are provided in Section 3.

NB: On rare occasions you may encounter a proposal for which you believe you are not qualified to review. If you sincerely believe that you are unable to perform a credible assessment of the scientific merit of the proposal, make no changes to the default review information for the proposal. With a default score of 0.0, that proposal will be excluded from the score normalization process. Also submit a Helpdesk ticket to the Proposal Review department, stating the proposal ID and that you believe you are unqualified to review the proposal. NRAO staff will then mark that review as Closed.

The fifth column on the review page allows you to Save or Cancel a review in progress. If you wish to defer completion of the review until later click on Save. If you wish to return the review to a pristine state or to its status following the previous Save, please click Cancel.

If the review is complete, please check the Completed box in column 4 and then click on Save in column 5. If you later find that you have made a mistake in checking the Completed box for a particular proposal, please submit a Helpdesk ticket to the Proposal Review department.

B. The second method to enter your reviews is to write them in a text file external to the PST, and then use the Import Science Reviews button to import them into the PST. This method is advantageous to those who wish to work offline. While preparation of the external text file is straightforward, it must nevertheless conform to some simple conventions. The easiest way to proceed, therefore, is to create an external reviews template by clicking the Export Science Reviews button, which produces a simple text file listing all proposals on which you are not conflicted. Here is an example of a template produced by clicking the Export Science Reviews button:

* ###
* Proposal ID%%%
* Review (text)%%%
* Score (0.1 - 9.9)%%%
* Completed Status (Yes/No)%%%
* ###
VLA/15B-345%%%
%%%
0.0%%%
No%%%
###
GBT/15B-012%%%
%%%
0.0%%%
No%%%
###
VLBA/15B-142%%%
%%%
0.0%%%
No%%%
###

.

.

and so on.

This template can be saved to a file on your computer that you can edit without being logged into the PST.  Alternatively, one can create the external text file manually, making sure that it conforms to the simple conventions outlined above.

The first six lines are comments that start with a "*" symbol and list the four entry fields which are separated by "%%%".  The proposals are separated by "###".   This format allows you too use multiple lines for each field; the pound and percent symbols may be used in your text (assuming you do not have three consecutive symbols).   In the default file there are no comments to the "Review" field and thus there are two "%%%"  expressions back-to-back.

The first entry is the proposal ID (e.g., VLA/15B-345).  The second entry is the review.  Multiple lines may be used here.  The third entry is the numerical score. Please read the instructions regarding proposal scores under section V.A above. The last entry is a simple “Yes” or “No” to indicate whether you have completed the review. If you import a proposal with the entry set to “Yes”, it will not be possible to over-write or otherwise change the review of that particular proposal thereafter. If the entry is set to “No”, it will be possible to change or over-write.

Here is an example of an external text file after some editing has been done:

* ###
* Proposal ID%%%
* Review (text)%%%
* Score (0.1 - 9.9)%%%
* Completed Status (Yes/No)%%%
* ###
VLA/15B-345%%%
This is my first science review.
It is tremendously exciting.%%%
5.5%%%
Yes%%%
###
GBT/15B-012%%%
This is my second science review.
Still a lot of fun.%%%
1.2%%%
No%%%
###
VLBA/15B-142%%%
This is my third science review.
I am now a seasoned veteran.%%%
7.8%%%
Yes%%%
###

To import the reviews, return to the PST reviews page and click on the Import Science Reviews button. At the top of the page, a text pane will appear. To the left of the text pane are three buttons: “Browse”, “Upload”, and “Cancel”. Clicking on the “Browse” button opens a browse window on your computer that allows you to identify the external text file with your science reviews. The directory path and name of the file then appear in the text pane. Click the “Upload” button to import the file into the PST. The PST will report the number of reviews successfully uploaded and whether any problems were encountered. Click “Cancel” if you decide not to import the reviews.

For those reviews that you indicated were incomplete (a “No” in the fourth field), you can either continue to revise them in the text file and upload again later; or you can complete them in the PST. It is also possible to re-export your reviews for further editing outside the PST, and then re-import them. Again, once you have indicated that you have completed a review with a “Yes” in the fourth entry field, no further changes to the review will be allowed by the PST. However, as noted in Section V.A, if you find that you have made a mistake by indicating that a particular proposal has been completed, please submit a Helpdesk ticket to the Proposal Review department.

VI. Once all reviews are completed, either by manually indicating that this is the case proposal by proposal in the PST, or by setting the fourth field of all proposals to “Yes” in an external review field that is then imported, the PST will acknowledge the fact and thank you!  You can also click the Complete All Reviews button to complete all reviews.

4 Guidelines for Science Reviews

The purpose of the proposal-selection process for NRAO telescopes is to prioritize and recommend the proposals that potentially are most valuable for the advancement of scientific knowledge. This does not necessarily mean recommending only those proposals that will provide sure results; it also includes a careful consideration of well-reasoned proposals that may be unconventional but provide opportunities for new discoveries. In the evaluation of proposals, we ask that reviewers think about how best to exploit the full capability of the unique scientific instruments that NRAO operates on behalf of the community. In this context, we ask the reviewers to take a constructive approach.

The proposals you will be reviewing were submitted in response to the Call for Proposals available at https://science.nrao.edu/observing/call-for-proposals

As noted in the Call, there are new opportunities for joint observations with the Hubble Space Telescope, the Chandra X-ray Observatory, the Swift Gamma-Ray Burst Mission and XMM-Newton. Requests for such joint observations should be justified both scientifically and technically; see the Call for specifics.

For VLA proposals, please be mindful of the configuration plans at

https://science.nrao.edu/facilities/evla/proposing/configpropdeadlines

If any aspect of these guidelines is unclear to you, please click the Helpdesk tab and submit a ticket to the Proposal Review department.

Scientific Merit

We ask panelists to evaluate proposals, first and foremost, on the basis of their scientific merit. The Observatory seeks proposals that may have high scientific impact, not just "sure things" with modest impact. We note that with each call for proposals, several proposal types may be submitted:

• Regular proposals are those that request less than 200 hrs total observing time on the GBT, EVLA, and/or the VLBA/HSA.

• Large proposals are those that request more than 200 hrs total observing time on the GBT, EBLA, and/or VLBA/HSA.  (N.B., starting in 20A, the definition of Large proposals has changed for the GBT.  See the GBO call for proposals for details.)

• Triggered proposals are those for pre-planned observations of transients whose event times are unknown a priori. Well-defined criteria are required to trigger an observation.

Therefore, in weighing the scientific merit of a given proposal, please be mindful of the differing resource requirements of each. Ultimately SRPs are responsible for producing a rank-ordered list of proposals, based on their scientific merit, for consideration by the TAC. The SRP must compare the relative merit of a diverse set of proposals and opinion will likely vary. Before finalizing its rankings, each SRP will meet via telecon to discuss the proposals and adjust the rankings to reflect consensus opinion. SRP activities during the telecon will be described in a separate set of guidelines.

Large Proposals

Recognizing that meeting large and influential scientific objectives often requires committing significant resources, the NRAO strongly encourages Large proposals, defined above. NRAO policy regarding large proposals may be found at:

https://science.nrao.edu/observing/proposal-types/largeproppolicy

Proposals that request large commitments of resources must provide a strong scientific justification for doing so. The scientific merits of Large proposals should be weighed against those of Regular and Triggered proposals. They should also be placed within the wider scientific context: how will a particular Large proposal influence the wider scientific community? Is it a critical component to multi-wavelength studies of a problem of current interest or does it stand on its unique merits? Will the proposed observations be regarded as an important resource by the wider community?

A plan for data reduction and data release is a mandatory part of the justification for any Large proposal. The data management plan should be considered by the SRP when assigning a score. The maximum proprietary period is one year, but the proposers are free to specify a shorter time interval. SRP reviewers are asked to assess the likelihood that useful scientific data products will be produced and made available to the wider community within a reasonable time period.

While the relevant SRP is expected to review Large proposals on the basis of criteria outlined herein, all TAC members are expected to be familiar with the Large proposals submitted each semester given the potential impact of large proposals on NRAO resources. Large proposals will be considered as a group by the TAC.

Other Review Criteria

While panelists should base their reviews above all on scientific merit, other factors may be considered.

Publication record. Due recognition should be given to teams who have an established publication record from past related proposals. In contrast, some observers may have considerable data that clearly has not been analyzed and published yet. Reviewers may take these positive and negative factors into account.

 Possibility of acquiring more appropriate data. Reviewers also may wish to consider the possibility that other data may be more appropriate for reaching the scientific goals of the proposal. For instance, the relevant data might already exist in the VLBA archive, or in the Canadian Galactic Plane Survey.

Selection of resources. Are the appropriate resources being requested to meet science objectives? For the VLA and VLBA, is/are the right configuration(s) and/or telescopes requested? SRP members must judge whether the requested observation are an appropriate observational approach to the meeting the stated science objectives.

NB: Most proposals receive an independent technical review to assess their feasibility. The technical reviews will be available for consideration following the submission of independent scientific reviews, but in advance of the SRP telecon. The technical reviews may be considered as an additional factor by the SRP as part of their discussion.

Resource requirements. Reviewers can and should take into account the amount of resources requested. For example, a particular proposal or scientific goal may be viewed as a valuable use of 4 hours of telescope time, but may not be as valuable if it requires 80 hours of time, considering the other proposals that it would displace.

Student status. We encourage the use of NRAO telescopes for student research, particularly for Ph.D. dissertations. In this case, it is most helpful if the proposal will say in some manner how the proposed observations will be used in the dissertation, and whether they are a sidelight or a main focus of the thesis. Reviewers are encouraged to comment on this topic, and may choose to support proposals more strongly if there appears to be a well-thought-out program of student research. If a dissertation plan is associated with a proposal, it can be accessed via a pdf view of the proposal.

Proposal length. The scientific justification must obey page limits. Regular and Triggered proposals are allowed a maximum of 4 one-sided pages with 11 point font (minimum) to present the scientific justification, including all figures, tables and references. Large proposals are allowed a maximum of 10 one-sided pages with 11 point font (minimum) to present the scientific justification, including all figures, tables and references. The cover information form is not counted as part of these page limits. The technical justification is included in the cover information form.

Numerical Scores

The NRAO proposal evaluation process uses a scale from 0.1 to 9.9, inclusive, with 0.1 being an outstanding proposal, and 9.9 being a very poor proposal. It is helpful if reviewers use the available dynamic range rather than giving every proposal a score between 2.9 and 3.1! Reviewers’ score distributions each are shifted to have a common mean (equal to 5) and linearly scaled to have a standard deviation of 2; the scores, so normalized, are later averaged for each proposal. This process works best if there is adequate dispersion in the scores prior to normalization.

As noted in Section 3.V, if you sincerely believe that you are unable to assess the scientific merit of a proposal, please do not modify the default score of that proposal; i.e., leave it as 0.0. This will ensure that it is excluded from the score normalization process and from the score average.

Comments

Comments from individual reviewers should be brief and to the point, observing the norms of professional courtesy and providing constructive feedback as appropriate. Please be alert to the role of students in a given proposal when providing feedback.

Comments from individual reviewers have two important roles.

(1) It is not uncommon for different reviewers to come to different conclusions about a given proposal. In resolving these differences, it is important to the function of the SRP for the individual reviewers to comment about what led them to their scoring. The comments from individual reviewers thus serve to guide the SRP to its consensus view about a proposal.

(2) Comments from individual reviewers will not be fully transmitted to the proposers. However, portions of review comments will carry over into the SRP consensus comments, which will be fully transmitted to the proposers and should help the proposers to understand the disposition of their proposals. Specific comments can be extremely helpful here, especially for proposals that are unsuccessful and need to be improved and resubmitted. The comments from individual reviewers thus serve as a source of specific points to appear in the SRP consensus comments.

Specific guidelines to the SRPs for developing consensus comments to the proposers prior to and during the SRP telecon are provided in a separate document.

5 NRAO Conflicts of Interest and Confidentiality Polices

Please review the NRAO Conflicts of Interest and Confidentiality Polices here.