1 NRAO Science Review Panels

Thank you for serving on the ad hoc GMVA-AGN Science Review Panel (SRP). You have been invited to serve on an SRP because of your broad knowledge and expertise in critical areas of contemporary astronomy and astrophysics. The key function of the SRPs, of which there are eight, is to review proposals for use of NRAO’s telescopes based on their scientific merit. The SRPs therefore play a critical role in determining NRAO’s science program, reflecting the scientific community’s judgment of what investigations are important, timely, and influential.

This brief guide provides instructions for entering your individual and independent reviews of proposals using the Proposal Submission Tool (PST; Section 2). Review guidelines are provided in Section 3. NRAO policies on conflicts of interests and confidentiality are fully covered in Section 4.

2 Entering Science Reviews Using the PST

I. Login through the NRAO portal at my.nrao.edu

You will enter the PST at the Dashboard, which allows you to perform a number of functions related to proposal preparation and review, access documentation, and modify your user profile. Please check that your user profile is up to date.

As an SRP member, seven tabs should be visible below the National Radio Astronomy Observatory banner:

• Dashboard

• Proposals

• Reviews

• Data Processing

• Obs Prep

• Helpdesk

• Profile

II. Click on the Reviews tab. If the Reviews tab is not visible, please click the Helpdesk tab and submit a ticket immediately to the Proposal Review department.

III. Upon clicking the Reviews tab you will see a new page with the same banner and primary tabs across the top. It should have one secondary tab below the primary tabs labeled My Reviews. Brief instructions are provided to SRP members in the grey box. More detailed instructions are provided here.

The remainder of the page displays all of the proposals assigned to the Science Review Panel (SRP), of which you are a member.

• Column 1 gives the PST proposal ID.

• Column 2 gives the legacy ID.

• Column 3 gives the proposal title.

• Column 4 gives the proposal authors.

• Column 5 indicates whether the SRP member is conflicted on a proposal.

• Column 6 gives the reason for a given conflict.

The PST automatically captures one type of conflict:

1. the SRP member is the PI or a co-I on a proposal under consideration by the SRP on which they serve.

If you believe any of the conflicts identified by the PST are in error, please promptly submit a Helpdesk ticket to the Proposal Review department.

Please review all other proposal titles and author lists for which the PST has not identified a conflict (unchecked box) and self-declare any conflicts. Examples of conflicts that the PST does not (and cannot) check include circumstances under which one or more of the following is true for the PI or a co-I on a given proposal:

1. they are a spouse or other family member of the reviewer

2. they are a close friend or active collaborator of the reviewer

3. they are a former student or advisor of the reviewer

4. any other reason that leads the reviewer to believe they cannot render a fair and impartial judgment on the scientific merit of the proposal

Consult Section 4 for full details. Once all conflicts have been identified click on Accept conflicts status in the upper right of the page. This takes you to the main review management page.

IV. The banner, primary, and secondary tabs on the review management page remain as before. Below the secondary My Reviews tab is a field and Search button on the left side that can be used to find proposals of interest. To the right are four buttons labeled Export Science Reviews, Import Science Reviews, Print, and Help. The Help button provides general help on the use of the User Portal for proposal preparation. The other three buttons are discussed below.

To the left of the page, below the Search field and button is an Options box that allows you to filter the proposals displayed on the page by telescope or by those that the proposers declare to be Joint Proposals.  You can select "GMVA" under the telescope filter to list only the GMVA proposals.  Additional filtering options may be added in future releases of the PST.

The Proposals List page contains 7 columns.

• Column 1, labeled “All”, allows one to print all proposals that the reviewer is not conflicted on. To do so, click the “All” button. The check boxes of all proposals should automatically be checked. Then click the Print button to the upper right of the page. A window is opened that allows you to select which portions of the proposals to print: the cover sheet, the science justification, the student support application, and/or the student dissertation plan. After making your selections, click on “Print” in the window to print all proposals to a pdf file that can, in turn, be saved to a file on your computer or sent to your printer. One can also use the check boxes in Column 1 to manually check one or more proposals to print via the Print button.

• Column 2 lists the PST proposal ID and also displays two icons. By clicking on the column’s heading, one can sort the proposals in numerical order. By clicking on the PST proposal ID in column 2, highlighted in blue, one can view the proposal. From that proposal’s page:

1. One can use the Options column to the left to navigate to different sections of the proposal (e.g., general, authors, scientific justification, etc.),

2. One can also enter the scientific reviews page by clicking on Reviews in the Options column.

3. One can also directly view other (unconflicted) proposals by clicking on the ID number in the Options column. Clicking My Reviews at the top of the Options column returns you to the main proposal management page.

4. A printer icon below the secondary My Reviews tab allows one to print the proposal in its entirety to a printer or to your computer’s hard disk.

5. Clicking on the pen-and-paper icon in column 2 takes you to the science review page for that proposal (see below); the printer icon in column 2 allows you to send a pdf copy of the proposal to a printer or to your computer’s hard disk.

• Column 3 shows the legacy ID.

• Column 4 gives the proposal title.

• Column 5 gives the name of the PI on the proposal.

• Column 6 indicates whether the reviewer is conflicted on the proposal. NB: if the reviewer is conflicted, they should not be able to view, print, or review the proposal in question.

• Column 7 indicates the status of the review, initially labeled Enter Review.

V. We now offer two ways to enter your proposal reviews. We describe each of these in Sections A and B.

A. One can use the PST to enter the reviews one by one. To enter a science review for a proposal on which you are not conflicted, you can click Enter Review in Column 7 or the pen-and-paper icon in column 2. A less direct path is to click on the proposal ID in column 2 and then click on Reviews under the Options bar from the proposal-viewing page.

The Scientific Reviews page summarizes the review for the proposal of interest.

• Column 1 gives your name.

• Column 2 shows the review comments.

• Column 3 shows a numerical score.

• Column 4 shows review status information.

Clicking on your name in column 1 allows you to enter your science review. Please type comments regarding the scientific merit of the proposal into the box in column 2. Please assign a numerical score of [0.1,9.9]. A low score is better than a high score! Guidelines regarding the science review are provided in Section 3.

NB: On rare occasions you may encounter a proposal for which you believe you are not qualified to review. If you sincerely believe that you are unable to perform a credible assessment of the scientific merit of the proposal, make no changes to the default review information for the proposal. With a default score of 0.0, that proposal will be excluded from the score normalization process. Also submit a Helpdesk ticket to the Proposal Review department, stating the proposal ID and that you believe you are unqualified to review the proposal. NRAO staff will then mark that review as Complete.

The fifth column on the review page allows you to Save or Cancel a review in progress. If you wish to defer completion of the review until later click on Save. If you wish to return the review to a pristine state or to its status following the previous Save, please click Cancel.

If the review is complete, please check the Completed box in column 4 and then click on Save in column 5. If you later find that you have made a mistake in checking the Completed box for a particular proposal, please submit a Helpdesk ticket to the Proposal Review department.

B. The second method to enter your reviews is to write them in a text file external to the PST, and then use the Import Science Reviews button to import them into the PST. This method is advantageous to those who wish to work offline. While preparation of the external text file is straightforward, it must nevertheless conform to some simple conventions. The easiest way to proceed, therefore, is to create an external reviews template by clicking the Export Science Reviews button, which produces a simple text file listing all proposals on which you are not conflicted. Here is an example of a template produced by clicking the Export Science Reviews button:

* ###
* Proposal ID%%%
* Review (text)%%%
* Score (0.1 - 9.9)%%%
* Completed Status (Yes/No)%%%
* ###
VLA/15B-345%%%
%%%
0.0%%%
No%%%
###
GBT/15B-012%%%
%%%
0.0%%%
No%%%
###
VLBA/15B-142%%%
%%%
0.0%%%
No%%%
###

.

.

and so on.

This template can be saved to a file on your computer that you can edit without being logged into the PST.  Alternatively, one can create the external text file manually, making sure that it conforms to the simple conventions outlined above.

The first six lines are comments that start with a "*" symbol and list the four entry fields which are separated by "%%%".  The proposals are separated by "###".   This format allows you too use multiple lines for each field; the pound and percent symbols may be used in your text (assuming you do not have three consecutive symbols).   In the default file there are no comments to the "Review" field and thus there are two "%%%"  expressions back-to-back.

The first entry is the proposal ID (e.g., VLA/15B-345).  The second entry is the review.  Multiple lines may be used here.  The third entry is the numerical score. Please read the instructions regarding proposal scores under section V.A above. The last entry is a simple “Yes” or “No” to indicate whether you have completed the review. If you import a proposal with the entry set to “Yes”, it will not be possible to over-write or otherwise change the review of that particular proposal thereafter. If the entry is set to “No”, it will be possible to change or over-write.

Here is an example of an external text file after some editing has been done:

* ###
* Proposal ID%%%
* Review (text)%%%
* Score (0.1 - 9.9)%%%
* Completed Status (Yes/No)%%%
* ###
VLA/15B-345%%%
This is my first science review.
It is tremendously exciting.%%%
5.5%%%
Yes%%%
###
GBT/15B-012%%%
This is my second science review.
Still a lot of fun.%%%
1.2%%%
No%%%
###
VLBA/15B-142%%%
This is my third science review.
I am now a seasoned veteran.%%%
7.8%%%
Yes%%%
###

To import the reviews, return to the PST reviews page and click on the Import Science Reviews button. At the top of the page, a text pane will appear. To the left of the text pane are three buttons: “Browse”, “Upload”, and “Cancel”. Clicking on the “Browse” button opens a browse window on your computer that allows you to identify the external text file with your science reviews. The directory path and name of the file then appear in the text pane. Click the “Upload” button to import the file into the PST. The PST will report the number of reviews successfully uploaded and whether any problems were encountered. Click “Cancel” if you decide not to import the reviews.

For those reviews that you indicated were incomplete (a “No” in the fourth field), you can either continue to revise them in the text file and upload again later; or you can complete them in the PST. It is also possible to re-export your reviews for further editing outside the PST, and then re-import them. Again, once you have indicated that you have completed a review with a “Yes” in the fourth entry field, no further changes to the review will be allowed by the PST. However, as noted in Section V.A, if you find that you have made a mistake by indicating that a particular proposal has been completed, please submit a Helpdesk ticket to the Proposal Review department.

VI. Once all (unconflicted) reviews are completed, either by manually indicating that this is the case proposal by proposal in the PST, or by setting the fourth field of all proposals to “Yes” in an external review field that is then imported, the PST will acknowledge the fact and thank you!

3 Guidelines for Science Reviews

The purpose of the proposal-selection process for NRAO telescopes is to prioritize and recommend the proposals that potentially are most valuable for the advancement of scientific knowledge. This does not necessarily mean recommending only those proposals that will provide sure results; it also includes a careful consideration of well-reasoned proposals that may be unconventional but provide opportunities for new discoveries. In the evaluation of proposals, we ask that reviewers think about how best to exploit the full capability of the unique scientific instruments that NRAO operates on behalf of the community, namely the VLA, VLBA/GMVA, GBT – and ALMA. In this context, we ask the reviewers to take a constructive approach.

The GMVA proposals you will be reviewing were submitted in response to the Call for Proposals available at

https://science.nrao.edu/observing/call-for-proposals

The large response was due, at least in part, to the availability of ALMA and the Korean VLBI Network (KVN) for the first time in semester 2016B. See 

https://science.nrao.edu/observing/call-for-proposals/2016B/vlba-hsa-vlbi-proposals

If any aspect of these guidelines is unclear to you, please click the Helpdesk tab and submit a ticket to the Proposal Review department.

Scientific Merit

We ask panelists to evaluate proposals, first and foremost, on the basis of their scientific merit. The Observatory seeks proposals that may have high scientific impact, not just "sure things" with modest impact. We note that with each call for proposals, several proposal types may be submitted:

• Regular proposals are those that request less than 200 hrs total observing time on the GBT, VLA, and/or the VLBA/HSA.

• Large proposals are those that request more than 200 hrs total observing time on the GBT, VLA, and/or VLBA/HSA.

• Triggered proposals are those for pre-planned observations of transients whose event times are unknown a priori. Well-defined criteria are required to trigger an observation.

Therefore, in weighing the scientific merit of a given proposal, please be mindful of the differing resource requirements of each. Ultimately SRPs are responsible for producing a rank-ordered list of proposals, based on their scientific merit, for consideration by the TAC. The SRP must compare the relative merit of a diverse set of proposals and opinion will likely vary. Normally, before finalizing its rankings, each SRP will meet via telecon to discuss the proposals and adjust the rankings to reflect consensus opinion. In the case of the ad hoc GMVA-AGN SRP, the need for such a telecon is left to the discretion of the panel. If the panel wishes to have one, please contact Dana Balser (dbalser@nrao.edu) and Jessica Utley (jutley@nrao.edu).

Other Review Criteria

While panelists should base their reviews above all on scientific merit, other factors may be considered.

Publication record. Due recognition should be given to teams who have an established publication record from past related proposals. In contrast, some observers may have considerable data that clearly has not been analyzed and published yet. Reviewers may take these positive and negative factors into account.

Possibility of acquiring more appropriate data. Reviewers also may wish to consider the possibility that other data may be more appropriate for reaching the scientific goals of the proposal. For instance, the relevant data might already exist in the VLBA archive, or in the Canadian Galactic Plane Survey.

Selection of resources. Are the appropriate resources being requested to meet science objectives? For the VLA and VLBA, is/are the right configuration(s) and/or telescopes requested? SRP members must judge whether the requested observation are an appropriate observational approach to the meeting the stated science objectives.

NB: All proposals receive an independent technical review to assess their feasibility. The technical reviews will be available for consideration following the submission of independent scientific reviews, but in advance of the SRP telecon. The technical reviews may be considered as an additional factor by the SRP as part of their telecon discussion.

Resource requirements. Reviewers can and should take into account the amount of resources requested. For example, a particular proposal or scientific goal may be viewed as a valuable use of 4 hours of telescope time, but may not be as valuable if it requires 80 hours of time, considering the other proposals that it would displace.

Student status. We encourage the use of NRAO telescopes for student research, particularly for Ph.D. dissertations. In this case, it is most helpful if the proposal will say in some manner how the proposed observations will be used in the dissertation, and whether they are a sidelight or a main focus of the thesis. Reviewers are encouraged to comment on this topic, and may choose to support proposals more strongly if there appears to be a well-thought-out program of student research. If a dissertation plan is associated with a proposal, it can be accessed via a pdf view of the proposal.

Proposal length. The scientific justification must obey page limits. Regular and Triggered proposals are allowed a maximum of 4 one-sided pages with 11 point font (minimum) to present the scientific justification, including all figures, tables and references. The cover information form is not counted as part of these page limits. The technical justification is included in the cover information form.

Numerical Scores

The NRAO proposal evaluation process uses a scale from 0.1 to 9.9, inclusive, with 0.1 being an outstanding proposal, and 9.9 being a very poor proposal. It is helpful if reviewers use the available dynamic range rather than giving every proposal a score between 2.9 and 3.1! Reviewers' score distributions each are shifted to have a common mean (equal to 5) and linearly scaled to have a standard deviation of 2; the scores, so normalized, are later averaged for each proposal. This process works best if there is adequate dispersion in the scores prior to normalization.

As noted in Section 2.V, if you sincerely believe that you are unable to assess the scientific merit of a proposal, please do not modify the default score of that proposal; i.e., leave it as 0.0. This will ensure that it is excluded from the score normalization process and from the score average.

Comments

Comments from individual reviewers should be brief and to the point, observing the norms of professional courtesy and providing constructive feedback as appropriate. Please be alert to the role of students in a given proposal when providing feedback.

Comments from individual reviewers have two important roles.

(1) It is not uncommon for different reviewers to come to different conclusions about a given proposal. In resolving these differences, it is important to the function of the SRP for the individual reviewers to comment about what led them to their score. The comments from individual reviewers thus serve to guide the SRP to its consensus view about a proposal.

(2) Comments from individual reviewers will not be fully transmitted to the proposers. However, portions of review comments will carry over into the SRP consensus comments, which will be fully transmitted to the proposers and should help the proposers to understand the disposition of their proposals. Specific comments can be extremely helpful here, especially for proposals that are unsuccessful and need to be improved and resubmitted. The comments from individual reviewers thus serve as a source of specific points to appear in the SRP consensus comments.

Specific guidelines to the SRPs for developing consensus comments to the proposers prior to and during the SRP telecon are provided in a separate document.

4 NRAO Conflicts of Interest and Confidentiality Polices

Peer-reviewed proposals to use NRAO facilities for scientific investigations are the bedrock on which observing time is allocated to the community. The NRAO proposal review and time allocation process is described in detail in the Guide to NRAO Proposal Evaluation and Time Allocation. The process relies on the scientific community to evaluate proposals based on their scientific merit and to make recommendations regarding time allocation. The process must be robust and free of real or perceived conflicts of interest and must maintain confidentiality. This document details AUI/NRAO conflicts of interest and confidentiality policies associated with the proposal evaluation and time allocation process.

Conflicts of Interest

For the purposes of the NRAO proposal evaluation and time allocation process, an individual is considered to have a potential conflict of interest if one or more of the following is true:

1. They are a PI or a co-I on a proposal under consideration.

2. They are affiliated with the same institution and department as the PI or a co-I on a proposal under consideration.

3. They are a spouse, partner or other family member of a proposal author.

4. They are a current or recent collaborator of a proposal author.

5. They are a former student or advisor of a proposal author.

6. They have any other reason to believe they cannot render a fair and impartial judgment on the scientific merit of the proposal.

In order to ensure that all proposals are treated fairly and without bias, such potential conflicts must be identified and declared. The response to a conflict depends on the role of the participant. The following participants in the NRAO proposal evaluation and time allocation process must declare conflicts of interest:

• Members of Science Review Panels (SRPs)

• NRAO staff responsible for technical reviews of proposals

• Members of the Time Allocation Committee (TAC)

• NRAO staff that support SRP and TAC meetings

• NRAO staff that participate in the Directors Review

Members of Science Review Panels. Science reviewers are external to NRAO and are subject to all conflict criteria enumerated above. The NRAO automatically captures conflicts of types (1) and (2) with the software tools used to manage the proposal review process. Members of SRPs are expected to otherwise declare those proposals on which they believe themselves to be conflicted. Conflicts of type (4) and (5) are in force if a given reviewer has collaborated with the proposal author, or has been a student of the author, or the thesis advisor of the author, within the last five years. Conflicted panelists will not be allowed to access or review the proposal(s) in question. During the course of SRP discussions, conflicted panelists will not hear or participate in discussion of the relevant proposal(s).

Technical Reviewers. Technical reviews are performed by NRAO staff. Technical reviewers are subject to all conflict types except for those of type (2). In particular, conflicts of type (4) and (5) are, as for science reviewers, only in force if the reviewer has collaborated, been a student, or the thesis advisor of a proposal author within the last five years. Proposals on which a technical reviewer is conflicted are assigned to another reviewer.

Members of the Time Allocation Committee. Each TAC member is the Chair of one of the eight SRPs. As such, they are subject to the same conflicts of interest policy as other SRP members during the science review of proposals. During the face-to-face TAC meeting, rigorous enforcement of all conflict types is impractical. Nevertheless, conflicts of types (1) and (6) are always enforced. TAC members may otherwise collectively identify proposals on which a member is believed to be conflicted. A conflicted TAC member must remove himself or herself from the room when the proposal in question is discussed and recommendations made.

NRAO Support Staff. NRAO staff support the SRP and TAC meetings. They are subject to all conflict types with the exception of type (2). If an NRAO staff member is conflicted during the course of an SRP discussion, they are prevented from hearing discussion of the proposal in question. NRAO staff members do not, in any event, participate in the discussion of the scientific merits of proposals under evaluation. If an NRAO staff member is conflicted during the course of a TAC meeting, they are subject to the same conflict criteria as TAC members.

Directors Review. The Directors Review is intended to review the time allocation process each semester to ensure that all relevant NRAO policies and procedures have been followed, resulting in a fair and transparent allocation of telescope time. NRAO staff members participating in the Directors review are subject to all conflict types except for type (2). Any participant of the Directors review that has a potential conflict must withdraw from discussion of the proposal(s) in question. If such conflicts involve the Director, the Director, too, must withdraw from discussion of any proposal(s) in question. To ensure that appropriate consideration is given to Observatory programmatics and priorities, the Head of Observatory Science Operations will lead the Directors Review in lieu of the Director in such cases. If the Directors Review results in changes to the science program recommended by the TAC, such changes must be recorded. An account of the Directors Review, a description of the changes made to the science program, and the justification for such changes will be prepared by the OSO Head and forwarded to the AUI President for review.

Confidentiality

Both NRAO staff and external members of the scientific community participate in the NRAO proposal evaluation and time allocation process. They will regularly be exposed to confidential information and proprietary data and ideas. The information, data, and ideas must be held in confidence and not disclosed to others. However, several exceptions apply according to the role of the individuals in question.

• Members of a given SRP may disclose and discuss the contents of any proposal assigned to them with one another, consistent with the conflicts of interest policy.

• Members of the TAC may disclose and discuss the contents of any proposal under consideration with each other, consistent with conflicts of interest policy.

• NRAO staff serving as technical reviewers may need to consult with each other to ensure a fair and complete assessment of technical elements of the proposal. Hence, technical details may be disclosed and discussed internally by relevant NRAO staff as needed.

• NRAO staff members that participate in the Directors Review may disclose and discuss the contents of any proposal under consideration with each other, consistent with the conflicts of interest policy.

The following individuals must comply with NRAO policies on confidentiality:

• Members of Science Review Panels

• Members of the Time Allocation Committee

• NRAO staff responsible for:

• technical reviews of proposals

• proposal management software tools and associated data bases

• providing SRP and TAC support

• scheduling approved proposals on NRAO telescopes

• NRAO staff that participate in the Directors Review of TAC recommendations

Proposals printed out for technical or science review must be disposed of by SRP and TAC members in a manner consistent with NRAO policy on confidentiality. Proposals and associated information (source conflicts, proposal rankings, pressure plots, etc.) printed out and made available to TAC members during NRAO TAC meetings must remain at the NRAO. NRAO staff will dispose of these materials in a manner consistent with NRAO policy on confidentiality.

Compliance

All participants in the NRAO proposal evaluation and time allocation will be asked to review the policies detailed herein and asked to confirm that they understand and agree to comply with them.