1 Overview

The NRAO panel-based proposal review process works in two stages.

First, members of a given Science Review Panel (SRP) independently review the proposals that fall within their science category. A panelist assigns a numerical score to each proposal and provides brief comments to justify the score. When a given panelist has completed all of the reviews they have been assigned, their scores are normalized. Once all panelists on an SRP have completed their reviews, the normalized scores for each proposal are used to form an average and a standard deviation. It is on the basis of these averaged, normalized scores that the preliminary rank-ordered list of proposals is produced for the SRP.

Second, the SRP meets remotely to discuss the proposals and their preliminary ranking, and to make adjustments to their consensus scores as deemed necessary by the SRP, supported by consensus comments by the panel. The proposals most likely to attract the attention of the SRP are those that show large standard deviations in their averaged, normalized scores. It is during the SRP meeting that members have the opportunity to resolve misunderstandings that may have led to a large variation in scores. It is not expected that a large number of proposals will need to be re-ranked by this process.

The proposals being reviewed were submitted in response to the Call for Proposals available at https://science.nrao.edu/observing/call-for-proposals

As noted in the Call, there are opportunities for joint observations with ALMA, the James Webb Space Telescope (JWST), the Hubble Space Telescope (HST), the Swift Gamma-Ray Burst Mission, Chandra, XMM-Newton, and the Neutron Star Interior Composition Explorer (NICER).  Requests for such joint observations should be justified both scientifically and technically; see the Call for specifics.

For VLA proposals, please be mindful of the configuration plans available at

https://science.nrao.edu/facilities/evla/proposing/configpropdeadlines

SRP meeting activities are supported in the Proposal Submission Tool (PST) available through the NRAO portal at my.nrao.edu. If any aspect of these instructions and guidelines is unclear to you, please click the Helpdesk tab in the PST and submit a ticket to the Proposal Review department.

2 Before the SRP Meeting

Prior to an SRP meeting, the preliminary rank-ordered list of proposals reviewed by the SRP will be made available to the panel on the Panel Reviews page. To navigate to the Panel Reviews page, click the Reviews tab in the PST and then click the Panel Reviews secondary tab. The Panel Reviews page lists the following information in tabular form and represents the primary working environment for the SRP meeting:

1. Proposal Code, PI, Type, Joint, Hours, Thesis, plus a Print Proposal icon

2. Title

3. Averaged Normalized Score

4. Standard Deviation of Averaged Normalized Score

5. SRP Consensus Score

6. SRP Consensus Comments for PI & TAC

7. SRP Consensus Comments for TAC Only

8. Status of the consideration by the SRP

9. Review toggle to open or close the Review display

In the list above, PI refers to the Principal Investigator and TAC refers to the Time Allocation Committee.

As an organizational aid for the meeting, the Chair may push the Export Panel Reviews button to export this information to a csv formatted file that lists: rank, proposal id, normalized mean score, standard deviation, title, P.I., the requested time in hours, is it a thesis proposal, is it a joint proposal, is it a multi-configuration proposal (VLA only), and the review type for each reviewer (indicated with initials) on this proposal.  Here is an example where we have abbreviated the review type (P->primary, S->secondary, T->tertiary, N->none, and C->conflict):

We strongly suggest that, prior to the SRP meeting, the primary reviewer enter a draft consensus review, based on the individual comments of their fellow panelists, into the expandable SRP Consensus box labeled “PI & TAC Comments”. Cues are present in this consensus box. We strongly encourage those entering consensus reviews to follow the basic structure suggested by these cues, namely, to provide a brief Summary of the proposed observations to reassure the proposers that the SRP understands the proposal; to describe the Proposal strengths and Proposal weaknesses; to alter the default phrase for Recommended time if appropriate; and to describe any Technical issues affecting ranking or recommended time. Regarding the latter, you may assume that the portion of the technical review in the box labeled “Comments To PI” will be transmitted in full to the proposers.

The secondary reviewer should read the draft consensus comments and provide any feedback to the primary.  During the meeting the tertiary reviewers should comment on the proposal.  All members who have read a proposal are welcome to participate in discussion of a proposal during the meeting.  The Chair will moderate the discussion and make any adjustment to the SRP score, weighting input from members appropriately taking into account their role (reviewer or otherwise) and any specialized knowledge they may bring.  We urge the panel to be alert to those proposals that serve as the basis for thesis research on the part of a graduate student. Such proposals are accompanied by dissertation plans, which the presenters should also be familiar with.

Individual science and technical reviews can be viewed when editing the consensus review (by either clicking the proposal ID or the Review link in the last column).  Alternatively you can click the Export Individual Reviews button to generate an ascii file that includes all of the Individual reviews in one place. 

3 SRP Meeting Logistics

The NRAO has coordinated with each SRP to set the meeting date(s) and time(s). Meetings will occur about 1-2 weeks after the individual review deadline. The meeting will be attended by the members and Chair of the SRP. The meeting will also be attended by an NRAO moderator who will manage meeitng mechanics, and by a member of the NRAO scientific staff who can provide input on technical, procedural, and policy issues. Contact information will be provided before the meeting.

4 SRP Meeting Function

The SRP comprises five members and the Chair. The Chair will normally lead the meeting. However, for those cases where proposals must be discussed on which the Chair is conflicted, an SRP member should be designated by the SRP as Chair pro tem to lead the meeting.

The SRP meeting is supported by software tools provided in the PST. During the meeting the members and Chair of the SRP should be logged into the PST and be viewing the Panel Reviews page.

During the first few minutes of the meeting, the SRP should establish a discussion order for the list of proposals. That discussion order should be clearly announced to all participants, including the NRAO moderator. While considering possible discussion orders, the SRP should note that the Panel Reviews page can be filtered by telescope and sorted by the Averaged Normalized Score, SRP Consensus Score, or Standard Deviation of the Average Normalized Score of the proposals. For example, a given SRP may decide to consider proposals by telescope, with the proposals ranked by score for that telescope. There might be a few proposals that are declared as joint with another NRAO telescope and the SRP may wish to consider such joint proposals back-to-back; a Joint Proposal filter can be set to help identify such proposals. The members and Chair of the SRP will be able to resort and refresh the Panel Reviews page during the course of the Panel’s work.

During the meeting the Chair should be aware of the conflicts shown on the Chair’s Reviews Summary page. We recommend that the Chair print that page so they can quickly glance at it. If the Chair notes conflicts for a proposal about to be discussed, the Chair should clearly announce those conflicts to the SRP members and the NRAO moderator.

We leave it to the discretion of the SRP to decide the depth to which a given proposal is discussed. It will likely be the case that the proposals with the lowest and highest Averaged Normalized Scores will be relatively non-controversial. Those proposals with Averaged Normalized Scores in the middle range and those proposals with large Standard Deviations in their Averaged Normalized Scores may therefore require the most discussion.

Consideration of a given proposal will normally involve the following steps:

1. Select either Proposal Code or Review to open a Review display with columns labeled Authors, Technical Reviews, Science Reviews, Consensus and Complete.

2. The primary presenter briefly summarizes the proposal and its (unattributed) Science Reviews Comments as distilled in the draft consensus comments. The secondary presenter may also contribute to the summary.

3. The proposal is then open for discussion. Any panel member that is conflicted on a particular proposal should not participate in the discussion or any decisions regarding revisions to the SRP Consensus Score or SRP Consensus Comments. We leave it to the discretion of the Chair and members as to whether a conflicted individual should disconnect from the meeting. (If it is decided that a conflicted individual should disconnect, the NRAO moderator will be able to reconnect with that individual following discussion of the proposal in question.) The SRP should consider the technical review provided by NRAO staff, and note any technical issues that affected the SRP Consensus Score or the recommended time. The SRP should also be alert to source conflicts. The SRP will be provided with a list of potential source conflicts. The SRP, based on their consideration of the proposals in question, will decide whether the conflicts are real. If they are real, they should advise the TAC which proposal has greater scientific merit in their view. This advice should be conveyed as text in the expandable SRP Consensus box labeled “TAC only Comments”. Selecting Save in the rightmost column will save the contents of the comment boxes. Selecting Cancel will return the contents of the comment boxes to their last saved state.

4. If the Averaged Normalized Score shows a large Standard Deviation, possible reasons should be discussed and the SRP Consensus Score should be adjusted as necessary.

5. By default, the value of the SRP Consensus Score is set to the value of the Averaged Normalized Score. If it is necessary to revise the SRP Consensus Score, the Chair enters the revised value then selects Save in the rightmost column. Reasons for adjusting the score may be noted in the SRP Consensus box labeled “TAC only Comments”. In particular, if the SRP Consensus score is adjusted because a thesis is involved, it is important to so inform the TAC by entering a comment to that effect. Selecting Cancel will return the content of the comment box to its last saved state.

6. It is important to add "TAC only Comments" for the following items: (i) Recommendation for joint external proposals (e.g., HST).  The TAC has the authority to recommend telescope time for external facilities and therefore specific recommendations from the SRP based on the scientific justification are useful in allocating these resources.  (ii) Recommendation for future VLA configurations.  If the TAC recommends that a proposal be approved for future VLA configurations it may displace a future proposal; so the SRP should indicate if this is justified based on the proposed science.   (iii) Whether or not you approve of the dissertation plan w.r.t. the proposal.  (iv) If you want to specifically kill a proposal.

7. For a VLA proposal that requests observations in multiple configurations, and that is not at the very top or bottom of the ranked list, please make sure that the Chair is prepared (e.g., via “TAC only Comments”) to discuss at the TAC meeting whether the multiple configurations must be scheduled for the proposal to be scientifically worthwhile.

8. If the Chair is conflicted, the Chair pro tem leads the discussion and, if need be, the NRAO moderator enters the revised SRP Consensus Score and selects Save in the rightmost column.

9. When the Chair or Chair pro tem is satisfied that the SRP Consensus Score and SRP Consensus Comments reflect the Panel consensus, the Chair or the NRAO moderator checks the Complete button and selects Save in the rightmost column; these actions indicate that consideration of the proposal is complete. It is possible to un-check the Complete button if necessary.

After all proposals have been considered, they need to be marked Complete. Proposals can be marked Complete one by one (see above), or en masse when the SRP Chair clicks the Complete All Panel Reviews button. After a brief consultation between the SRP Chair and Dana Balser, the SRP Chair will declare the panel’s work finalized by clicking the Finalize Panel Reviews button. The Finalize Panel Reviews button should in any case require a “confirm” before performing the action. Once the action has been performed, no modifications to the SRP Consensus Scores or SRP Consensus Comments are allowed.

Our experience during prior semesters has shown that this finalize step was not necessarily possible at the conclusion of the meeting. In some cases consensus reviews still needed to be revised. We intend to be flexible on this front. Our objective is for each proposal to receive the attention it deserves and to allow the SRP the time and flexibility it needs to complete its work. NRAO staff will work with the SRPs to ensure that the process is brought to completion in a reasonable way.

The final ranked list of proposals plus the SRP Consensus Scores, PI & TAC Comments, and TAC Only Comments will be made available to the TAC.

5 Additional Considerations

The SRP will be considering the science cases for disparate telescopes (VLA, VLBA, GBT) and proposal types (large, regular, triggered). While the technical reviews are available to the SRP, the panel should be primarily concerned with scoring the proposals on the basis of the science cases made. We are aware, however, that the proposed science can be closely coupled to an instrument: the VLBA is suitable for certain astrometric measurements whereas the GBT is not; the GBT is suitable for certain low-column-density measurements of HI or molecules, whereas the VLA is not. The SRP should feel free to comment on the suitability of the proposed observational strategy to address the science objectives of a particular proposal.

Most proposals submitted on the NRAO semester deadlines are expected to be “regular” proposals; that is, those that request less than 200 hrs total observing time on one or more NRAO telescopes.

“Large” proposals are those that request total allocations exceeding 200 hrs total observing time on one or more NRAO telescopes. The SRP will evaluate large proposals along with all other proposal types. In the spirit of “extraordinary requests require extraordinary justification,” large proposals will be held to a higher standard. They must meet all submission requirements stated in the

NRAO Large Proposal Policy

https://science.nrao.edu/observing/proposal-types/largeproppolicy

and offer a compelling science justification for the time request made. Large proposals will be compared with other large proposals by the TAC. Unless they have very good scores, they will not fare well.

The third type of proposal submitted for the normal semester proposal deadline is the “triggered” proposal. These proposals request time to observe phenomena that are predictable in general, but not in detail, e.g., a proposal to observe the next flaring X-ray binary that meets specific criteria. The proposal must include clearly stated triggering criteria for observing the source type and phenomenon of interest. The proposal must also include estimates of the number of events that must be observed to constitute a statistically meaningful sample, the rate at which the events of interest are expected to occur, and the amount of observing time it is anticipated will be needed to meet observational and scientific goals. It is the role of the SRP to judge whether the triggering criteria, sample size, and net time request are well-considered.

6 Final Notes

We thank the SRP for its willingness to participate in the NRAO proposal evaluation and time allocation process. We welcome any suggestions that you can offer to help us make the process more efficient and robust, recognizing that we must at the same time ensure that we treat each proposal fairly and with complete confidentiality. Above all, the process should result in the best, most exciting, and highest impact science programs being scheduled on NRAO telescopes.