Science Review Panels

All proposals for use of NRAO facilities are peer reviewed by members of the scientific community. Each SRP comprises at least six members, a Chair and five or more additional panelists. The term of an SRP member is normally two years. The responsibility of an SRP is to review and to produce a rank-ordered list of those proposals submitted in a given semester that fall within its scientific purview. The SRP Chair also serves on the Telescope Allocation Committee (TAC).

SRP Proposal Review

The key function of an SRP is to review proposals based on scientific merit. Shortly after a proposal submission deadline (see the timeline), proposals within a given science category are made available to the relevant SRP for review. The SRP uses tools provided by the PST for proposal review.

The purpose of the entire proposal-selection process for NRAO telescopes is to select the proposals that potentially are most valuable for the advancement of scientific knowledge. This does not necessarily mean giving high marks to those proposals that will provide sure results; it also includes a careful consideration of well-reasoned proposals that may be unconventional, but provide opportunities for new discoveries. In the evaluation of proposals, we ask that the reviewers think about how best to exploit the full capability of the unique scientific instruments that NRAO operates for the community, namely the VLA, VLBA, GBT, and ALMA. In this context, we ask the reviewers to take a constructive approach. For instance, the TAC is eager to receive comments about ways in which proposals might be implemented to enhance their scientific value, such as the allocation of additional time, additional telescope configurations, or even additional NRAO telescopes. The goal should not be to provide the bare minimum of observing time to get the maximum number of observers on the telescopes, but rather to maximize the scientific impact of the observations undertaken with the telescopes.

Review Criteria

The Science Review Panels are charged with reviewing proposals within their purview on the basis of scientific merit. We ask panelists to carefully consider the question: "How scientifically valuable is this use of VLA/VLBA/GBT time, compared to the other proposals under review?" We seek proposals that may have high scientific impact, not just "sure things" with modest impact. For the VLBA, proposals that require extra resources such as the GBT also should be held to higher standards, and the need for these resources must be quantitatively justified. We leave the exact details of this to the individual reviewers. But if proposals that require a lot of resources to accomplish their goals are judged to be very valuable scientifically, we strongly encourage the reviewers to give them a very good score, and to comment on the importance of doing the entire project. Particularly in such cases, it is quite reasonable for reviewers to evaluate whether the proposal team can marshal the resources and expertise to complete and publish the observations.

The SRPs are not responsible for evaluating the technical feasibility of a proposal. The technical feasibility of a given proposal is addressed in a separate technical review by an NRAO staff member. The technical review offers no opinion regarding the scientific merit of a given proposal and therefore provides no numerical score. It assesses whether the proposed observing program can be successfully implemented with the resources requested in the proposal.

While panelists should base their individual reviews above all on scientific merit, other factors may be considered:

Publication record. Due recognition should be given to teams who have an established publication record fro past related proposals. In contrast, some observers may have considerable data that clearly has not been analyzed and published yet. Reviewers may take these positive and negative factors into account.

Possibility of acquiring more appropriate data. Reviewers  may also wish to consider the possibility that other data may be more appropriate for reaching the scientific goals of the proposal. For instance, the relevant data might already exist in the VLBA archive.

Selection of resources. Reviewers may judge whether the requested observations are an appropriate approach to meeting the stated science objectives.

Resource requirements. Reviewers may take into account the amount of resources requested. For example, a particular proposal or scientific goal may be viewed as a valuable use of 4 hours of telescope time, but may not be as valuable if it requires 80 hours of time.

Student status. We encourage the use of NRAO telescopes for student research, particularly for Ph.D. dissertations. In this case, it is most helpful if the proposal will say in some manner how the proposed observations will be used in the dissertation, and whether they are a sidelight or a main focus of the thesis. Reviewers are encouraged to comment on this topic, and may choose to support proposals more strongly if there appears to be a well-thought-out program of student research. If a dissertation plan is associated with a proposal, the plan can be consulted.

The NRAO uses a numerical scale from 0.1 to 9.9, with a low score being better and a high score being worse. It is helpful if individual reviewers make use of the available scoring range rather than giving every proposal a score between, e.g., 2.9 and 3.1! Reviewers' score distributions each are shifted to have a common mean (equal to 5) and linearly scaled to have a standard deviation of 2; the scores, so normalized, are later averaged for each proposal. This process works best if there is adequate dispersion in the scores prior to normalization.

Comments by individual reviewers are very important for the review process. It is not uncommon for different reviewers to come to quite different conclusions about proposals. Therefore, in order to reconcile such differences when the SRP meets, it is important for panelists to comment about what factors lead them to their numerical score. Do not think "I'm giving it a very good score, so I don't have to say why." Let your fellow panelists know what the specific scientific impact is likely to be. Comments by reviewers should observe the norms of professional courtesy.

Conflicts of Interest

The proposal management tools in the PST automatically flag those proposals on which panelists are conflicted:

1. the panelist is a PI or co-I on the proposal in question
2. the panelist is at the same institution as the PI or co-I on the proposal in question

In these cases, the proposal is assigned to the SRP Chair for review. In addition, before beginning their assigned review, panelists must self-declare whether they believe themselves to be conflicted on any particular proposal - that is, if they believe they cannot render a fair and unbiased opinion on a given proposal, they should declare themselves to be conflicted. Examples include proposals by teams that include a reviewer's spouse, significant other, or other immediate family members; it may include past students or advisors, collaborators on other projects, or competitors, and so on. We leave this issue to the reviewer's discretion and conscience. If a panelist self-declares a conflict, the proposal in question is again assigned to the SRP Chair for review.

Each SRP normally has about a month to provide individual reviews of all proposals for which it is responsible. Once each panelist has reviewed all their proposals, they must submit their final reviews. Once all panelists have done so, the scores for each reviewer are normalized and then averaged for each proposal. The averaged normalized score is used to establish a preliminary rank-ordered list of proposals for each SRP.

SRP Meetings

Each SRP then meets via telecon to review the preliminary rank-ordered list of proposals. SRP meetings are not expected to be more than a few hours in duration. It is sometimes the case that reviewer opinion and, hence, scores differ considerably for one or more proposals. The reasons for these differences may lie in a misinterpretation of the proposal's scientific justification, in the reviewers' understanding of the observational strategy, or a genuine difference of opinion concerning the scientific merit of the proposal. The purpose of the SRP meeting is to reconcile such differences. If it is deemed necessary, the rank of a proposal may be changed by the SRP Chair.

Once the SRP has completed its review and agrees on the final proposal rankings, the Chair certifies the rankings. Those rankings are then used to define a linear-rank score for each of the N proposals considered by the SRP. Specifically, the proposals are ranked R = 1, 2, ... N, and each proposal acquires a linear-rank score of 10*R / N. Quartile boundaries for linear-rank scores are 2.5, 5.0 and 7.5.