Observing > Prop Eval & Time Alloc > Documentation > SRP Telecons: Instructions and Guidelines

SRP Telecons: Instructions and Guidelines

by Dana Balser last modified Nov 20, 2017

N.B., This document is under construction for 18B...

1 Overview

The NRAO panel-based proposal review process works in two stages.

First, members of a given Science Review Panel (SRP) independently review the proposals that fall within their science category. A panelist assigns a numerical score to each proposal and provides brief comments to justify the score. When a given panelist has completed all of the reviews they have been assigned, their scores are normalized. Once all panelists on an SRP have completed their reviews, the normalized scores for each proposal are used to form an average and a standard deviation. It is on the basis of these averaged, normalized scores that the preliminary rank-ordered list of proposals is produced for the SRP.

Second, the SRP meets by telecon to discuss the proposals and their preliminary ranking, and to make adjustments to their consensus scores as deemed necessary by the SRP, supported by consensus comments by the panel. The proposals most likely to attract the attention of the SRP are those that show large standard deviations in their averaged, normalized scores. It is during the SRP telecon that members have the opportunity to resolve misunderstandings that may have led to a large variation in scores. It is not expected that a large number of proposals will need to be re-ranked by this process.

The proposals being reviewed were submitted in response to the Call for Proposals available at https://science.nrao.edu/observing/call-for-proposals

As noted in the Call, there are new opportunities for joint observations with the Hubble Space Telescope and/or the Swift Gamma-Ray Burst Mission. Requests for such joint observations should be justified both scientifically and technically; see the Call for specifics.

For VLA proposals, please be mindful of the configuration plans available at

https://science.nrao.edu/facilities/evla/proposing/configpropdeadlines

SRP telecon activities are supported in the Proposal Submission Tool (PST) available through the NRAO portal at my.nrao.edu. If any aspect of these instructions and guidelines is unclear to you, please click the Helpdesk tab in the PST and submit a ticket to the Proposal Review department.

 

2 Before the SRP Telecon

Prior to an SRP telecon, the preliminary rank-ordered list of proposals reviewed by the SRP will be made available to the panel on the Panel Reviews page. To navigate to the Panel Reviews page, click the Reviews tab in the PST and then click the Panel Reviews secondary tab. The Panel Reviews page lists the following information in tabular form and represents the primary working environment for the SRP telecon:

1. Proposal Code, PI, Type, Joint, Hours, Thesis, plus a Print Proposal icon

2. Title

3. Averaged Normalized Score

4. Standard Deviation of Averaged Normalized Score

5. SRP Consensus Score

6. SRP Consensus Comments for PI & TAC

7. SRP Consensus Comments for TAC Only

8. Status of the consideration by the SRP

9. Review toggle to open or close the Review display

In the list above, PI refers to the Principal Investigator and TAC refers to the Time Allocation Committee.

In the interest of efficiency and fairness, we ask the SRP Chair to assign a primary and secondary presenter for each proposal. When doing these assignments, the Chair should take note of the conflicts shown on the Chair’s Reviews Summary secondary tab. As an organizational aid for the telecon, the Chair may push the Export Panel Reviews button to export reviews and conflict information to a csv formatted file that lists: rank, proposal id, normalized mean score, standard deviation, title, P.I., the requested time in hours, is it a thesis proposal, is it a joint proposal, is it a multi-configuration proposal (VLA only), and if the reviewer (indicated with initials) is conflicted on this proposal.  Here is an example:

 

Export Panel Review csv File Example
RankProposalScoreStd.TitleP.I.TimeThesisJointMulti.DMJARvJKLRSN
1

GBT/15B -101

2.97 1.24 Black Holes Smith 3.50 Yes Yes
2 VLA/15B -121 4.01 2.01 Stars Li 45.0 Yes Yes
etc.

 

We suggest that, prior to the SRP telecon, the primary presenter enter a draft consensus review, based on the individual comments of their fellow panelists, into the expandable SRP Consensus box labeled “PI & TAC Comments”. Cues are present in this consensus box. We strongly encourage those entering consensus reviews to follow the basic structure suggested by these cues, namely, to provide a brief

Summary of the proposed observations to reassure the proposers that the SRP understands the proposal; to describe the Proposal strengths and Proposal weaknesses; to alter the default phrase for Recommended time if appropriate; and to describe any Technical issues affecting ranking or recommended time. Regarding the latter, you may assume that the portion of the technical review in the box labeled “Comments To PI” will be transmitted in full to the proposers.

The secondary presenter is also expected to be familiar with the proposal. We urge both presenters to be alert to those proposals that serve as the basis for thesis research on the part of a graduate student. Such proposals are accompanied by dissertation plans, which the presenters should also be familiar with. For a typical proposal cycle, each SRP member would serve as primary presenter for about 10 proposals and as secondary presenter for a similar number.

Individual science and technical reviews can be viewed when editing the consensus review (by either clicking the proposal ID or the Review link in the last column).  Alternatively you can click the Export Individual Reviews button to generate an ascii file that includes all of the Individual reviews in one place. 

 

3 SRP Telecon Logistics

The NRAO has coordinated with each SRP to set the telecon date(s) and time(s). Telecons will occur about 1-2 weeks after the individual review deadline. The telecon will be attended by the members and Chair of the SRP. The telecon will also be attended by an NRAO moderator who will manage telecon mechanics, and by a member of the NRAO scientific staff who can provide input on technical, procedural and policy issues. A telephone number and pass code will be provided prior to the telecon.

 

4 SRP Telecon Function

The SRP comprises five members and the Chair. The Chair will normally lead the telecon. However, for those cases where proposals must be discussed on which the Chair is conflicted, an SRP member should be designated by the SRP as Chair pro tem to lead the telecon.

The SRP telecon is supported by software tools provided in the PST. During the telecon the members and Chair of the SRP should be logged into the PST and be viewing the Panel Reviews page.

During the first few minutes of the telecon, the SRP should establish a discussion order for the list of proposals. That discussion order should be clearly announced to all telecon participants, including the NRAO moderator. While considering possible discussion orders, the SRP should note that the Panel Reviews page can be filtered by telescope and sorted by the Averaged Normalized Score, SRP Consensus Score, or Standard Deviation of the Average Normalized Score of the proposals. For example, a given SRP may decide to consider proposals by telescope, with the proposals ranked by score for that telescope. There might be a few proposals that are declared as joint with another NRAO telescope and the SRP may wish to consider such joint proposals back-to-back; a Joint Proposal filter can be set to help identify such proposals. The members and Chair of the SRP will be able to resort and refresh the Panel Reviews page during the course of the Panel’s work.

During the telecon the Chair should be aware of the conflicts shown on the Chair’s Reviews Summary page. We recommend that the Chair print that page so they can quickly glance at it. If the Chair notes conflicts for a proposal about to be discussed, the Chair should clearly announce those conflicts to the SRP members and the NRAO moderator.

We leave it to the discretion of the SRP to decide the depth to which a given proposal is discussed. It will likely be the case that the proposals with the lowest and highest Averaged Normalized Scores will be relatively non-controversial. Those proposals with Averaged Normalized Scores in the middle range and those proposals with large Standard Deviations in their Averaged Normalized Scores may therefore require the most discussion.

Consideration of a given proposal will normally involve the following steps:

  1. Select either Proposal Code or Review to open a Review display with columns labeled Authors, Technical Reviews, Science Reviews, Consensus and Complete.

  2. The primary presenter briefly summarizes the proposal and its (unattributed) Science Reviews Comments as distilled in the draft consensus comments. The secondary presenter may also contribute to the summary.

  3. The proposal is then open for discussion. Any panel member that is conflicted on a particular proposal should not participate in the discussion or any decisions regarding revisions to the SRP Consensus Score or SRP Consensus Comments. We leave it to the discretion of the Chair and members as to whether a conflicted individual should disconnect from the telecon. (If it is decided that a conflicted individual should disconnect, the NRAO moderator will be able to reconnect with that individual following discussion of the proposal in question.) The SRP should consider the technical review provided by NRAO staff, and note any technical issues that affected the SRP Consensus Score or the recommended time. The SRP should also be alert to source conflicts. The SRP will be provided with a list of potential source conflicts. The SRP, based on their consideration of the proposals in question, will decide whether the conflicts are real. If they are real, they should advise the TAC which proposal has greater scientific merit in their view. This advice should be conveyed as text in the expandable SRP Consensus box labeled “TAC only Comments”. Selecting Save in the rightmost column will save the contents of the comment boxes. Selecting Cancel will return the contents of the comment boxes to their last saved state.

  4. If the Averaged Normalized Score shows a large Standard Deviation, possible reasons should be discussed and the SRP Consensus Score should be adjusted as necessary.

  5. By default, the value of the SRP Consensus Score is set to the value of the Averaged Normalized Score. If it is necessary to revise the SRP Consensus Score, the Chair enters the revised value then selects Save in the rightmost column. Reasons for adjusting the score may be noted in the SRP Consensus box labeled “TAC only Comments”. In particular, if the SRP Consensus score is adjusted because a thesis is involved, it is important to so inform the TAC by entering a comment to that effect. Selecting Cancel will return the content of the comment box to its last saved state.

  6. For a VLA proposal that requests observations in multiple configurations, and that is not at the very top or bottom of the ranked list, please make sure that the Chair is prepared (e.g., via “TAC only Comments”) to discuss at the TAC meeting whether the multiple configurations must be scheduled for the proposal to be scientifically worthwhile.

  7. If the Chair is conflicted, the Chair pro tem leads the discussion and, if need be, the NRAO moderator enters the revised SRP Consensus Score and selects Save in the rightmost column.

  8. When the Chair or Chair pro tem is satisfied that the SRP Consensus Score and SRP Consensus Comments reflect the Panel consensus, the Chair or the NRAO moderator checks the Complete button and selects Save in the rightmost column; these actions indicate that consideration of the proposal is complete. It is possible to un-check the Complete button if necessary.

After all proposals have been considered, they need to be marked Complete. Proposals can be marked Complete one by one (see above), or en masse when the SRP Chair clicks the Complete All Panel Reviews button. After a brief consultation between the SRP Chair and Dana Balser, the SRP Chair will declare the panel’s work finalized by clicking the Finalize Panel Reviews button. The Finalize Panel Reviews button should in any case require a “confirm” before performing the action. Once the action has been performed, no modifications to the SRP Consensus Scores or SRP Consensus Comments are allowed.

Our experience during prior semesters has shown that this finalize step was not necessarily possible at the conclusion of the telecon. In some cases consensus reviews still needed to be revised. We intend to be flexible on this front. Our objective is for each proposal to receive the attention it deserves and to allow the SRP the time and flexibility it needs to complete its work. NRAO staff will work with the SRPs to ensure that the process is brought to completion in a reasonable way.

The final ranked list of proposals plus the SRP Consensus Scores, PI & TAC Comments, and TAC Only Comments will be made available to the TAC.

 

5 Additional Considerations

The SRP will be considering the science cases for disparate telescopes (VLA, VLBA, GBT) and proposal types (large, regular, triggered). While the technical reviews are available to the SRP, the panel should be primarily concerned with scoring the proposals on the basis of the science cases made. We are aware, however, that the proposed science can be closely coupled to an instrument: the VLBA is suitable for certain astrometric measurements whereas the GBT is not; the GBT is suitable for certain low-column-density measurements of HI or molecules, whereas the VLA is not. The SRP should feel free to comment on the suitability of the proposed observational strategy to address the science objectives of a particular proposal.

Most proposals submitted on the NRAO semester deadlines are expected to be “regular” proposals; that is, those that request less than 200 hrs total observing time on one or more NRAO telescopes.

“Large” proposals are those that request total allocations exceeding 200 hrs total observing time on one or more NRAO telescopes. The SRP will evaluate large proposals along with all other proposal types. In the spirit of “extraordinary requests require extraordinary justification,” large proposals will be held to a higher standard. They must meet all submission requirements stated in the

NRAO Large Proposal Policy

https://science.nrao.edu/observing/proposal-types/largeproppolicy

and offer a compelling science justification for the time request made. Large proposals will be compared with other large proposals by the TAC. Unless they have very good scores, they will not fare well.

The third type of proposal submitted for the normal semester proposal deadline is the “triggered” proposal. These proposals request time to observe phenomena that are predictable in general, but not in detail, e.g., a proposal to observe the next flaring X-ray binary that meets specific criteria. The proposal must include clearly stated triggering criteria for observing the source type and phenomenon of interest. The proposal must also include estimates of the number of events that must be observed to constitute a statistically meaningful sample, the rate at which the events of interest are expected to occur, and the amount of observing time it is anticipated will be needed to meet observational and scientific goals. It is the role of the SRP to judge whether the triggering criteria, sample size, and net time request are well-considered.

 

6 Final Notes

We thank the SRP for its willingness to participate in the NRAO proposal evaluation and time allocation process. We welcome any suggestions that you can offer to help us make the process more efficient and robust, recognizing that we must at the same time ensure that we treat each proposal fairly and with complete confidentiality. Above all, the process should result in the best, most exciting, and highest impact science programs being scheduled on NRAO telescopes.