N.B., The TAC process has evovled since 2014 and therefore some of the advice listed below may be out-of-date.

Advice to New (and Returning) SRP Chairs

with input from A. Baker, T. Bania, J. Cordes, M. Eracleous, J. Lazio, S. Lizano, M. Putman, J., Rosenberg, D. Sanders, R. Zavala, et al. – July 7, 2014

This document has been assembled with the goal of helping the chairs of NRAO Science Review Panels (SRPs) fulfill their obligations, including shepherding their panels’ proposals through a Time Allocation Committee (TAC) whose practices may initially be unfamiliar. The insights provided here represent a mixture of NRAO policy and SRP/TAC custom as of semester 2014B.

I. SRP telecon checklist
As motivated in the sections that follow below, it is helpful if your SRP during its telecon tracks the following seven considerations for each proposal it reviews:

1. Is any SRP member conflicted on this proposal?
If so, (s)he should be excluded from the ensuing discussion unless you have exercised your discretion based on, e.g., length of author list or size of institution (see below).

2. Is this a student thesis proposal?
If so, the SRP should evaluate the dissertation plan with respect to the proposal (see below).

3. Is this a triggered proposal?
If so, the SRP should comment to the proposers and/or TAC on whether the trigger criteria are defined with adequate specificity and are appropriate for the proposal’s science goals. You should also be prepared (e.g., with the help of a TAC-only comment) to discuss at the TAC meeting which of two or more competing proposals with similar trigger criteria should have
priority in the event of a conflict.1

4. Is this a VLA proposal requesting time in multiple configurations?
If so, and especially if any of these configurations can only be scheduled in future semesters, you should be prepared (e.g., with the help of a TAC-only comment) to discuss at the TAC meeting whether all configurations must be scheduled for the project to be scientifically useful.

5. Is this a VLBA proposal requesting GBT or phased VLA time?
If so, you should be prepared (e.g., with the help of a TAC-only comment) to discuss at the TAC meeting whether GBT and/or phased VLA observations must be scheduled for the project to be scientifically useful.

6. Does this proposal feature a target conflict?
In advance of the SRP telecon, NRAO will provide you with a list of possible target conflicts with current and previously approved proposals from your panel. You should be prepared (e.g., with the help of a TAC-only comment) to identify for the TAC any definite (i.e., same frequency and configuration) conflict and discuss how it should be resolved.

7. Does this proposal request observations with a non-NRAO telescope?
If so, and especially in cases (e.g., Hubble) where funding is at stake, the non-NRAO observations should be closely connected to the proposal’s goals, and the proposal should be well-motivated overall. It may be convenient for you to assign each of your fellow SRP members the job of keeping track
of 1–2 of these considerations during the telecon.

Competition may also exist with previously approved proposals. For the VLA, a list of all currently active triggered proposals can be found at http://www.aoc.nrao.edu/~schedsoc/ActiveTriggeredProposalsVLA.txt.

II. SRP Workflow

Past experience recommends a few “best practices” for managing your SRP’s workflow during its phase of the review.

• Heading into the SRP telecon, it is useful for the chair to flag for particular discussion any proposal that has a large standard deviation and/or a small number of individual scores contributing to the averaged normalized score.

• It is helpful if primary reviewers can construct draft versions of consensus comments---on the basis of individual reviewer comments, and to the extent that consensus actually exists---before the SRP telecon. This approach makes it possible to discuss and edit the wording of consensus comments in near-real time during the telecon without sacrificing efficiency.

• Draft (and final) consensus comments should close with the explicit statement “Recommended time: The SRP recommends no alteration to the proposed time request” or an appropriate modification thereof. For a proposal deemed a complete waste of time, this statement should become “Recommended time: The SRP time recommends that no time be scheduled.” Consensus comments should also include all of the information prompted for by the other cues in the NRAO proposal review system. It is possible to construct the consensus comments in the form of a bulleted Cue:Response list, or in the form of a paragraph, as long as all of the required information is present.

• If the list of possible source conflicts that is provided to the SRP includes clashes with proposals from a previous semester, it is a good idea to request (again, before the telecon) information from an NRAO contact about the authors, contents, and dispositions of those proposals, so that you and unconflicted members of your panel will be able to assess whether the conflicts are significant.

• SRPs should be aware that current default practice is to consider thesis proposals during the SRP telecon, whereby the SRP score may or may not be adjusted based on the merits of the dissertation plan with respect to the proposal.  This information should be passed onto the TAC via the SRP comments to the TAC textbox.  The TAC cannot change the score but may use this information when evaluating resource and scheduling issues.

• During the SRP telecon, if NRAO conflict policies would result in a huge fraction of the panel having to sit out discussion of a particular proposal, it may be advisable for the chair to use discretion in how rigidly to enforce those policies (e.g., considering order within a very long author list, or size of institution) so that the proposal can be effectively reviewed.

• Some past panels have found the following procedure for discussing proposals during the SRP telecon to be useful:

– The primary reviewer (designated by the chair well before the telecon) starts the discussion by giving a one-minute summary of the proposal, and then his/her own one-minute assessment of it. (Some preparation by the primary reviewer is needed for these opening remarks to be delivered smoothly and efficiently.)

– The secondary reviewer (also designated well in advance by the chair) makes additional comments, if (s)he has any.

– Other panelists contribute their comments to the discussion.

– The panel revises the proposal’s averaged normalized score (keeping in mind that only a change in its relative ranking will change its fate) if the discussion has changed anyone’s assessment.

– During the discussion, the primary reviewer takes note of any points that are not already reflected in the draft consensus comments, so that (s)he can revise those comments accordingly during or soon after the telecon.
Typically, the best and worst proposals will require less discussion than proposals with intermediate averaged-normalized scores, but the panel should make sure in all cases that primary reviewers have been given enough material to write coherent and constructive consensus comments.

• The ranked list of proposals that emerges from the SRP telecon should reflect panelists’ preferences after any recommendations for reduced time allocations have been taken into account. (In other words: a proposal that elicits greater SRP enthusiasm after its time request has been scaled back should have its ranking adjusted accordingly.)

• It is helpful for the SRP chair before the end of the telecon to poll the rest of the panelists about (a) any proposals they are extremely enthusiastic about, and (b) any proposals they believe should not get telescope time under any circumstances. As noted above, proposals in the latter category should be recommended for 0% allocations.

• Once each SRP completes its work, its averaged normalized scores are converted to linear-rank scores.

III. TAC Workflow

The linear-rank scores from the SRPs are on a scale from 0 (a high-ranked proposal) to 10 (a low-ranked proposal). The quartile boundaries for all proposals considered by the TAC are 2.5, 5.0 and 7.5. Understanding some aspects of how these linear-rank scores are interpreted and used by the TAC may be helpful in ensuring that your SRP’s rankings fully reflect the wishes of your panel’s members, and that you are well prepared to translate those wishes to decisions at the TAC meeting.

• Large proposals need higher rankings to get on the telescope. (Because of the impact of large proposals on the schedule, a dedicated TAC telecon will take place before the face-to-face TAC meeting, so that large proposals can be discussed and assigned priorities before the rest of the schedule is developed.)

• GBT proposals that require fixed (i.e., not dynamic) scheduling– including VLBA proposals that make use of the GBT– need higher rankings to get on the telescope.

• A VLBA proposal that requests use of the GBT or phased VLA (“Y27”) can be in trouble if its linear-rank score does not put it high up in the GBT or VLA rank-ordered proposal list. At that point, whether the proposal can deliver scientifically useful data without the use of GBT/Y27 becomes a crucial consideration.

• Triggered proposals with longer trigger timescales impact the schedule less than those with short trigger timescales, and therefore will be treated more like regular proposals when discussed by the TAC.

• For VLA proposals that request observations in configurations beyond those offered in the upcoming semester, the possible outcomes are (a) upcoming configs approved, future configs also approved; (b) upcoming configs approved, future configs put on hold for consideration by a future TAC; (c) upcoming configs approved, future configs prorated on the basis of which sources are actually observed in upcoming configs; (d) upcoming configs approved, future configs rejected; and (e) both upcoming configs and future configs rejected. Roughly
speaking (a) ! (b) ! (c) ! (d) ! (e) as the proposal grade declines. This full range of
outcomes is also possible for proposals that request only time in future configurations if and only if the principal pnvestigator (PI) is a graduate student.

• Translation of proposal ranks to scheduling priorities of A (must do), B (best effort), C (filler), and N (do not observe) depends on observing frequency, use of shared risk modes, oversubscription in particular configurations/weather bands/LST ranges, and length of sessions, in addition to the linear-rank score from the SRP.

– For observations in upcoming VLA configurations, translation to A/B/C/N scheduling priorities is driven by NRAO’s “prioritizer” software tool, which can be re-run multiple times during a TAC meeting to adjust to the evolving distribution of scheduling priorities. For any session within a given proposal, the TAC may opt to “fix” (i.e., force) a priority of A*, B*, C*, or N* (“A-star,” “B-star,” “C-star,” or “N-star”), or to “float” it (i.e., leave it to the prioritizer to sort out).

– For observations in future VLA configurations, the TAC can assign A*/B*/C*/N* grades at its discretion (or H* for holding over), based on the linear-rank score from the SRP, the degree of connectedness to observations in upcoming configurations, the relative amounts of time requested for upcoming and future configurations, and the desire not to overburden future semesters in terms of time.

• When all of the linear-rank SRP scores are released to the TAC, the merged ranked list of proposals may include some that are in your science area, but that you haven’t actually seen up to now because they were held over from a previous semester’s TAC for future (now present!) consideration. After the release of the linear-rank SRP scores but before the TAC meeting, you should familiarize yourself with the contents of such proposals and the consensus comments and technical reviews they previously received, so that you can be prepared to make
recommendations for TAC decisions.

• Tracking SRP members progress. To see which panel members have completed their conflict of interest status click on Reviews Summary tab. An (N) next to a name indicates a panel member needs to complete their conflict status review. Once all panel members have reviewed their conflict status the assignment of primary and secondary reviewers can be made.