Technical Reviews

This document provides instructions for entering your technical reviews of proposals using the Proposal Submission Tool (PST; Section 2). Review guidelines are provided in Section 3. NRAO policies on conflicts of interests and confidentiality are fully covered in Section 4. 

Entering Technical Reviews Using the PST

I. Login through the NRAO portal at my.nrao.edu

You will enter the PST at the Dashboard, which allows you to perform a number of functions related to proposal preparation and review, access documentation, and modify your user profile.

As a technical reviewer, seven tabs should be visible below the National Radio Astronomy Observatory banner:

• Dashboard

• Proposals

• Reviews

• Data Processing

• Obs Prep

• Helpdesk

• Profile

II. Click on the Reviews tab. If the Reviews tab is not visible, please click the Helpdesk tab and submit a ticket immediately to the Proposal Review department.

III. Upon clicking the Reviews tab you will see a new page that has the same banner and primary tabs across the top. It should have one secondary tab below the primary tabs labeled My Reviews. Brief instructions are provided to technical reviewers in the grey box. More detailed instructions are provided here.

The remainder of the page displays all of the proposals assigned to you for technical review.

• Column 1 gives the PST proposal ID.

• Column 2 gives the legacy ID.

• Column 3 gives the proposal title.

• Column 4 gives the proposal authors.

• Column 5 indicates whether the technical reviewer is conflicted on a proposal.

• Column 6 gives the reason for a given conflict.

The PST automatically captures one type of conflict: the reviewer is the PI or a co-I on a proposal under consideration by the reviewer. If you believe any of the conflicts identified by the PST are in error, please submit a Helpdesk ticket to the Proposal Review department.

Please examine all other proposal titles and author lists for which the PST has not identified a conflict (unchecked box) and self-declare any conflicts based on this information. Whether or not a technical reviewer self-declares a conflict is left largely to their discretion. The main consideration is whether or not the reviewer believes that he/she can make a fair and impartial assessment of the technical soundness or feasibility of the proposed observations. If not, the reviewer should declare themself conflicted. A technical review should remain completely neutral regarding the scientific merit of the proposal.

Once all conflicts have been identified click on Accept conflicts status in the upper right of the page. This takes you to the main review management page.

IV. The banner, primary, and secondary tabs on the review management page remain as before. Below the secondary My Reviews tab is a field and Search button on the left side that can be used to find proposals of interest. To the right are four buttons labeled Export Technical Reviews, Import Technical Reviews, Print, and Help. The Help button provides general help on the use of the User Portal for proposal preparation. The other three buttons are discussed below.

To the left of the page, below the Search field and button is an Options box that allows you to filter the proposals displayed on the page by telescope. Additional filtering options may be added in future releases of the PST.

The rest of the page contains 7 columns.

• Column 1, labeled “All”, allows one to print all proposals that the reviewer is not conflicted on. To do so, click the “All” button. The check boxes of all proposals should automatically be checked. Then click the Print button to the upper right of the page. A window is opened that allows you to select which portions of the proposals to print: the cover sheet, the science justification, the student support application, and/or the student dissertation plan. After making your selections, click on “Print” in the window to print all proposals to a pdf file that can, in turn, be saved to a file on your computer or sent to your printer. One can also use the check boxes in Column 1 to manually check one or more proposals to print via the Print button.

• Column 2 lists the PST proposal ID and also displays two icons. By clicking on the column’s heading, one can sort the proposals in numerical order. By clicking on the PST proposal ID in column 2, highlighted in blue, one can view the proposal. From that proposal’s page:

1. One can use the Options column to the left to navigate to different sections of the proposal (e.g., general, authors, scientific justification, etc.),

2. One can also enter the scientific reviews page by clicking on Reviews in the Options column.

3. One can also directly view other (unconflicted) proposals by clicking on the ID number in the Options column. Clicking My Reviews at the top of the Options column returns you to the main proposal management page.

4. A printer icon below the secondary My Reviews tab allows one to print the proposal in its entirety to a printer or to your computer’s hard disk.

5. Clicking on the pen-and-paper icon in column 2 takes you to the technical review page for that proposal (see below); the printer icon in column 2 allows you to send a pdf copy of the proposal to a printer or to your computer’s hard disk.

• Column 3 shows the legacy ID.

• Column 4 gives the proposal title.

• Column 5 gives the name of the PI on the proposal.

• Column 6 indicates whether the reviewer is conflicted on the proposal. NB: if the reviewer is conflicted, they should not be able to view, print, or review the proposal in question.

• Column 7 indicates the status of the review, initially labeled Enter Review.

V. We now offer two ways to enter your technical reviews. We describe each in Sections A and B.

A. One can use the PST to enter the reviews one by one. To enter a technical review for a proposal on which you are not conflicted, you can click Enter Review in Column 7 or the pen-and-paper icon in column 2. A less direct path is to click on the proposal ID in column 2 and then click on Reviews under the Options bar from the proposal-viewing page.

The Technical Reviews page summarizes the review for the proposal of interest.

• Column 1 gives your name.

• Column 2 shows the review comments for the proposer.

• Column 3 shows comments for the Science Review Panel (SRP) and Time Allocation Committee (TAC). These comments are not seen by the proposer.

• Column 4 shows review status information.

Clicking on your name in column 1 allows you to enter your technical reviews and, if necessary, comments for the SRP/TAC. Please type comments regarding the technical merit of the proposal into the box in column 2. If there are comments that the SRP/TAC should see, but that the proposer should not, please enter those into the box in column 3 (it's ok to leave column 3 blank).  Beginning for the 19A semester, there are 3 "headings" to move cues/comments to.  For details about doing this, see below.

The fifth column on the review page allows you to Save or Cancel a review in progress. If you wish to defer completion of the review until later click Save. If you wish to return the review to a pristine state or to its status following the previous Save, please click Cancel.

If the review is complete, please check the Completed box in column 4 and then click on Save in column 5. If you later find that you have made a mistake in checking the Completed box for a particular proposal, please submit a Helpdesk ticket to the Proposal Review department.

B. The second method to enter your reviews is to write them in a text file external to the PST, and then use the Import Technical Reviews button (on the My Reviews page) to import them into the PST. This method is advantageous to those who wish to work offline. While preparation of the external text file is straightforward, it must nevertheless conform to some simple conventions. The easiest way to proceed, therefore, is to create an external reviews template by clicking the Export Technical Reviews button, which produces a simple text file listing all proposals on which you are not conflicted. Here is an example of a template produced by clicking the Export Technical Reviews button:

* Each entry of the file should follow the format:
* ###
* Proposal ID%%%
* Review (text)%%%
* Comments For TAC (text)%%%
* Completed Status (Yes/No)%%%
* ###
GBT/19A-331%%%
Dates:
Observing Time:
Mapping:
RFI Considerations:
Overhead:
Joint External Proposal:
Novel Considerations:
Pulsar Considerations:
Min/Max LST:

Technical review comments:
----------------------------------------------

Technical information supplied was satisfactory:
----------------------------------------------

Not applicable:
----------------------------------------------

%%%
%%%
No%%%
###
GMVA/18B-046%%%
Stations Requested:
Sessions:
Correlator Setup:
Integration Time:
Imaging Considerations:
Phase Referencing:
Min/Max GST:
Special Technical Considerations:

Technical review comments:
----------------------------------------------

Technical information supplied was satisfactory:
----------------------------------------------

Not applicable:
----------------------------------------------
%%%
%%%
No%%%
###VLA/21A-005%%%
Combined Telescopes:
Array Configuration:
Future Semesters:
Scheduling Restrictions:
Min/Max LST:
Receivers Requested:
Correlator Setup:
Mosaic Requirements:
Sensitivity:
Integration Time:
Dump Time:
Imaging Considerations:
Polarimetric Considerations:
RFI Considerations:
Joint External Proposal:
Special technical/RSRO:
Technical review comments:
----------------------------------------------
Technical information supplied was satisfactory:
----------------------------------------------
Not applicable:
----------------------------------------------
%%%
%%%
No%%%

and so on.

This template can be saved to a file on your computer that you can edit without being logged into the PST. Alternatively, one can create the external text file manually, making sure that it conforms to the simple conventions outlined below.

The first six lines are comments that start with a "*" symbol and list the four entry fields which are separated by "%%%".  The proposals are separated by "###".   This format allows you to use multiple lines for each field; the pound and percent symbols may be used in your text (assuming you do not have three consecutive symbols).   In the default file there are no comments (or cues) to the "Comments to the SRP/TAC" field and thus there are two "%%%"  expressions back-to-back.

The first entry is the proposal ID (e.g., GBT/19A-331).  The second entry is the technical review.  This entry has been populated with cues,  where each cue corresponds to a text box in the Technical Justification page. We have added the 3 sections at the bottom, so reviewers can move the cues and comments to any of the 3 sections.   A reviewer would move the entry (with comments) to "Technical Review comments:", or just move the cue to either "Technical information supplied was satisfactory" or  "Not applicable".   An example of the review would look after completion, for the VLA, is shown here (with technical review comments in blue):

VLA/21A-005%%%

Technical review comments:
----------------------------------------------

Scheduling Restrictions:   The proposers say that the target is nighttime for the D configuration, but it is really a daytime object.
Receivers Requested:  The proposers request the L band receiver, but the proposal is about observing the 22 GHz water line.
Integration Time:  The integration time is calculated with the subband bandwidth of 4 MHz, not the 62.5 kHz channel width in the correlator setup.  Thus the rms noise for the integration time given will ~4 mJy/beam rather than the proposers' value of 0.5 mJy/beam.

Technical information supplied was satisfactory:
----------------------------------------------

Combined Telescopes:
Array Configuration:
Future Semesters:
Min/Max LST:
Correlator Setup:
Sensitivity:
Dump Time:
Joint External Proposal:

Not applicable:
----------------------------------------------

Mosaic Requirements:
Imaging Considerations:
Polarimetric Considerations:
RFI Considerations:
Special technical/RSRO:

%%%
%%%
No
%%% 

and so on.

The third entry is used for comments to the SRP/TAC.  The last entry is a simple “Yes” or “No” to indicate whether you have completed the review. If you import a proposal with the entry set to “Yes”, it will not be possible to over-write or otherwise change the review of that particular proposal thereafter. If the entry is set to “No”, it will be possible to change or over-write.

To import the reviews, return to the My Reviews page and click on the Import Technical Reviews button. At the top of the page, a text pane will appear. To the left of the text pane are three buttons: “Browse”, “Upload”, and “Cancel”. Clicking on the “Browse” button opens a browse window on your computer that allows you to identify the external text file with your technical reviews. The directory path and name of the file then appear in the text pane. Click the “Upload” button to import the file into the PST. The PST will report the number of reviews successfully uploaded and whether any problems were encountered. Click “Cancel” if you decide not to import the reviews.

For those reviews that you indicated were incomplete (a “No” in the fourth field), you can either continue to revise them in the text file and upload again later; or you can complete them in the PST. It is also possible to re-export your reviews for further editing outside the PST, and then re-import them. Again, once you have indicated that you have completed a review with a “Yes” in the fourth entry field, no further changes to the review will be allowed by the PST. However, as noted in Section V.A, if you find that you have made a mistake by indicating that a particular proposal has been completed, please submit a Helpdesk ticket to the Proposal Review department.

VI. Once all (unconflicted) reviews are completed, either by manually indicating that this is the case proposal by proposal in the PST, or by setting the fourth field of all proposals to “Yes” in an external review field that is then imported, the PST will acknowledge the fact and thank you!

Guidelines for Technical Reviewers

Here are some guidelines on technical reviews of proposals for the VLA, the VLBA, the GMVA and the GBT. Please be as careful as possible in your reviews. It is very important that a proposal not be downgraded (or rejected) by the SRP or TAC because of a faulty technical review. Please be as succinct as possible, providing only as much detail as needed. No comments should be made upon the science proposed except as it applies to the technical requirements for that science. Finally, please ensure that your comments conform to professional standards; i.e., avoid gratuitous remarks or personal opinions.

That said, the science reviewers and the TAC have requested that the technical reviews be more uniform, as much as can be done given that the technical reviews are for different telescopes. To help achieve this goal (and to focus the proposers' technical ideas), we have implemented into the PST a “Technical Justification” page. Currently, the proposers are required to fill out text boxes and upload .pdf (VLA) or .png (VLBA, GMVA) files for the exposure calculators, as well as VLA spectroscopic setup information (RCT-proposing).

The capabilities advertised for the VLA, the VLBA, and the GBT can be found in the call for proposals:

https://science.nrao.edu/observing/call-for-proposals/ for the VLA, VLBA/HSA, and GMVA,  and

https://greenbankobservatory.org/category/portal/gbt/proposing/  for the GBT.

It is very important that technical reviewers familiarize themselves with what is offered in the Call for Proposals. Not knowing or understanding what is offered for that cycle's Telescope Resources is one of the main reasons for incorrect technical reviews.

For technical reviews for all telescopes, cues are already populated in the upper part of the technical review box.  In the lower part, there are 3 sections, labeled "Technical review comments:", "Technical information supplied was satisfactory:", and "Not applicable:".  The idea for technical reviews is to move all the cues (with comments for the first section) into one of those 3 sections.   Also, there is a button on the technical review page which allows one  to view the Technical Justification from the proposal.

Very Large Array

For the most part, the items discussed below are taken from the VLA Technical Justification Page. Most are self-explanatory; also the Technical Justification page gives further instructions:

1. Combination of data: If requested, what other configurations or telescopes will the proposed data be combined with ?

2. Array configuration: Why is the specific configuration needed ? Is the angular extent of the source and the largest angular size included ? Please check whether Resources, Technical Justification, and Scientific Justification specify the same array configuration.

3. Describe the use of subarrays:

   • the number of subarrays and distribution of antennas between them,
   • a summary of the frequency bands and correlator configuration (as you will explain and detail further below) and observing modes used in each of the subarrays,
   • as well as any other specific details that would be of interest related to the subarray observing.
4. Future semesters:   Have the proposers explained why observations in a future semester (i.e. one not included in the call) are needed ?  Why cannot the future semester observations be requested in a future call ?

5. Scheduling restrictions: Daytime/nighttime; target elevation; required dates; number of LST passes available (for Large proposals) ?

6. Minimum/Maximum LST range for sessions.  If the range is different than what the PST calculates, the proposer is asked to justify why.

7. Receivers requested: Which receivers are needed and why.

8. Correlator setups: 3-bit / 8-bit (bandwidth), etc.

9. Mosaic Requirements: Is mosaicking needed ? How is it to be done ?

10. Required Sensitivity: What sensitivity does the science require ? Include frequency or velocity width assumed.

11. What on-source integration time is needed to achieve the required sensitivity ? Include considerations such as source confusion, RFI, self-noise, overhead (if different than that given by the exposure calculator). Upload calculator graphics, for each resource.

12. Correlator dump time, data rate, total volume: Dump time and data rate can be taken from resource or RCT-proposing. Total volume follows.

13. Imaging issues: Wide fractional bandwidths, ionosphere, nearby strong sources, complex source structure. Can target be self-calibrated ? How does obsersver plan to ameliorate these issues ?

14. Polarimetric observations: parallactic angle coverage needed ?

15. RFI problems: In particular satellite belt, etc.

16.  For joint external proposals, provide technical information for the external telescope (HST, Chandra, Swift, etc.)

17. Special technical considerations: RSRO uses this space to discuss who fills the residency requirements, etc.

Note that we specifically do not ask the technical reviewers to tell us what would be necessary (in terms of time, bandwidth, etc.) for the project to be successful. We ask the technical reviewers to answer the question: What will the proposers get for what they proposed, and how does that relate to the science that is proposed? In other words, we do not ask technical reviewers to re-write the technical sections of proposals. This is one of the most important parts of the VLA technical review as far as it pertains to proposals getting downgraded or rejected by the SRP or TAC. There are some, but very few, cases when this comment does not apply. If there is a definite need to mention the amount of time required to obtain the requested noise level put this in “Comments to the SRP/TAC”. Please heed this warning about technical reviews.

Trigger criteria should be summarized in the “Comments to the SRP/TAC”.

For requests for joint external proposals, the request should be copied to "Comments to the SRP/TAC."

Technical reviewers should consult the web page about proposing:

https://science.nrao.edu/facilities/vla/proposing/

The VLA exposure calculator toolcan be found here:

https://science.nrao.edu/facilities/vla/docs/manuals/propvla/determining/source

RCT-proposing instructions can be found here:

https://science.nrao.edu/facilities/vla/docs/manuals/propvla/proposer-generated-resource

and the VLA configuration plans and proposal deadlines can be found at

https://science.nrao.edu/facilities/evla/proposing/configpropdeadlines

If you need to consult on a technical review, please contact Amy Mioduszewski.

Very Long Baseline Array

For the most part, the items discussed below are taken from the VLBA Technical Justification Page. Most are self-explanatory; also the Technical Justification page gives further instruction.

1. Stations requested: Minimum number, optional and required stations. Justify why HSA is needed for science. Verify that all stations can sample/record the same observing mode.

2. Future semesters: Explain why future semesters are needed; include information on how critical the future observations are to the overall project, and why a proposal for the future semester cannot be submitted at a future deadline.

3. Receivers requested: What receivers are required for the science case. Dual polarization ?

4. Scheduling considerations: weather, preferred dates, a series of observations with specific spacing, minimum scheduling block length: What are minimum scheduling blocks ? If 24-hr blocks, can break points be inserted ?

5. Correlator setup: Use of pulsar processing, multiple phase centers, multiple correlator passes, wide-field phase centers ? Justify number of phase centers if > 100.

6. Self-calibration: Can target be self-calibrated ? Phase referencing – estimate flux density. Extra time (VLA or VLBA) needed for calibrators ?  If possible, identify phase reference sources.

7. Sensitivity requirements: What is the sensitivity needed for science goal ? What is the total time needed to achieve this sensitivity ? Exposure calculator graphics uploaded. Assume minimum number of stations, include u-v coverage considerations, etc. Baseline sensitivity for non-imaging experiments.

8. Integration time needed: What is the total time needed to achieve the sensitivity in number 6 ? Exposure calculator graphics uploaded. Assume minimum number of stations, include u-v coverage considerations, etc.

9. Imaging considerations: Potential imaging issues. For example, Sensitivity/dynamic range limited; wide fractional bandwidths, ionosphere, nearby strong sources, complex source structure, etc.

10. Polarization observations: VLA observations needed for EVPA of calibrators ? How soon after (or before) VLBA observations ?

11. Accurate flux density calibration: Specify extra calibration steps to be taken if very precise flux calibration is needed.

12. Correlator output data rate: calculate size of output FITS files.
13. Joint External Proposal:  For joint external proposals, use this space to provide technical information about the external telescope (HST, Swift, Chandra, etc.)
14. Minimum/Maximum GST for sessions:  If the range is different than what the PST calculates, the proposer is asked to justify why.
15. Special/other/RSRO technical considerations.  VLBA RSRO proposals use this section to describe resident observer, and their technical expertise.

Note that we specifically do not ask the technical reviewers to tell us what would be necessary (in terms of time, bandwidth, etc.) for the project to be successful. We ask the technical reviewers to answer the question: What will the proposers get for what they proposed, and how does that relate to the science that is proposed? In other words, we do not ask technical reviewers to re-write the technical sections of proposals. This is very likely the single most important part of the technical review as far as it pertains to proposals getting downgraded or rejected by the SRP or TAC. There are some, but very few, cases when this comment does not apply. If there is a definite need to mention the amount of time required to obtain the requested noise level put this in “Comments to the SRP/TAC”. Please heed this warning about technical reviews.

Trigger criteria should be summarized in the “Comments to the SRP/TAC”.

Joint proposal requests should be copied to "Comments to the SRP/TAC."

The EVN Observation Planner may be found at

https://planobs.jive.eu/

The old EVN calculator may be found at

https://services.jive.eu/evn-calculator/cgi-bin/EVNcalc.pl

Guidelines on using these (including the sometimes confusing difference between our basebands and EVN subbands) can be found at:

https://science.nrao.edu/facilities/vlba/docs/manuals/propvlba/exposure-and-overhead

If you need to consult on a technical review, please contact Robert Minchin.

High Sensitivity Array

HSA proposals are a sub-category of VLBA proposals and are generally treated in the same manner, with the same technical justification inputs. Proposals requesting the GBT will have a technical review from Green Bank as well as one from the VLBA, but VLBA technical reviewers should check that the technical requirements of the phased VLA (Y27) are addressed for proposals requesting this. In particular, please pay attention to the area of the phased beam (the same as the VLA synthesized beam in that configuration, not the primary beam) and the cycle time required. Both of these are particularly important at higher frequencies and in more extended configurations. 

If you need to consult on a technical review, please contact Robert Minchin.

Global Millimeter VLBI Array

We have implemented a Technical Justification for the GMVA.

1. Stations requested: There is a standard set of stations for the GMVA, but “Other” stations could be requested (or the entire standard set may not be required).

2. Session constraints: The GMVA is scheduled in 2 sessions per year.

3. Correlator setup: All GMVA projects are correlated at the Bonn correlator.

4. Sensitivity/Integration Time:  What is the science justification for the needed sensitivity; for some projects, u-v coverage is more important than sensitivity.

5. Imaging:  Potential imaging issues expected (e.g. due to wide fractional bandwidths, ionosphere, nearby strong sources, comples source structure, etc.)

6. Phase referencing:  Is phase referencing needed, or can self-cal be used ?  If possible, identify phase-reference sources.

7. Minimum/maximum GST:  If the range is different than what the PST calculates, the proposer is asked to justify why.
8. Special technical considerations: any other technical requirements/issues.

For technical reviews of GMVA proposals please concentrate on the technical capabilities of the VLBA, GBT,  and the DiFX correlator (if it is requested);  we also rely on you to give whatever technical detail you think is appropriate for those proposals. GMVA proposals requesting the VLBA and the GBT receive separate technical reviews for each telescope.

If you need to consult on a technical review, please contact Robert Minchin.

Green Bank Telescope

Please provide short, succinct responses to the following topics. Use full sentences so that the PIs have the impression that you have read and understood the proposal.  You should not use the templates provided in the PST as they are guidelines for potential topics to cover in your review.  Please make sure to include all relevant information if you find an issue or believe that the requested time is not satisfactory.

1. Ensure that the requested setup(s) are both possible and will work as expected by the proposers. If not then please provide a potential alternative if possible.

2. Please check that the sensitivity calculation is roughly correct and, if not, state what the correct value is for the proposed observations and possible implications (e.g., the proposers need four times the requested time allocation in order to meet their science goals). Note that the sensitivity calculator is designed to only be correct to within 5-10%.

3. For continuum observations make sure that they are have taken into account 1/f noise.
4. For maps make sure that they have calculated the time for a map correctly.  Double check that they did not confuse time per pixel with time ber beam.  Also make sure that sampling times are within limits and that telescope motion and accelerations are OK.

5. Can different hardware or different observing techniques be used to make the observations more successful? Present these as alternatives that the proposers could employ and state why they may be better than the proposed hardware or technique.

6. Is RFI a potential issue for the proposed observations and did the proposers present a plan to deal with this RFI?

7. For Large Proposals you must also consider if the data management plan is sufficient. Large proposals should address data reduction, data storage and data releases.

8. If extra scientific staff support or experience beyond what the project team currently possesses will be needed in order for the proposers to successfully carry out their observations then please note this in the comments to the SRP/TAC.

Here is an example of a good technical reivew:

The PIs wish to observe HI using the GBT spectrometer. They use a standard setup and a standard observing technique. Their estimated observing time is correct. They did not address the potential of RFI in the observations. However, this should not be a problem for HI observations.

Here is an example of a bad technical review for the same project:

Setup: Good

Observing technique: OK

Time estimate: Good

RFI: not addressed

NRAO Conflicts of Interest and Confidentiality Polices

Please review the NRAO Conflicts of Interest and Confidentiality Polices here.