Normalizing Referee Ratings

Normalizing Referee Ratings

Normalizing Referee Ratings

EVLA & VLBA

Initially, the EVLA and VLBA referees use a scale of 0.0 through 9.9, with lower numbers (e.g., 1.0) being better than higher numbers (e.g., 9.0).

Then, for each referee, we assign a normalized rating of 0.0 to the best-rated proposal and 4.0 to the worst-rated proposal, and distribute the rest of that referee's proposals evenly between normalized ratings of 0.0 and 4.0. This linear-ramp method ensures that each referee's normalized median rating is 2.0, and places all referees and all referee categories on an equivalent scale. Each proposal is then assigned a mean normalized rating that is the average of the normalized ratings from the referees who evaluated the proposal. For each proposal, the Proposal Selection Committee takes consistency of the normalized ratings into account fairly strongly, not just looking at the mean normalized rating, which may be influenced by a single referee giving a much higher or lower normalized rating than all the other referees.

In their referees' reports, the proposers do not receive any normalized ratings, but do receive the rating assigned by each referee, as well as a notation of the median rating assigned by that referee.

Connect with NRAO

The NSF National Radio Astronomy Observatory and NSF Green Bank Observatory are facilities of the U.S. National Science Foundation operated under cooperative agreement by Associated Universities, Inc.

Founded in 1956, the NSF NRAO provides state-of-the-art radio telescope facilities for use by the international scientific community. NSF NRAO telescopes are open to all astronomers regardless of institutional or national affiliation. Observing time on NSF NRAO telescopes is available on a competitive basis to qualified scientists after evaluation of research proposals on the basis of scientific merit, the capability of the instruments to do the work, and the availability of the telescope during the requested time. NSF NRAO also provides both formal and informal programs in education and public outreach for teachers, students, the general public, and the media.

The U.S. National Science Foundation (U.S. NSF) is an independent federal agency created by Congress in 1950 "to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense..."

AUI collaborates with the scientific community and research sponsors to plan, build, and operate cutting-edge facilities. We cultivate excellence, deliver value, enhance education, and engage the public.