The Productivity and Impact of The Leukemia & Lymphoma Society Scholar Program: The Apparent Positive Effect of Peer Review

A study was conducted to compare the “productivity” of a cohort of research grant applicants selected by peer review to be scholars of The Leukemia Society of America (now The Leukemia & Lymphoma Society) with a matched cohort of applicants not so selected during the period 1981 to 1990. One hundred and twenty-four scholars and 124 nonfunded applicants were studied. Two bibliometric variables and their derivatives were examined from the Institute of Scientific Information database: the number of papers published and the number of citations to those papers. Published papers were measured through December 31, 1999, and citation counts to these papers through December 31, 2000. Scholars published 10,301 papers through the period of observation and nonfunded applicants published 6442 papers. Scholarsʹ papers were cited 419,798 times, whereas nonfunded applicantsʹ papers were cited 245,586 times. The mean citations per paper were 52 for scholars and 38 for nonfunded applicants. The papers published per scholar, citations per scholar, and citations per paper per scholar were significantly greater than the corresponding measures for nonfunded applicants (P< 0.0001 in each case). Scholarʹs papers were cited 30% more often, whereas nonfunded applicants were cited 10% more frequently, than a comparison group of scientists publishing in the same journal in the same year. High-impact papers, e.g., papers that were cited more than 200 times, were nearly three times as frequent among scholars (494 papers) as among nonfunded applicants (173 papers). This difference was highly significant. The good (better than baseline) performance of nonfunded applicants may be a reflection of self-selection among the applicant pool for this competitive award; the more productive performance of the scholars is probably the result of the selection decisions made during the peer-review process.