In its 1995 report evaluating 147 PhD programs in physics, the National Research Council published a perceived quality Q(i) and a perceived five- year change dQ(i) for each program. The median assigned quality in the sample was 2.79. The nominal period of review was 1987-1992. Similar evaluations were published for other PhD programs in science and engineering. The quality evaluations are in fact a poll of a representative cross section of the faculty in the discipline with the respondents being provided only with a list of the current faculty of each program.
These evaluations, though admittedly subjective, are of crucial importance to many of the nation's science programs since they quantify the respect of the peer scientific community for each program. As a result they impact the good fortune of many programs, affecting the attractiveness of the program to potential students and faculty and to funding agencies and contract reviewers. Their importance lies not so much in the way they are directly used but in reflecting the overall respect for a program that is disseminated in many subconscious and sub-liminal ways in the nation's classrooms and seminar rooms. It is, therefore, surprising that more analysis of these rankings has not been done by deans and department chairmen and by departmental committees which often go to great pains attempting to relieve the symptoms of a low rank while not addressing the root causes.
There is presumably an assumption by many that these rankings are determined by factors out of the control of many departments such as the ability to finance large faculty expansions or to attract Nobel laureates. The rankings are only partially a measure of the quality and productivity of the faculty in each program. They are also affected by historical factors and by geographical and other prejudices. However, to a great extent they are a measure of the research profile of a program: How is the research program of a department distributed over the various broad sub-specialties of physics, i.e, to what extent is the department doing, in the appropriate proportions, what ,in the eyes of the academic physics community, a physics program should be doing. It is the purpose of this study to quantify these various factors to whatever extent is possible. We consider a 129 member sub-sample of the top-rated 132 physics departments omitting three departments whose faculty profiles were not given in the AIP directories consulted (AIP Directory of Physics Faculties 1989/90, in one case 1990/91). The 129 departments have 4117 regular faculty, an average of 31.9. The average faculty size among those departments above the median in NRC perceived quality (for this sample) was 41.3. If we had included faculty in separate astronomy departments this number would go to 46.
It is well known that there is a correlation between NRC perceived quality and the size of a program.
In the plot shown here the NRC perceived quality of each department is plotted
on the y axis arranged on the x axis in order of decreasing perceived quality.
The NRC perceived quality in
this sample of 129 departments varies from 1.61 to 4.91. On the other hand the
perceived quality per faculty member, shown (times 10) in the jagged curve at the
bottom is approximately flat although with large fluctuations.
The fact that the perceived quality per faculty member does not fall with
decreasing rank is an impressive vote of confidence of the peer community in
the talents of the individual faculty of the smaller schools. It, presumably, is a
result of the fact that, in the last thirty years, there
has been a very tight academic job market for physicists allowing essentially
all PhD programs to hire extremely well qualified faculty. If there is
something more to be learned from the NRC rankings, it must lie in the
significant fluctuations in per-faculty quality. To quantify this we have
made a fit to the perceived qualities of the form:Here N(i) is the total number of full time faculty (Assistant, Associate, and Full Professors) in the i'th department. No reasonable fit is found with the intercept a0 constrained to zero
(at=.091, average fractional fluctuation greater than 24.9%). We have scanned the space of a0 and at calculating, for each pair, the mean absolute value of the fractional discrepancies between the predicted quality Qt(i) and the actual NRC perceived quality Q(i).
= ave |Qt(i)/Q(i) - 1 | The minimum
(= 0.11986) corresponds to
a0 = 1.59003 and
at = 0.03950. This fit is shown in fig. 2.
The monotonically decreasing blue line is the predicted quality while the jagged maroon
line is the actual NRC assigned quality.
Sixteen departments depart
from the predicted quality by more than 2. Among
these the worst cases are a small but highly respected physics department
(University A)whose
theoretical quality based only on numbers of faculty is 47.5% below the NRC quality
and a relatively large department (University B)
whose theoretical quality in the fit is 52.8% above the NRC assigned quality.
The average fractional fluctuation is 12%. This suggests that 88% of the perceived
quality of a department is determined by the size of its faculty with only
12% on the average being determined by other factors including the excellence
of an individual faculty relative to the average. The
fit also predicts that a department can, on the average,
improve its perceived quality by .0395 by a random faculty addition
in a research specialty chosen with a weight defined by the relative
weight of that specialty in the academic community as a whole.
It is, however, also interesting to ask what weights the physics community assigns to the relative research profile of a department. Presumably the physics peer community does not assign equal weights to all branches of physics. Within physics there are some areas that are more directly related to the short term technological advance of society and others whose technological relevance lies, at best, in the distant future and whose current value is only basic knowledge. What balance does the physics community regard as optimal between these two very different goals of physics research? To probe this question we have used the information in the AIP directory of physics faculties (1989/90) to separate each program into the number of faculty in Particle Physics: Np , Nuclear Physics: Nn, Plasma Physics: Npl, Condensed Matter Physics: Ncm, and Astronomy: Na. To avoid double counting we have used the self descriptions of each faculty member to identify their primary research interest. In particle physics we have included mathematical physics, general relativity, cosmology, and cosmic ray physics. In condensed matter we have included all faculty not assigned to one of the other sub-divisions. Again we do not count faculty members in separate Astronomy departments. We have defined the theoretical quality by finding the best fit to the formula
Again the best fit is defined by seeking the values of the a coefficients
that minimize the absolute value of the average
fractional discrepancy .
The best fit (=.11249) is given by the coefficients
| a0 | 1.7435 |
|---|---|
| ap | 0.06514 |
| an | .03517 |
| apl | .0416 |
| acm | .02276 |
| aa | .03765 |
Although the mean fractional discrepancy in this fit is not significantly better than in the fit of fig. 2, it clearly shows that not all areas of physics are equally weighted in determining the rank of a department. One must emphasize, of course, that these weights are not at all a measure of the intrinsic importance of a research area in society as a whole but only in academia as judged by physics faculty. The fit quantifies to some extent the well known property of the highly ranked departments as emphasizing the less applied areas of physics over the more applied. The current fit correctly places four of the NRC top ten in the top ten versus only three for the fit of fig. 2. In addition, for this fit there are only 11 departments whose fractional discrepancy is twice the mean versus 16 for the fit of fig. 2. The fit still, however, seriously underestimates the quality of University A and overestimates that of University B relative to the NRC assigned qualities. One should note perhaps that University B's physics department is a department of physics and applied physics and it could be that they are subconsciously penalized for this since it has a large and not severely unbalanced faculty. It is clear on examining the makeup of the highly ranked departments that applied physics, important as it is to society as a whole, has a strictly limited place in primary physics departments.
The fit of fig. 3 shows that one can predict the NRC quality of a physics program to within 11.2% on the average by a simple formula depending only on the numbers of faculty in various research areas. Presumably therefore, other factors including the excellence of a physics faculty above or below the average determines the fluctuations. It is well known that the highly ranked departments are more likely to have nobel laureates on the faculty and this should certainly play a role in their assigned quality. To test this we have counted the number of nobel laureates nnobel in each of the departments. We find 41 nobel laureates (since 1960) in 17 departments in the NRC survey although a few are recently deceased. To test for a dependence on this factor we have added a term to the theoretical quality formula writing
Seeking again the values of the a coefficients that minimize the mean absolute fractional discrepancy we find the fit of Fig. 4.
and the best
fit values of the a
coefficients given in the table below. Again the blue line is the
predicted quality in predicted rank order while the purple line
gives the actual NRC perceived quality.| a0 | 1.8016 |
|---|---|
| ap | 0.05607 |
| an | 0.02992 |
| apl | 0.04189 |
| acm | 0.01891 |
| aa | 0.03535 |
| aNobel | 0.30585 |
It is reassuring that consideration of Nobel laureates on a department's faculty decreases the fluctuations evident in Fig. 3 among the highly ranked departments. The mean absolute fractional discrepancy between the predicted and actual perceived qualities is 0.09985 in the fit of Fig. 4. The fit suggests that each Nobel laureate on a department's faculty is worth more than five of any other appointments in its effect on rank. The perceived quality of only eight physics departments deviates by more than twice the average of the previous fit (Fig. 3). The current fit places eight of the NRC top ten departments in the top ten. In addition, the perceived quality of University A which was notoriously hard to fit in the previous, purely programatic formulae is now adequately predicted. The worst underestimate of perceived quality is now only 25.4% while the overestimate of the NRC quality assigned to University B has decreased to 41.2%.
Finally, we have noted that a somewhat better fit can be obtained by replacing the faculty numbers in the last fit, including the number of Nobel laureates, by their square roots.
Npart
+anuc
Again the blue line gives the predicted quality and the now somewhat less
jagged maroon line gives the NRC assigned qualities. The mean absolute
fractional discrepancy between the predicted and actual perceived
qualities is now