MUP Metrics – Center for Measuring University Performance

Research and Development

Source: Higher Education Research and Development Survey

Each year, the National Science Foundation (NSF) collects data from hundreds of academic institutions on expenditures for research and development in science and engineering fields and classifies them by source of funds (e.g., federal government, state and local government, industry, etc.). These data are the primary source of information on academic research and development (R&D) expenditures in the United States. Included in this survey are all activities specifically organized to produce research outcomes that are separately budgeted and accounted for. This “organized research” may be funded by an external agency or organization (“sponsored research”) or by a separately budgeted organizational unit within the institution (“university research”). This report excludes activities sponsored by external agencies that involve instruction, training (except training in research techniques, which is considered organized research), and health service, community service or extension service projects.

All Federally Funded Research Labs (FFRLs) are excluded from these academic expenditures data, including the following: Jet Propulsion Laboratory (California Institute of Technology); Los Alamos National Lab, Lawrence Livermore Lab, Lawrence Berkeley Lab (University of California); Software Engineering Institute (Carnegie Mellon); Argonne National Laboratory (University of Chicago); National Astronomy and Ionospher Center (Cornell); Ames Laboratory (Iowa State University); Lincoln Laboratory (MIT); Plasma Physics Lab (Princeton); and SLAC National Accelerator Laboratory (Stanford). The NSF data no longer classify the Applied Physics Lab (APL) at Johns Hopkins as an FFRL, but federal funds support the vast majority of research conducted there. The APL makes up more than one-half of Johns Hopkins’ total and federal R&D expenditures.

While inconsistencies in reporting (known and unknown) do exist here, as in any survey of this type, problems arise mostly when one breaks out the data by source of funds. NSF expects institutions to use year-end accounting records to complete this report, and there are nationally recognized accounting guidelines for higher education institutions. However, there are also countless variations in institutional policy that determine whether the university reports a particular expenditure as coming from one source or another, or possibly not counted at all. Take federal formula funds for agriculture (e.g., Hatch-McIntire, Smith-Lever) as an example. We conducted an informal survey of the appropriate institutions in the Association of American Universities (AAU) and found that two out of eleven land grants did not include any of these federal funds in their 1997 NSF data, while others included all or some of these monies. Because these funds make up a very small percentage of the total research expenditures in any given year, the impact on our total research rankings is slight. The agriculture formula funds will have a somewhat greater, but still small, impact on the federal research rankings. NSF notes, “An increasing number of institutions have linkages with industry and foundations via subcontracts, thus complicating the identification of funding source. In addition, institutional policy may determine whether unrestricted state support is reported as state or as institutional funds.” 1

We believe that the reporting inconsistencies in the data are relatively minor when using the total research expenditures and the federal research expenditures component. Federal and state government audits of institutional accounting make deceptive practices highly unlikely, even though these entities do not audit the NSF data directly. NSF goes to great lengths to verify the accuracy of the data, especially federal expenditure data—checking them against several other federal agencies that collect the same or similar information. In fact, all major federal agencies and their subdivisions submit data to NSF identifying research obligations to universities each year. Historically, the NSF data have tracked very closely the data reported by universities.2 Further, for their National Patterns of R&D Resources series, NSF prefers to use the figures reported by the performers of the work (that is, academic institutions, industry, nonprofits) because they believe that the performers are in the best position to accurately report these expenditures.

In some sections of this report, these expenditure data are deflated to constant 1983 dollars to show real change over time. While NSF uses the Gross Domestic Price (GDP) implicit price deflator in its reports on federal trends in research, we use the Higher Education Price Index (HEPI) because of its narrower focus. Originally developed by Research Associates of Washington and currently managed by Commonfund Institute, the HEPI illustrates the effect of inflation on college and university operations.3 In contrast, the GDP implicit price deflator is based on change in the entire U.S. economy and, as noted by NSF itself, “[its] use more accurately reflects an ‘opportunity cost’ criterion rather than measure of cost chanages in doing research. That is, the GDP deflator, when applied to R&D expenditure or funding data, reflects [the value of R&D in terms of the amount of other goods and services that could have been purchased with the same amount of money].,”4

The research trend data always reflect the most recent published data available because NSF allows institutions to submit revised figures for up to two years. Each year, NSF reports data for the current year as well as for the previous seven years. NSF’s published nationwide totals for academic R&D expenditures will not always match the corresponding totals in this study due to NSF’s sampling procedures for smaller or non-reporting institutions. In some years, rather than identifying the institutions individually, NSF provides one aggregate figure for all sampled institutions.
__________________________
1. Academic R&D Expenditures, FY 2009: Technical Notes (On-line: http://www.nsf.gov/statistics/nsf11313/content.cfm?pub_id=4065&id=3 )

2. National Patterns of R&D Resources, 1998: Technical Notes (On-line: http://www.nsf.gov/sbe/srs/nsf99335/appa.htm )

3. About HEPI, Commonfund Institute (On-line: http://www.commonfund.org/CommonfundInstitute/HEPI)

4. National Patterns of Research Development Resources, 2003: Technical Notes (On-line: http://www.nsf.gov/statistics/nsf05308/appa.htm )

Endowment Assets

Source: NACUBO Endowment Study

Institutions report the market value of their endowment assets as of June 30 to three different sources, and they quite often use three different values. For this project, we use the National Association of College and University Business Officers (NACUBO) Endowment Study because of NACUBO’s long history of reporting endowments of higher education institutions, their emphasis on using audited financial statements, and their focus on net assets (i.e., includes returns on investments and excludes investment fees and other withdrawals). NACUBO conducts its study annually and reports the results each February in the Chronicle of Higher Education.

Another source for endowment assets is the Council for Aid to Education’s (CAE) annual Voluntary Support of Education (VSE) survey, cosponsored by the Council for Advancement and Support of Education (CASE) and the National Association of Independent Schools. The VSE survey is useful as a secondary resource because it provides more single-campus data than the other two sources. For those institutions that report a system-wide total to NACUBO, we often use the VSE data to calculate a campus’ percentage contribution to the entire system, applying that factor to the NACUBO figure. In other cases, we may substitute the VSE figure when the institution indicates that this is a good data source.

The National Center for Education Statistics (NCES) Integrated Postsecondary Education Data System (IPEDS) Finance Survey also collects information on endowment assets; but these data are available much later. In our inaugural report of The Top American Research Universities in 2000, we noted the wide variation in the reporting of endowment market value between all three sources. An examination of the 1997 endowment figures showed only one university (University of North Carolina at Chapel Hill) had submitted the same figure to each of the three organizations. In a more recent study of major research universities, we found about one-third of the all institutions report identical figures but just seven universities in our over $40 million federal research group. In the earlier study, we found that endowment assets reported to IPEDS tended to be lower than NACUBO or VSE data, but this is no longer true. In general, the greater the endowment the likelihood that the figures reported to the three sources will vary. Both studies found no consistent pattern to explain reporting variations among the institutions.

Annual Giving

Source: Council for Aid to Education’s Voluntary, Support of Education (VSE) Survey

The Council for Aid to Education (CAE), an independent subsidiary of RAND, has produced the Voluntary Support of Education (VSE) Survey since 1986. The annual giving data include all contributions actually received during the institution’s fiscal year in the form of cash, securities, company products, and other property from alumni, non-alumni individuals, corporations, foundations, religious organizations, and other groups. Not included in the totals are public funds, earnings on investments held by the institution, and unfulfilled pledges.

CAE’s VSE Data Miner service, available online, provides 10 years of data for all participating institutions (more than 1,600 universities and colleges). Although this is a subscription-based service and requires a user ID and password, limited access is available online at
[http://www.cae.org/vse].

National Academy Members

Source: National Academy of Sciences, National Academy of Engineering, and National Academy of Medicine (formerly Institute of Medicine) membership online directories

One of the highest honors that academic faculty can receive is membership in the National Academy of Sciences (NAS), the National Academy of Engineering (NAE), or the National Academy of Medicine (NAM). All three are private, nonprofit organizations and serve as advisors to the federal government on science, technology, and medicine. Nominated and voted on by active members, newly elected members of these organizations receive life terms. Individuals elected to membership come from all sectors—academia, industry, government, and not-for-profit agencies or organizations. Member election dates are in February (NAE), April (NAS), and October (NAM).

The data collected for these rankings use active or emeritus members at their affiliated work institution, as reported in the online membership directories. In all cases, we were able to determine the specific campus for individual members. We re-check institutional affiliation annually to account for established members who have changed employers or whose membership is no longer active.

Faculty Awards in the Arts, Humanities, Science, Engineering and Health

Source: Directories or web-based listings for multiple agencies or organizations.

For this category, we collect data from several prominent grant and fellowship programs in the arts, humanities, science, engineering, and health fields.

Included in this measure are:

• American Council of Learned Societies (ACLS) Fellows
• Beckman Young Investigators
• Burroughs Wellcome Fund Career Awards
• Cottrell Scholars
• Fulbright American Scholars
• Getty Scholars in Residence
• Guggenheim Fellows
• Howard Hughes Medical Institute Investigators
• Lasker Medical Research Awards
• MacArthur Foundation Fellows
• Andrew W. Mellon Foundation Distinguished Achievement Awards
• National Endowment for the Humanities (NEH) Fellows
• National Humanities Center Fellows
• National Institutes of Health (NIH) MERIT (R37)
• National Medal of Science and National Medal of Technology
• NSF CAREER awards (excluding those who are also PECASE winners)
• Newberry Library Long-term Fellows
• Pew Scholars in Biomedicine
• Presidential Early Career Awards for Scientists and Engineers (PECASE)
• Robert Wood Johnson Policy Fellows
• Searle Scholars
• Sloan Research Fellows
• US Secretary of Agriculture Honor Awards
• Woodrow Wilson Fellows

While the vast majority of these programs clearly identify a particular campus, in a few instances we used the institution’s web-based directories to determine the correct campus.

Doctorates Awarded

Source: NCES IPEDS Completions Survey.

Each year, universities report their degrees awarded to the NCES in the IPEDS Completions Survey. IPEDS provides straightforward instructions for reporting doctoral degrees awarded, and we do not find any inconsistencies in reporting among the universities included in our rankings. IPEDS asks each institution to identify the number of research doctorates degrees awarded between July 1 and June 30.

Most institutions in our study submit degree data by campus or offer doctoral degrees solely or primarily at the main campus.

In addition to doctorate degrees, The MUP Center presents degrees awarded at other levels—associate’s, bachelor’s, master’s, and professional degrees—in the Student Characteristics table.

Postdoctoral Appointees

Source: NSF/Division of Science Resource Statistics (SRS) Survey of Graduate Students and Postdoctorates in Science and Engineering.

Each year, NSF and NIH collect data from all institutions offering graduate programs in any science, engineering, or health field. The Survey of Graduate Students and Postdoctorates in Science and Engineering (also called the Graduate Student Survey or GSS) reflects graduate enrollment and postdoctoral employment at the beginning of the academic year. Postdoctorates are defined in the GSS as “individuals with science and engineering PhD’s, MD’s, DDS’s or DVM’s and foreign degrees equivalent to U.S. doctorates who devote their primary effort to their own research training through research activities or study in the department under temporary appointments carrying no academic rank.” The definition excludes clinical fellows and those in medical residency training programs unless the primary purpose of their appointment is for research training under a senior mentor.

In the methodological notes for this survey,5 NSF indicates that it verifies the data with the institutional coordinator when dramatic year-to-year fluctuations are noted. In addition, efforts have been made recently by NSF to reduce the number of institutions that assigned a zero when they should have recorded it as non-response. Some historical data may have a zero when it should be blank (for non-response) even though The MUP Center has been trying to fix these errors for the past several years.

Although each doctorate-granting campus submits data separately, NSF often aggregates them in its published reports. In all cases, we obtain the single-campus data for these schools directly from NSF.

__________________________
5. Survey Methodology: Survey of Graduate Students and Postdoctorates
in Science and Engineering (On-line: http://www.nsf.gov/statistics/srvygradpostdoc/

SAT Scores

Source: NCES IPEDS Survey, SAT and ACT Scores.

IPEDS reports the 25th and 75th percentiles for verbal and quantitative SAT I scores for most institutions in our study. For our measure, we calculated the median of that range. Some institutions report the ACT instead of the SAT to IPEDS and some report both. We select the test which has the greatest percentage of students reporting. To convert ACT scores, we use a conversion table provided by The College Board to generate a comparable SAT equivalent score. 6 When an institution submits neither an SAT nor ACT score, we substitute data from other national data sources.
__________________________
6.ACT and SAT Concordance Tables, November 6, 2009 (On-line: http://www.collegeboard.org/research/html/rn_indx.html)

Other Measures of Undergraduate Quality

National Merit and Achievement Scholars

Source: The National Merit Scholarship Corporation Annual Report

The National Merit Scholarship Corporation (NMSC) is an independent, nonprofit organization that awards scholarships to the nation’s outstanding high school seniors based on their academic achievement, qualifying test scores, high school principal and counselor recommendations, and their activities, interests, and goals. The NMSC names approximately 15,000 National Merit Finalists each February. Of these, about 8,000 will receive a National Merit $2,500 Scholarship, a corporate-sponsored scholarship, or a college-sponsored scholarship.

National Achievement Scholars are selected and funded in a similar fashion and represent the nation’s outstanding African-American students. Ideally, the National Hispanic Scholars Program should also be included in this category, but it does not track the enrollment of its scholarship winners. Should it do so in the future, we will include these students in The MUP Center‘s data. In this study, Merit and Achievement scholarships are credited to the main campus if the NMSC Annual Report does not indicate a branch campus.

While the number of National Merit and National Achievement award winners in the entering class provides an indication of the attractiveness of a university’s undergraduate program to outstanding students, it is also an indicator that is sensitive to institutional policies on financial aid. Because the number of Merit Scholars is small, relatively small changes in institutional aid policies can have a significant impact on the number of National Merit Scholars enrolling in institutions. The average SAT score provides a broader-based and more reliable measure of overall undergraduate quality; for those reasons, we prefer the SAT scores to the number of National Merit and Achievement Scholars as an indicator of undergraduate quality.

Institutional Characteristics

Medical Schools

Source: NCES IPEDS Completions Survey, MD degrees awarded.

Although the IPEDS Institutional Characteristics Survey does have a “medical” field that indicates whether an institution grants a medical degree, we choose not to use its data because it includes medical degrees in Veterinary Medicine, Dentistry, and other professional health-related fields. For our measure, we determined whether a particular campus awarded any MD degrees during the academic year. If the institution did not submit any data to IPEDS for that year, we then looked at whether it was accredited by the American Medical Association to determine whether the institution has a medical school.

Land Grant Institutions

Source: National Association of State Universities and Land Grant Colleges.

The first Morrill Act in 1862 appropriated federal funds for universities to provide agricultural and technical education to their citizens. A second Morrill Act in 1890 expanded eligibility to include several historically black colleges and universities, and in 1994 several Native American tribal colleges were recognized as land grant institutions. Today, there is at least one land grant institution in each state and U.S. territory and in the District of Columbia. Of the 105 institutions, most are public universities. Federal land grant institutions receive both federal and state dollars in support of their agricultural and extension activities.

While land grant status technically applies to some university systems, such as the University of California and the University of Nebraska, for our study we designate as land grant institutions only those schools that actually perform that function (e.g., UC-Berkeley, UC-Davis, UC-Riverside, Nebraska-Lincoln). In these cases, the land grant field will identify whether an institution is part of a system-wide land grant and whether the vast majority of the activity occurs on that campus. For example, UC-Davis is coded as “Yes-System” while UCLA is coded as “No-System.” We consider the 1890 institutions as land grant institutions, but we identify them separately because they do not perform extension activities.

Research Focus

Source: NSF Higher Education Research and Development Survey
In addition to reporting expenditure data by source of funds, NSF identifies in what major disciplines the money is expended. In the Research by Discipline table we provide the proportion of total and federal expenditures in each discipline for those institutions with more than $40 million in federal research. These data are useful for developing groups of similar institutions for peer analysis.

The Institutional Characteristics table provides a summary measure of an institution’s research strength and concentration based on these discipline-level expenditures. Universities with 95-100% of their federal research dollars spent in one particular discipline are coded as “all.” We identify institutions with 75-94% in one area as “heavy” and label those with 50-74% of their expenditures concentrated as “strong.” Other universities with 25-49% in one or more disciplines we describe as “moderate.” A few institutions have expenditures distributed fairly evenly across the disciplines; those we code as “mixed.”

In some cases, where an institution reports as a multi-campus entity, we made adjustments to break out the discipline-level expenditure data by single campus. Typically, this involved moving all or a portion of the life sciences expenditures to the health or medical center campus. IPEDS fall enrollment and graduate degrees by discipline data also were used to help in this effort.

While these data offer some insight as to the research structure of a university, their usefulness is limited. For example, we may be tempted to use the life sciences as a surrogate for medical research, but we must remember that they also include agricultural and biological sciences. Further, the growing trend toward multidisciplinary and interdisciplinary projects may make it more difficult for universities to accurately reflect expenditures by discipline or sub-discipline.The MUP Center chooses not to break out these sub-disciplines because the data are increasingly prone to error as further adjustments are made.

Student Characteristics

Fall Enrollment

Source: NCES IPEDS Fall Enrollment Survey.

Each November, institutions report their current fall headcount enrollment to the IPEDS Fall Enrollment Survey. Enrollment figures include both degree seeking and non-degree seeking students. The MUP Center provides the headcount enrollment by level as presented by IPEDS, along with the percentage of those attending part-time. Graduate students include those seeking specialist degrees in engineering and education and first professional students which includes those seeking degrees in medical fields, such as Chiropractic, Dentistry, Medicine, Optometry, Osteopathic Medicine, Pharmacy, Podiatry, and Veterinary Medicine, as well as those seeking degrees in Law and Theology.

Each campus in our study submits enrollment data by campus, except for the few institutions identified in each year’s Top American Research University publication. Because this is an informational item and not one of The MUP Center‘s nine quality measures, we did not attempt to adjust these figures.

Federal Research with and without Medical School Research

AAMC Federal Research

Source: Association of American Medical Colleges

The Association of American Medical Colleges collects data on federally sponsored research at medical colleges through on the Liaison Committee on Medical Education (LCME) Part I-A, Annual Financial Questionnaire. We calculate each medical school’s federal R&D by summing the recorded dollars and a portion of the relative administrative costs. We exclude the not-recorded dollars.

Data Notes

The raw data used for The Top American Research Universities project—obtained from federal agencies and national organizations—often contain information on single campus institutions, multiple campus institutions, and state university systems, without clearly identifying the distinctions. This makes national comparisons difficult and unreliable.

To increase the validity and usefulness of these data, The Center for Measuring University Performance adjusts the original reported figures, when necessary, to ensure that all data represent the strength of a single-campus institution. The MUP Center bases its adjustments on information gathered from the reporting agency or from the university itself. In cases where the published data represent a single campus, we do not adjust the data. When the data represent more than one campus, we first attempt to obtain a figure directly from NSF (for research expenditures and postdoctorates), from the institution itself, or from the university system office that submitted the data. If unavailable from those primary sources, we use an estimated or substitute figure derived from information found on the institution’s web site. As a last resort, we use prior-year data as a substitute.

If the institution provides an estimate representing at least 97% of the originally published figure, we credit the full amount to the main campus. Otherwise, we use the estimate provided by the institution.

The MUP Center does not adjust the private university data because of multi-campus or system-wide reporting. We treat all private universities in this study as single campus institutions, because while some may have multiple campuses, they generally are in or around a single city and considered an integral part of the main campus. Furthermore, private institutions generally do not break out their data by regional, branch, or affiliated campus, as often happens with public institutions.

The tables found in the back of each annual Top American Research Universities publication outline the various adjustments or substitutions that were made to the original data that particular reporting year for those institutions with more than $40 ($20 in prior years) million in federal research. Institutions are listed alphabetically and include both private and public universities.