The Guide provides a brief overview as to how citation data is used in some key University Rankings and Assessment Exercises.
For REF 2021, Clarivate Analytics (Web of Science) will provide citation data which will be used by 11 of the 34 sub-panels (sub-panels 1-9,11 and 16).
Details of how citation data will be used in the assessment are set out in general terms in ‘Guidance on submissions’ at paragraphs 288 to 292. Information on how these sub-panels will make use of citation data to inform their assessment is set out in the ‘Panel criteria and working methods’ at paragraphs 274 to 282. Panel guidance clearly states that "No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of outputs. No output will be privileged or disadvantaged on the basis of the publisher, where it is published or the medium of its publication." [PARA 207]
The Stern Review, published in July 2016, recommended that citation metrics could be used to support REF sub-panels in their assessment of outputs, but that "it is not currently feasible to assess research outputs in the REF using quantitative indicators alone." [PARA 75] It was also recommended that assessment panels should set out explicitly how they intended to use bibliometric data in their working methods.
For REF2014, citation data from Scopus was used by 11 of the 36 sub-panels, "as additional information about the academic significance of submitted outputs". Journal level metrics such as the Journal Impact Factor, were not supposed to be used to inform the assessment of research outputs by sub-panels.
The QS World University Rankings uses a 6-year citation data window, for publications over a 5-year period. Citation data from Elsevier's Scopus database is used in their ranking methodology. Self-citations are excluded, and citations are normalised to account for different publication and citation cultures across disciplines.
'Citations per faculty' accounts for a 20% weighting in the calculation of the overall scores.
The Times Higher Education (THE) World University Rankings use a 6-year citation data window, for publications over a 5-year period. Citation data from Elsevier's Scopus database is used in their ranking methodology. Citations are normalised to account for different publication and citation cultures across disciplines, and a fractional counting approach is used for co-authored papers to ensure all authors institutions receive at least 5% of the 'value', with those that provide the most contributors receiving a proportionally larger contribution.
'Citations (research influence)' accounts for a 30% weighting in the calculation of the overall scores.
The ShanghaiRankings' Academic Ranking of World Universities (ARWU) use publication and citation data from Web of Science in their ranking methodology.
Indicators used include a measure of the number of Highly Cited Researchers as identified in Web of Science, the number of papers published in Nature and Science journals, and a measure of publication output.
Together these indicators accounted for a 60% weighting in the calculation of the overall scores.
[In February 2020, China’s Ministry of Science and Technology and Ministry of Education announced two policy documents directing universities and research institutes not to use SCI-related metrics for university or discipline rankings]