The Guide provides a brief overview as to how citation data is used in some key University Rankings and Assessment Exercises.
For further information on the REF please contact Research & Innovation Services, or for University Rankings, the Strategic Planning Office.
For REF 2021, Clarivate Analytics (Web of Science) will provide citation data was used by 11 of the 34 sub-panels (sub-panels 1-9,11 and 16).
Details of how citation data was used in the assessment are set out in general terms in ‘Guidance on submissions’ at paragraphs 288 to 292. Information on how these sub-panels would make use of citation data to inform their assessment is set out in the ‘Panel criteria and working methods’ at paragraphs 274 to 282. Panel guidance clearly states that "No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of outputs. No output will be privileged or disadvantaged on the basis of the publisher, where it is published or the medium of its publication." [PARA 207]
The Stern Review, published in July 2016, recommended that citation metrics could be used to support REF sub-panels in their assessment of outputs, but that "it is not currently feasible to assess research outputs in the REF using quantitative indicators alone." [PARA 75] It was also recommended that assessment panels should set out explicitly how they intended to use bibliometric data in their working methods.
For REF2014, citation data from Scopus was used by 11 of the 36 sub-panels, "as additional information about the academic significance of submitted outputs". Journal level metrics such as the Journal Impact Factor, were not supposed to be used to inform the assessment of research outputs by sub-panels.
The QS World University Rankings uses a 6-year citation data window, for publications over a 5-year period. Citation data from Elsevier's Scopus database is used in their ranking methodology. Self-citations are excluded, and citations are normalised to account for different publication and citation cultures across disciplines.
'Citations per faculty' accounts for a 20% weighting in the calculation of the overall scores.
The Times Higher Education (THE) World University Rankings use a 6-year citation data window, for publications over a 5-year period. Citation data from Elsevier's Scopus database is used in their ranking methodology. Citations are normalised to account for different publication and citation cultures across disciplines, and a fractional counting approach is used for co-authored papers to ensure all authors institutions receive at least 5% of the 'value', with those that provide the most contributors receiving a proportionally larger contribution.
'Citations (research influence)' accounts for a 30% weighting in the calculation of the overall scores.
The ShanghaiRankings' Academic Ranking of World Universities (ARWU) use publication and citation data from Web of Science in their ranking methodology.
Indicators used include a measure of the number of Highly Cited Researchers as identified in Web of Science, the number of papers published in Nature and Science journals, and a measure of publication output.
Together these indicators accounted for a 60% weighting in the calculation of the overall scores.
[In February 2020, China’s Ministry of Science and Technology and Ministry of Education announced two policy documents directing universities and research institutes not to use SCI-related metrics for university or discipline rankings]
The CWTS Leiden Rankings use publication, collaboration and citation data from Web of Science in their ranking methodology.
Indicators used include the number and proportion of a university's publications that, compared with other publications in the same field and from the same year, are most frequently cited.
The methodology also uses data from Unpaywall and Web of Science to produce a ranking of university's based on the proportion of their output which can be identified as available in an open access format.
In the CWTS Leiden Ranking 2022, Durham University places 150th Globally as an indication of the proportion of output which is cited in the top 10% most cited output, and places 4th in the UK and 5th in the World as an indication of the proportion of output which we make available in an open access form.
