Skip to Main Content Page Title
Library logo

Library Research Support: Responsible Metrics

Support for Research Staff & Research Students

Responsible Metrics @Durham University

This guide collects guidance for authors and research managers as to Durham University's approach to using publication & citation metrics  as research indicators at an institutional level, and recognising the importance of using such indicators responsibly.

  1. Overview: What is meant by Responsible Metrics?
  2. Drivers: Introduction to the Metric Tide report, DORA, Leiden Manifesto and other initiatives
  3. Activity at Durham: An Overview of what the Responsible Metrics Working Group at Durham is working towards
  4. Durham University Policy:  The University's Policy and Statement on Responsible Metrics

Responsible Metrics

What do we mean by 'Responsible Metrics'?

Various Research Indicators, or metrics, are used to 'measure' the impact of a researcher, their outputs, or the outputs of an organisational unit (research group, department, Institute or University). Many institutions use them in recruitment, probation, promotion or other processes. Many researchers use them to make important decisions, such as where to publish their work. They also form part of the calculations used in university rankings.

Durham University's Policy and Statement recognise that the use of quantitative measures have a place and a value in research assessment, but advocates for an accountable approach to their use alongside other measure, to provide a more rounded picture of 'research impact'.

Responsible Metrics at Durham University

A University statement of principles, and a University policy on responsible metrics were opened up to a University-wide consultation in September 2018:

  • Both were approved at University Executive Committee on 11th September 2018..
  • The amended policy and statement were subsequently agreed by University Senate on 16th October 2018.

Durham University Responsible Metrics Policy & Statement

Our policy and statement on the responsible metrics set out Durham's approach to the use of metrics and indicators for the purposes or research assessment.

"The University ... recognises that improperly used metrics can reinforce existing biases that impinge on the University’s mission to enhance diversity and equality amongst its academic staff, and it is important that they are used with due awareness of their limitations."

Durham University's Responsible Use of Metrics Policy

"The statement outlines the University's approach to the responsible use of metrics. It builds on the ten Principles for responsible research evaluation contained within the Leiden Manifesto for Research Metrics"

Durham University's Responsible Use of Metrics Statement

Activity at Durham University

Agreeing a Policy and Statement on the responsible use of metrics is just the start. In November 2019, University Research Management Committee and University Provost Board agreed to re-establishing the University Metrics Working Group under the remit of the Deputy to the Vice Provost (Research), with the following recommendations:

  • Group membership was to be expanded to include  academic representation from each Faculty, and Human Resources.
  • The group was intended to develop an implementation plan (including potential resource requirements) for meeting institutional obligations under University Policy and DORA.
  • The group was to map use of metrics across the institution and to report annually on implementation of policy.

DORA implementation and consultation timetable.

What is the Metric Tide?

In July 2015, The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management outlined five keys areas to inform the responsible use of metrics:

  • Robustness: basing metrics on the best possible data in terms of accuracy and scope;
  • Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment;
  • Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results;
  • Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system;
  • Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.

The report made a number of recommendations to UK HEIs, including to;

"carefully select quantitative indicators that are appropriate to their institutional aims and context ... explain why they are using these as a means to achieve particular ends ... [and] where possible, alternative indicators that support equality and diversity should be identified and included."

... to provide

"Clear communication of the rationale for selecting particular indicators, and how they will be used as a management tool... [and] consider signing up to DORA, or drawing on its principles and tailoring them to their institutional contexts."

... and to

"pay due attention to the equality and diversity implications of research assessment choices"

The Metric Tide Report

What is DORA?

The full agreement can be found here.

Two key principles for DORA institutions include:

  • being explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
  • for the purposes of research assessment, considering the value and impact of all research outputs (including datasets and software) in addition to research publications, and considering a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.

Durham University and DORA

On 16th October 2018, University Senate approved the proposal that the University became a signatory to the San Francisco Declaration on Research Assessment (DoRA), whilst "noting the importance of monographs in particular academic disciplines."

On 8th November 2018, Professor Claire Warwick, Pro-Vice-Chancellor (Research) signed DoRA on behalf of Durham University.

 

San Francisco Declaration on Research Assessment

What is the Leiden Manifesto?

To support researchers and managers, five experts led by Diana Hicks, professor in the School of Public Policy at Georgia Institute of Technology, and Paul Wouters, director of CWTS at Leiden University, proposed 10 principles for the measurement of research performance: the Leiden Manifesto for Research Metrics published as a comment in Nature.

Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, & Ismael Rafols. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429-42931. https://doi.org/10.1038/520429a 

 

 

The Leiden Manifesto for Research Metrics from Diana Hicks on Vimeo.

Further Reading

DORA

San Francisco Declaration on Research Assessment

Statement on Responsible Metrics

Durham University Statement on the Responsible use of Metrics: "The statement outlines the University's approach to the responsible use of metrics. It builds on the ten Principles for responsible research evaluation contained within the Leiden Manifesto for Research Metrics"

Policy on Responsible Metrics

Durham University Policy on Responsible Metrics: "The University ... recognises that improperly used metrics can reinforce existing biases that impinge on the University’s mission to enhance diversity and equality amongst its academic staff, and it is important that they are used with due awareness of their limitations."

undefined  undefined  undefined  undefined  undefined  undefined