Guides: Library Research Support: Responsible Metrics

What do we mean by 'Responsible Metrics'?

Various Research Indicators, or metrics, are used to 'measure' the impact of a researcher, their outputs, or the outputs of an organisational unit (research group, department, Institute or University). Many institutions use them in recruitment, probation, promotion or other processes. Many researchers use them to make important decisions, such as where to publish their work. They also form part of the calculations used in university rankings.

Durham University's Policy and Statement recognise that the use of quantitative measures have a place and a value in research assessment, but advocates for an accountable approach to their use alongside other measure, to provide a more rounded picture of 'research impact'.

The Metric Tide Report (2015)

What is the Metric Tide?

In July 2015, The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management outlined five keys areas to inform the responsible use of metrics:

Robustness: basing metrics on the best possible data in terms of accuracy and scope;
Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment;
Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results;
Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system;
Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.

The report made a number of recommendations to UK HEIs, including to;

"carefully select quantitative indicators that are appropriate to their institutional aims and context ... explain why they are using these as a means to achieve particular ends ... [and] where possible, alternative indicators that support equality and diversity should be identified and included."

... to provide

"Clear communication of the rationale for selecting particular indicators, and how they will be used as a management tool... [and] consider signing up to DORA, or drawing on its principles and tailoring them to their institutional contexts."

... and to

"pay due attention to the equality and diversity implications of research assessment choices"

Durham University Policy and Statement

A University statement of principles, and a University policy on responsible metrics were opened up to a University-wide consultation which closed on September 21st 2018:

Both were approved at University Executive Committee on 11th September 2018..
The amended policy and statement were subsequently agreed by University Senate on 16th October 2018.

Agreeing a Policy and Statement on the responsible use of metrics is just the start. In November 2019, University Research Management Committee and University Provost Board agreed to re-establishing the University Metrics Working Group under the remit of the Deputy to the Vice Provost (Research), with the following recommendations:

Group membership was to be expanded to include academic representation from each Faculty, and Human Resources.
The group was intended to develop an implementation plan (including potential resource requirements) for meeting institutional obligations under University Policy and DORA.
The group was to map use of metrics across the institution and to report annually on implementation of policy.

In April/May 2020, a survey of current use of metrics across University departments was completed.

DORA Implementation and Consultation Timetable DORA Implementation and Consultation Timetable

The San Francisco Declaration on Research Assessment (DORA) (2012)

What is DORA?

The full agreement can be found here.

Two key principles for DORA institutions include:

being explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
for the purposes of research assessment, considering the value and impact of all research outputs (including datasets and software) in addition to research publications, and considering a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.

Durham University and DORA

On 16^th October 2018, University Senate approved the proposal that the University became a signatory to the San Francisco Declaration on Research Assessment (DoRA), whilst "noting the importance of monographs in particular academic disciplines."

On 8^th November 2018, Professor Claire Warwick, Pro-Vice-Chancellor (Research) signed DoRA on behalf of Durham University.

The Leiden Manifesto (2015)

What is the Leiden Manifesto?

To support researchers and managers, five experts led by Diana Hicks, professor in the School of Public Policy at Georgia Institute of Technology, and Paul Wouters, director of CWTS at Leiden University, proposed 10 principles for the measurement of research performance: the Leiden Manifesto for Research Metrics published as a comment in Nature.

Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, & Ismael Rafols. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429-42931. https://doi.org/10.1038/520429a