Toggle Main Menu Toggle Search

Open Access padlockePrints

A new approach to impact case study analytics

Lookup NU author(s): Dr Jiajie Zhang, Professor Paul WatsonORCiD, Professor Barry Hodgson



This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).


The 2014 Research Excellence Framework (REF) assessed the quality of university research in the UK. 20% of the assessment was allocated according to peer review of the impact of research, reflecting the growing importance of impact in UK government policy. Beyond academia, impact is defined as a change or benefit to the economy, society, culture, public policy or services, health, the environment, or quality of life. Each institution submitted a set of four-page impact case studies. These are predominantly free-form descriptions and evidences of the impact of study. Numerous analyses of these case studies have been conducted, but they have utilised either qualitative methods or primary forms of text searching. These approaches have limitations, including the time required to manually analyse the data and the frequently inferior quality of the answers provided by applying computational analysis to unstructured, context-less free text data. This paper describes a new system to address these problems. At its core is a structured, queryable representation of the case study data. We describe the ontology design used to structure the information and how semantic web related technologies are used to store and query the data. Experiments show that this gives two significant advantages over existing techniques: improved accuracy in question answering and the capability to answer a broader range of questions, by integrating data from external sources. Then we investigate whether machine learning can predict each case study’s grade using this structured representation. The results provide accurate predictions for computer science impact case studies.

Publication metadata

Author(s): Zhang J, Watson P, Hodgson JB

Publication type: Article

Publication status: Published

Journal: Data & Policy

Year: 2022

Volume: 4

Print publication date: 28/09/2022

Online publication date: 28/09/2022

Acceptance date: 30/06/2022

Date deposited: 28/09/2022

ISSN (electronic): 2632-3249

Publisher: Cambridge University Press


DOI: 10.1017/dap.2022.21


Altmetrics provided by Altmetric