Toggle Main Menu Toggle Search

Open Access padlockePrints

Measuring expert performance at manually classifying domain entities under upper ontology classes

Lookup NU author(s): Dr Phillip Lord



This is the authors' accepted manuscript of an article that has been published in its final definitive form by Elsevier BV, 2019.

For re-use rights please refer to the publisher's terms and conditions.


© 2018 Background: Classifying entities in domain ontologies under upper ontology classes is a recommended task in ontology engineering to facilitate semantic interoperability and modelling consistency. Integrating upper ontologies this way is difficult and, despite emerging automated methods, remains a largely manual task. Problem: Little is known about how well experts perform at upper ontology integration. To develop methodological and tool support, we first need to understand how well experts do this task. We designed a study to measure the performance of human experts at manually classifying classes in a general knowledge domain ontology with entities in the Basic Formal Ontology (BFO), an upper ontology used widely in the biomedical domain. Method: We recruited 8 BFO experts and asked them to classify 46 commonly known entities from the domain of travel with BFO entities. The tasks were delivered as part of a web survey. Results: We find that, even for a well understood general knowledge domain such as travel, the results of the manual classification tasks are highly inconsistent: the mean agreement of the participants with the classification decisions of an expert panel was only 51%, and the inter-rater agreement using Fleiss’ Kappa was merely moderate (0.52). We further follow up on the conjecture that the degree of classification consistency is correlated with the frequency the respective BFO classes are used in practice and find that this is only true to a moderate degree (0.52, Pearson). Conclusions: We conclude that manually classifying domain entities under upper ontology classes is indeed very difficult to do correctly. Given the importance of the task and the high degree of inconsistent classifications we encountered, we further conclude that it is necessary to improve the methodological framework surrounding the manual integration of domain and upper ontologies.

Publication metadata

Author(s): Stevens R, Lord P, Malone J, Matentzoglu N

Publication type: Article

Publication status: Published

Journal: Journal of Web Semantics

Year: 2019

Volume: 57

Print publication date: 01/08/2019

Online publication date: 06/09/2018

Acceptance date: 26/08/2018

Date deposited: 12/11/2018

ISSN (print): 1570-8268

ISSN (electronic): 1873-7749

Publisher: Elsevier BV


DOI: 10.1016/j.websem.2018.08.004


Altmetrics provided by Altmetric