Toggle Main Menu Toggle Search

Open Access padlockePrints

The Newcastle University research output collection, currently available on ePrints, will shortly be moving to a new open repository platform, Figshare. To prepare for the data migration we have paused adding new content to ePrints, and will resume once the new repository is launched. During this time you will continue to have access to ePrints (but no new content will appear). We will share updates here when available.

A Systematic Evaluation of Evidence-Based Methods in Cyber Security User Studies

Lookup NU author(s): Professor Thomas GrossORCiD

Downloads

Licence

This is the final published version of a report that has been published in its final definitive form by School of Computing, Newcastle University, 2019.

For re-use rights please refer to the publisher's terms and conditions.

Abstract

Background. In the recent years, there has been a movement to strengthen evidence-based methods in cyber security under the flag of “science of security.” It is therefore an opportune time to take stock of the state-of-play of the field. Aim. We evaluated the state-of-play of evidence-based methods in cyber security user studies. Method. We conducted a systematic literature review study [1] of cyber security user studies from relevant venues in the years 2006–2016. We established a qualitative coding of the included sample papers with an a priori codebook of 9 indicators of reporting completeness [2]. We further extracted effect sizes for papers with parametric tests on differences between means for a quantitative analysis of effect size distribution and post-hoc power. Results. We observed that only 21% of studies replicated existing methods while 78% provided the documentation to enable future replication. With respect to internal validity, we found that only 24% provided operationalization of research questions and hypotheses. We observed that reporting did largely not adhere to APA guidelines as relevant reporting standard [3]: only 6% provided comprehensive reporting of results that would support meta-analysis. We, further, noticed a considerable reliance on p-value significance, where only 1% of the studies provided effect size estimates [4]. Of the tests selected for quantitative analysis, 80% reported a trivial to small effect, while only 28% had post-hoc power (1 − β ≥ 80%). Only 16% were still statistically significant after Bonferroni correction for the multiple-comparisons made. Conclusions. This study offers a first evidence-based reflection on the state-of-play in the field and indicates areas that could help advancing the field’s research methodology.

Publication metadata

Author(s): Coopamootoo K, Gross T

Publication type: Report

Publication status: Published

Series Title: School of Computing Technical Report Series

Year: 2019

Pages: 24

Online publication date: 01/07/2019

Acceptance date: 02/04/2019

Report Number: 1528

Institution: School of Computing, Newcastle University

Place Published: Newcastle upon Tyne

URL: https://www.ncl.ac.uk/media/wwwnclacuk/schoolofcomputingscience/files/trs/1528.pdf

Share