Toggle Main Menu Toggle Search

Open Access padlockePrints

A Systematic Evaluation of Evidence-Based Methods in Cyber Security User Studies

Lookup NU author(s): Professor Thomas GrossORCiD



This is the final published version of a report that has been published in its final definitive form by School of Computing, Newcastle University, 2019.

For re-use rights please refer to the publisher's terms and conditions.


Background. In the recent years, there has been a movement to strengthen evidence-based methods in cyber security under the flag of “science of security.” It is therefore an opportune time to take stock of the state-of-play of the field. Aim. We evaluated the state-of-play of evidence-based methods in cyber security user studies. Method. We conducted a systematic literature review study [1] of cyber security user studies from relevant venues in the years 2006–2016. We established a qualitative coding of the included sample papers with an a priori codebook of 9 indicators of reporting completeness [2]. We further extracted effect sizes for papers with parametric tests on differences between means for a quantitative analysis of effect size distribution and post-hoc power. Results. We observed that only 21% of studies replicated existing methods while 78% provided the documentation to enable future replication. With respect to internal validity, we found that only 24% provided operationalization of research questions and hypotheses. We observed that reporting did largely not adhere to APA guidelines as relevant reporting standard [3]: only 6% provided comprehensive reporting of results that would support meta-analysis. We, further, noticed a considerable reliance on p-value significance, where only 1% of the studies provided effect size estimates [4]. Of the tests selected for quantitative analysis, 80% reported a trivial to small effect, while only 28% had post-hoc power (1 − β ≥ 80%). Only 16% were still statistically significant after Bonferroni correction for the multiple-comparisons made. Conclusions. This study offers a first evidence-based reflection on the state-of-play in the field and indicates areas that could help advancing the field’s research methodology.

Publication metadata

Author(s): Coopamootoo K, Gross T

Publication type: Report

Publication status: Published

Series Title: School of Computing Technical Report Series

Year: 2019

Pages: 24

Online publication date: 01/07/2019

Acceptance date: 02/04/2019

Report Number: 1528

Institution: School of Computing, Newcastle University

Place Published: Newcastle upon Tyne