Browse by author
Lookup NU author(s): Professor Thomas GrossORCiD
This is the final published version of a report that has been published in its final definitive form by School of Computing, Newcastle University, 2019.
For re-use rights please refer to the publisher's terms and conditions.
Background. In the recent years, there has been a movement to strengthen evidence-based methods in cyber security under the flag of “science of security.” It is therefore an opportune time to take stock of the state-of-play of the field. Aim. We evaluated the state-of-play of evidence-based methods in cyber security user studies. Method. We conducted a systematic literature review study [1] of cyber security user studies from relevant venues in the years 2006–2016. We established a qualitative coding of the included sample papers with an a priori codebook of 9 indicators of reporting completeness [2]. We further extracted effect sizes for papers with parametric tests on differences between means for a quantitative analysis of effect size distribution and post-hoc power. Results. We observed that only 21% of studies replicated existing methods while 78% provided the documentation to enable future replication. With respect to internal validity, we found that only 24% provided operationalization of research questions and hypotheses. We observed that reporting did largely not adhere to APA guidelines as relevant reporting standard [3]: only 6% provided comprehensive reporting of results that would support meta-analysis. We, further, noticed a considerable reliance on p-value significance, where only 1% of the studies provided effect size estimates [4]. Of the tests selected for quantitative analysis, 80% reported a trivial to small effect, while only 28% had post-hoc power (1 − β ≥ 80%). Only 16% were still statistically significant after Bonferroni correction for the multiple-comparisons made. Conclusions. This study offers a first evidence-based reflection on the state-of-play in the field and indicates areas that could help advancing the field’s research methodology.
Author(s): Coopamootoo K, Gross T
Publication type: Report
Publication status: Published
Series Title: School of Computing Technical Report Series
Year: 2019
Pages: 24
Online publication date: 01/07/2019
Acceptance date: 02/04/2019
Report Number: 1528
Institution: School of Computing, Newcastle University
Place Published: Newcastle upon Tyne
URL: https://www.ncl.ac.uk/media/wwwnclacuk/schoolofcomputingscience/files/trs/1528.pdf
