Toggle Main Menu Toggle Search

Open Access padlockePrints

Statistical Reliability of 10 Years of Cyber Security User Studies

Lookup NU author(s): Professor Thomas Gross



This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).


In recent years, cyber security security user studies have been appraised in meta-research, mostly focusing on the completeness of their statistical inferences and the fidelity of their statistical reporting. However, estimates of the field's distribution of statistical power and its publication bias have not received much attention. In this study, we aim to estimate the effect sizes and their standard errors present as well as the implications on statistical power and publication bias. We built upon a published systematic literature review of 146 user studies in cyber security (2006--2016). We took into account 431 statistical inferences including t-, chi-squared-, r-, one-way F-tests, and Z-tests. In addition, we coded the corresponding total sample sizes, group sizes and test families. Given these data, we established the observed effect sizes and evaluated the overall publication bias. We further computed the statistical power vis-a-vis of parametrized population thresholds to gain unbiased estimates of the power distribution. We obtained a distribution of effect sizes and their conversion into comparable log odds ratios together with their standard errors. We, further, gained funnel-plot estimates of the publication bias present in the sample as well as insights into the power distribution and its consequences. Through the lenses of power and publication bias, we shed light on the statistical reliability of the studies in the field. The upshot of this introspection is practical recommendations on conducting and evaluating studies to advance the field.

Publication metadata

Author(s): Gross T

Publication type: Conference Proceedings (inc. Abstract)

Publication status: Published

Conference Name: 10th International Workshop on Socio-Technical Aspects in Security (STAST 2020)

Year of Conference: 2020

Pages: 171-190

Online publication date: 22/06/2021

Acceptance date: 14/08/2020

Date deposited: 27/11/2020

ISSN: 0302-9743

Publisher: Springer Verlag


DOI: 10.1007/978-3-030-79318-0_10

Library holdings: Search Newcastle University Library for this item

Series Title: Lecture Notes in Computer Science

ISBN: 9783030793173