Browse by author
Lookup NU author(s): Professor Thomas Gross
This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
In recent years, cyber security security user studies have been appraised in meta-research, mostly focusing on the completeness of their statistical inferences and the fidelity of their statistical reporting. However, estimates of the field's distribution of statistical power and its publication bias have not received much attention. In this study, we aim to estimate the effect sizes and their standard errors present as well as the implications on statistical power and publication bias. We built upon a published systematic literature review of 146 user studies in cyber security (2006--2016). We took into account 431 statistical inferences including t-, chi-squared-, r-, one-way F-tests, and Z-tests. In addition, we coded the corresponding total sample sizes, group sizes and test families. Given these data, we established the observed effect sizes and evaluated the overall publication bias. We further computed the statistical power vis-a-vis of parametrized population thresholds to gain unbiased estimates of the power distribution. We obtained a distribution of effect sizes and their conversion into comparable log odds ratios together with their standard errors. We, further, gained funnel-plot estimates of the publication bias present in the sample as well as insights into the power distribution and its consequences. Through the lenses of power and publication bias, we shed light on the statistical reliability of the studies in the field. The upshot of this introspection is practical recommendations on conducting and evaluating studies to advance the field.
Author(s): Gross T
Publication type: Conference Proceedings (inc. Abstract)
Publication status: Published
Conference Name: 10th International Workshop on Socio-Technical Aspects in Security (STAST 2020)
Year of Conference: 2020
Online publication date: 22/06/2021
Acceptance date: 14/08/2020
Date deposited: 27/11/2020
Publisher: Springer Verlag
Library holdings: Search Newcastle University Library for this item
Series Title: Lecture Notes in Computer Science