Toggle Main Menu Toggle Search

Open Access padlockePrints

Pre-Registration: Power, PPV and Publication Bias of Cyber Security User Studies

Lookup NU author(s): Professor Thomas Gross



This is the final published version of a report that has been published in its final definitive form by Newcastle University, 2020.

For re-use rights please refer to the publisher's terms and conditions.


Background. Cyber security security user studies have been scrutinized in recent years on their reporting completeness for statistical inferences as well as their statistical reporting fidelity. However, other benchmarks of sound research, such as statistical power, estimates of Positive Predictive Value (PPV) and publication bias have been largely absent in the meta-research on the field. Aim. We aim to estimate the power, PPV, and publication bias distribution over an SLR-derived sample of cyber security user studies. Method. Based on an earlier published SLR of 146 cyber security user studies, we will extract correctly reported test triplets (test statistic, degrees of freedom, and p-value), the overall study sample sizes and group sizes of statistical tests, in addition to test families and multiple-comparison corrections. Based on that data we will compute effect sizes for parametric comparisons between conditions in the form of t-tests, χ 2 -tests, or one-way Ftests. We will convert all such effect sizes into Standardized Mean Differences (SMD, Hedges g) for comparisons across studies. Based on these posthoc effect size estimates, we will compute we will estimate confidence intervals as well funnel plots for the estimation of publication biases. Furthermore, we evaluate detection sensitivity, statistical power and PPV in face of parametrized a priori effect size thresholds. Anticipated Results. While we expect based on earlier results that the sample will only partially yield usable effect size estimates (and thereby estimates for further benchmarks), we anticipate that the results will offer a plethora of data characterizing the field. Anticipated Conclusions. We anticipate that the benchmarks provided will offer an empirical evidence base to inform the community how we are doing and substantiate recommendation on how to advance the field.

Publication metadata

Author(s): Gross T

Publication type: Report

Publication status: Published

Series Title: School of Computing Technical Report Series

Year: 2020

Pages: 11

Print publication date: 01/07/2020

Acceptance date: 01/01/1900

Report Number: 1540

Institution: Newcastle University

Place Published: Newcastle upon Tyne