Browse by author
Lookup NU author(s): Professor Thomas GrossORCiD
This is the final published version of a report that has been published in its final definitive form by Newcastle University, 2020.
For re-use rights please refer to the publisher's terms and conditions.
Background. Cyber security security user studies have been scrutinized in recent years on their reporting completeness for statistical inferences as well as their statistical reporting fidelity. However, other benchmarks of sound research, such as statistical power, estimates of Positive Predictive Value (PPV) and publication bias have been largely absent in the meta-research on the field. Aim. We aim to estimate the power, PPV, and publication bias distribution over an SLR-derived sample of cyber security user studies. Method. Based on an earlier published SLR of 146 cyber security user studies, we will extract correctly reported test triplets (test statistic, degrees of freedom, and p-value), the overall study sample sizes and group sizes of statistical tests, in addition to test families and multiple-comparison corrections. Based on that data we will compute effect sizes for parametric comparisons between conditions in the form of t-tests, χ 2 -tests, or one-way Ftests. We will convert all such effect sizes into Standardized Mean Differences (SMD, Hedges g) for comparisons across studies. Based on these posthoc effect size estimates, we will compute we will estimate confidence intervals as well funnel plots for the estimation of publication biases. Furthermore, we evaluate detection sensitivity, statistical power and PPV in face of parametrized a priori effect size thresholds. Anticipated Results. While we expect based on earlier results that the sample will only partially yield usable effect size estimates (and thereby estimates for further benchmarks), we anticipate that the results will offer a plethora of data characterizing the field. Anticipated Conclusions. We anticipate that the benchmarks provided will offer an empirical evidence base to inform the community how we are doing and substantiate recommendation on how to advance the field.
Author(s): Gross T
Publication type: Report
Publication status: Published
Series Title: School of Computing Technical Report Series
Year: 2020
Pages: 11
Print publication date: 01/07/2020
Acceptance date: 01/01/1900
Report Number: 1540
Institution: Newcastle University
Place Published: Newcastle upon Tyne
URL: https://www.ncl.ac.uk/media/wwwnclacuk/schoolofcomputingscience/files/trs/1540.pdf
