Toggle Main Menu Toggle Search

Open Access padlockePrints

Pre-Registration: Statistical Reporting in Cyber Security User Studies

Lookup NU author(s): Professor Thomas Gross



This is the final published version of a report that has been published in its final definitive form by Newcastle University, 2020.

For re-use rights please refer to the publisher's terms and conditions.


Background. User studies in cyber security in the widest sense often rely on statistical inference to show that effects seen in (quasi-)experiments and surveys are significant, and that the null hypothesis can be rejected. In such Null Hypothesis Significance Testing (NHST), the community relies on sound reporting to ascertain the credibility of the results. Aim. We investigate the prevalence of statistical misreporting as well as the relation to publication venue and year. Method. Based on a systematic literature review of user studies in cyber security from selected venues in the 10 years 2006–2016, we will evaluate that prevalence of statistical misreporting using the R package statcheck. We will offer a systematic quantification of insufficient reporting, reporting inconsistencies and decision errors. We further conduct correlational ordinal/multinomial logistic regressions to establish the relation to publication venue and year. Anticipated Results. We anticipate descriptive statistics and graphs of the prevalence of statistical misreporting. We further intend to obtain logistic-regression models on the relation of predictors Venue and Year on coded statcheck outcomes in an correlational study. Anticipated Conclusions. We anticipate a systematic overview of statistical misreporting over time and venues, yielding an evidence-based estimate how we are performing as a field as well as what our trajectory is.statistical reporting, evidence-based methods, meta-research, pre-registration

Publication metadata

Author(s): Gross T

Publication type: Report

Publication status: Published

Series Title: School of Computing Technical Report Series

Year: 2020

Pages: 11

Print publication date: 01/02/2020

Acceptance date: 01/01/1900

Report Number: 1538

Institution: Newcastle University

Place Published: Newcastle upon Tyne