Browse by author
Lookup NU author(s): Professor Thomas GrossORCiD
This is the final published version of a report that has been published in its final definitive form by Newcastle University, 2020.
For re-use rights please refer to the publisher's terms and conditions.
Background. User studies in cyber security in the widest sense often rely on statistical inference to show that effects seen in (quasi-)experiments and surveys are significant, and that the null hypothesis can be rejected. In such Null Hypothesis Significance Testing (NHST), the community relies on sound reporting to ascertain the credibility of the results. Aim. We investigate the prevalence of statistical misreporting as well as the relation to publication venue and year. Method. Based on a systematic literature review of user studies in cyber security from selected venues in the 10 years 2006–2016, we will evaluate that prevalence of statistical misreporting using the R package statcheck. We will offer a systematic quantification of insufficient reporting, reporting inconsistencies and decision errors. We further conduct correlational ordinal/multinomial logistic regressions to establish the relation to publication venue and year. Anticipated Results. We anticipate descriptive statistics and graphs of the prevalence of statistical misreporting. We further intend to obtain logistic-regression models on the relation of predictors Venue and Year on coded statcheck outcomes in an correlational study. Anticipated Conclusions. We anticipate a systematic overview of statistical misreporting over time and venues, yielding an evidence-based estimate how we are performing as a field as well as what our trajectory is.statistical reporting, evidence-based methods, meta-research, pre-registration
Author(s): Gross T
Publication type: Report
Publication status: Published
Series Title: School of Computing Technical Report Series
Year: 2020
Pages: 11
Print publication date: 01/02/2020
Acceptance date: 01/01/1900
Report Number: 1538
Institution: Newcastle University
Place Published: Newcastle upon Tyne
URL: https://www.ncl.ac.uk/media/wwwnclacuk/schoolofcomputingscience/files/trs/1538.pdf
