Replication: Challenges in using Data Logs to Validate Phishing Detection Ability Metrics
The Security Behavior Observatory (SBO) is a longitudinal field-study of computer security habits that provides a novel dataset for validating computer security metrics. This paper demonstrates a new strategy for validating phishing detection ability metrics by comparing performance on a phishing signal detection task with data logs found in the SBO. We report: (1) a test of the robustness of performance on the signal detection task by replicating Canfield, Fischhoff, and Davis (2016), (2) an assessment of the task's construct validity, and (3) evaluation of its predictive validity using data logs. We find that members of the SBO sample had similar signal detection ability compared to members of the previous mTurk sample and that performance on the task correlated with the Security Behavior Intentions Scale (SeBIS). However, there was no evidence of predictive validity, as the signal detection task performance was unrelated to computer security outcomes in the SBO, including the presence of malicious software, URLs, and files. We discuss the implications of these findings and the challenges of comparing behavior on structured experimental tasks to behavior in complex real-world settings.
C. I. Canfield et al., "Replication: Challenges in using Data Logs to Validate Phishing Detection Ability Metrics," Proceedings of the 13th Symposium on Usable Privacy and Security (2017, Santa Clara, CA), pp. 271-284, USENIX Association, Jul 2019.
13th Symposium on Usable Privacy and Security, SOUPS 2017 (2017: Jul. 12-14, Santa Clara, CA)
Engineering Management and Systems Engineering
Center for Research in Energy and Environment (CREE)
Keywords and Phrases
Computer crime; Correlation detectors; Information retrieval; Security of data; Security systems, Construct validity; Data log; Detection ability; Detection tasks; Longitudinal field study; Phishing; Phishing detections; Real world setting, Signal detection
International Standard Book Number (ISBN)
Article - Conference proceedings
© 2019 USENIX Association, All rights reserved.
01 Jul 2019