Replication: Challenges in using Data Logs to Validate Phishing Detection Ability Metrics

Abstract

The Security Behavior Observatory (SBO) is a longitudinal field-study of computer security habits that provides a novel dataset for validating computer security metrics. This paper demonstrates a new strategy for validating phishing detection ability metrics by comparing performance on a phishing signal detection task with data logs found in the SBO. We report: (1) a test of the robustness of performance on the signal detection task by replicating Canfield, Fischhoff, and Davis (2016), (2) an assessment of the task's construct validity, and (3) evaluation of its predictive validity using data logs. We find that members of the SBO sample had similar signal detection ability compared to members of the previous mTurk sample and that performance on the task correlated with the Security Behavior Intentions Scale (SeBIS). However, there was no evidence of predictive validity, as the signal detection task performance was unrelated to computer security outcomes in the SBO, including the presence of malicious software, URLs, and files. We discuss the implications of these findings and the challenges of comparing behavior on structured experimental tasks to behavior in complex real-world settings.

Meeting Name

13th Symposium on Usable Privacy and Security, SOUPS 2017 (2017: Jul. 12-14, Santa Clara, CA)

Department(s)

Engineering Management and Systems Engineering

Research Center/Lab(s)

Center for Research in Energy and Environment (CREE)

Keywords and Phrases

Computer crime; Correlation detectors; Information retrieval; Security of data; Security systems, Construct validity; Data log; Detection ability; Detection tasks; Longitudinal field study; Phishing; Phishing detections; Real world setting, Signal detection

International Standard Book Number (ISBN)

978-193197139-3

Document Type

Article - Conference proceedings

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2019 USENIX Association, All rights reserved.

Publication Date

01 Jul 2019

Link to Full Text

Share

 
COinS