How Users Investigate Phishing Emails that Lack Traditional Phishing Cues

Abstract

Phishing is still one of the prevalent threats targeting private persons and organizations. Current teaching best practices often advocate cue-based investigation methods. Previous research primarily confronted participants with phishing emails showing such indicators to assess the success of different education measures. Our large-scale mixed-methods study challenges the behavior of 4,729 participants with four phishing emails that lack technical cues. The phishing emails concerned entirely fictitious entities and were directed at participants in their private lives, recruited from the online education platform openHPI. For our analysis, we apply the human-in-the-loop model for interaction with phishing content to investigate participant behavior when their learned best practices for detection fail. The primary indicator of enhanced phishing resiliency observed in our study was awareness of missing context to the supposed entity. Such context is often successfully enhanced by web searches, significantly contributing to decreased phishing susceptibility.

Appendices

A Appendix: Survey Instrument

The survey questions are translated from German for publication in this manuscript. The following sub-sections layout the survey instrument used to obtain the responses presented throughout the manuscript.

1.1 A.1 Demography

• Q1: Please enter your email address.

• Q2: How old are you?

• Q3: Which gender would you associate yourself with?

• Q4: Which is your highest level of education?

• Q5: In which industry are you currently working? (Multi-Select among primary industry groups according to UN ISIC Rev.4 [31])

1.2 A.2 Phishing Emails and Reactions

• Q6: In the past 4 months, we have sent 4 phishing emails as part of this study. In the following questions, we would like to know whether and how you reacted to the corresponding emails. You can view the four emails again here:

• Q7: Have we successfully persuaded you to enter data during our campaign?

• Q8: Which of the four phishing emails do you remember? (Multi-Select)

• Q9: What was the major reason for a reaction to the email? (Matrix-Select, one reason per email)

  • Curiosity

  • Fear

  • Pressure

  • Financial Interest

  • Trust

  • Authority

  • I did not react to this mail.

  • Prefer not to answer.

• Q10: Please provide more information on your reaction. (Freetext)

• Q11: Which of the emails gave you the feeling that something was wrong? (Multi-Select)

• Q12: Please explain your feelings on the emails. (Freetext answer for each email)

• Q13: What did you do when you had off feelings with an email? (Multi-Select)

  • Visit the main webpage

  • Perform a web search

  • View the website imprint

  • View the website data privacy declaration

  • Investigate the website source code

  • Investigate the link target

  • Investigate the sender

  • Investigate the email header

• Q14: Have you carried out any further checks? (Freetext)

• Q15: Which precautions have you taken for your investigation? (Multi-Select)

  • I did not take special precautions.

  • VPN

  • TOR

  • Deactivate JavaScript

  • Deactivate Cookies

  • Use a special browser

  • Use a sandbox / virtual machine

  • Issue WHOIS / RDAP request for the IP / domain

• Q16: Did you implement any other precautions or technical measures? (Freetext)

1.3 A.3 IT-Context and Sensitization

• Q17: How often do you use IT-Devices for your work and in your leisure time?

• Q18: Estimate, how many emails you receive per day in your private and work contexts.

• Q19: Did you previously participate in courses or training for cybersecurity awareness?

• Q20: Which types of trainings did you previously participate in? (Multi-Select)

  • Classroom training (including digital group training)

  • Awareness information emails

  • Test phishing emails (outside this study)

  • Computer-based training

  • Online courses

  • Information videos

  • Social media content

  • Documentations (TV, Youtube)

  • Podcasts and radio

  • Print media (newspapers, flyer)

  • Posters and billboard advertisement

  • Other (Freetext)

• Q21: How long ago did you participate in your last training?

• Q22: Have you previously been affected by a security incident? (Multi-Select)

  • Reacted to a phishing email

  • Malware infection

  • Lost a password

  • Lost data

  • Lost access to an account

  • Stolen devices

  • Lost money

  • Other (Freetext)

B Appendix: Large Scale Images of Phishing Content

The paper incorporates tiny graphics as an overview of the emails and webpages employed throughout the four iterations of our phishing study. Here, we provide the following images for readers who want to look at larger-scale variants.

Fig. 8.

Large-scale screenshots of German phishing content sent throughout the four iterations.

Appendix: HITL-Model: Figures

Presented in the paper were shortened versions of the two taxonomies that highlight aspects which were more frequently named by study participants. However, in case fellow researchers would be designing similar studies, even answers from single participants could be helpful to understand what behavior to expect. Therefore, we present Figs. 9 and 10, showing the full range of participant responses to the survey on the respective stages in the HITL model.

Fig. 9.

Hierarchical overview of noticed properties named in survey responses. Participants noticed aspects within the Email, Website and during their Online Searches. $^*$ Numbers in [brackets] refer to the count of mentions.

Fig. 10.

Hierarchical overview of expected and suspected properties named in survey responses. We differentiate between Global expectations that could be identical across participants and Personal expectations, such as a concrete shipment. $^*$ Numbers in [brackets] refer to the count of mentions.

D Appendix: Resulting Correlations

In Table 3, we summarized the most important correlations we observed between our participant responses and their interaction with our phishing emails and web pages. The analysis has brought us to identify the highlighted observations as particularly important, e.g. because we further observed mentions of the aspects in qualitative answers. Additionally, Table 4 provides an overview of all impactful aspects derived during our analysis. In the table, we group the findings by the iteration they were reported of, with General applying to answers given to general, overarching question not directly targeted towards single interventions. Inside each iteration, we differentiate between the different phases of the HITL interaction model: Notice (N), Expect (E), Suspect (S), Investigate (Inv.), and Act (A). We compare the performance of the group that reported the respective feature (Share of Participants) to the performance of the General Population. Depending on whether participants who reported the respective feature performed better or worse, we indicate whether the respective group of participants reacted (React.) more or less often than their peers. The reaction translates to the phishing susceptibility, as indicated in Table 3 in the manuscript. An increased amount of reaction and, thereby, increased susceptibility hereby indicates worse behavior. Below, we provide one example of how to read the table:

Reading Example: People that highlighted General Expections towards how (third party) entities perform email communication (global/entities/email) performed better than their peers. Out of the 32 people who highlighted the respective feature, only 3.1% Clicked on the links provided in the emails, while generally, 24% of participants clicked on the links provided. This observation is statistically significant, as confirmed with a $\chi ^2$ test for significance resulting in $p = 0.001$.

Table 4. Overview of all Correlations observed between Participant Responses clustered to the HITL model.