Costs and Benefits of Authentication Advice

Advice regarding password phrases was the most commonly given advice we encountered. This implies that advice is mostly concerned with making passwords “strong.” While this is important for some attacks, for attacks such as phishing and keylogging the strength of the password is irrelevant [16, 52]. Within the category Phrases, the statements: Don’t use published phrases and Substitute symbols for letters had contradictions. For don’t use published phrases, the advice given was:

The last piece of advice directly contradicts the first. This inconsistency makes it no surprise that users seem disinclined to follow security advice [1, 25]. The advice statement Substitute symbols for letters is proposed by two sources but is advised against by a third. We know from Warner [48] that passwords with simple character substitutions are weak. Yet, 2 of 3 pieces of advice recommend it. This could stem from the attitude that it is “better than nothing.”

We collected six pieces of advice telling users to never reuse passwords and three pieces telling users to not reuse passwords for certain sites. In addition, we found three pieces of advice encouraging users to alter and reuse their passwords and three pieces telling users to not alter and reuse their passwords. There is little agreement among the password reuse advice.

Das et al. estimate that 43%–51% of users reuse passwords across sites [12]. They also provide algorithms that improve an attacker’s ability to exploit this fact. Florêncio, Herley, and Van Oorschot [15] declare that “far from being unallowable, password reuse is a necessary and sensible tool in managing a portfolio” of credentials. They recommend grouping passwords according to their importance and reusing passwords only within those groups. Interestingly, the advice we collected in the category Don’t reuse certain passwords gave a slightly different take on this advice. The advice mostly asked users to not use the password used for their site anywhere else, e.g.,

“Never use your Apple ID password for other online accounts.” Most organisations gave advice prioritising their own accounts. Only one piece of advice suggested using a unique password for any important accounts [17].

An alternative to grouping accounts for reuse is to alter and then reuse a password. This advice was given by three sources and rejected by three sources. Alterations to passwords are often predictable. Using a cross-site password-guessing algorithm, Das et al. [12] were able to guess approximately 10% of non-identical password pairs in less than 10 attempts and approximately 30% in less than 100 attempts.

Composition restrictions are regularly enforced by websites, but the advice given is not consistent from site to site. It is interesting to note that Herley [23] hypothesises that different websites may deliberately have policies that are restrictive to different degrees, as this can help ensure that users do not share passwords between sites. We collected 12 pieces of advice encouraging Enforce restrictions on characters and only 1 piece of advice against it. The source rejecting character restrictions was the NIST 2017 authentication guidelines [19]. These guidelines have received praise from the authentication research community [10] and specifically considered usability issues, which is why they moved away from some of the more burdensome, less effective advice still popular in the security community at that time. This raises the question of whether organisations will begin to disseminate these new security practices or continue to enforce their stringent password restrictions.

Kelley et al. show that strict composition rules do increase the security of passwords [27]. However, similar security can be offered by password blocklists or mandating minimum of 16-character passwords. In a related study, Komanduri et al. [28] showed that the 16-character password restriction was less annoying and less difficult for users. In addition, when composition rules are enforced, the probability of a user including a “1” as their number and an “!” as their symbol is high [46]. So, an attacker who knows the composition restriction in place could potentially tailor their attacks to suit it. Tan et al. also echo this finding, stating that requiring passwords to have multiple character classes has at most minor benefits to password strength and can even reduce effective security [45].

We found 5 pieces of advice telling organisations to Store password history to eliminate reuse, 1 encouraged organisations to Enforce a minimum password age, and 10 were in favour of Changing passwords if compromise is suspected. If organisations do store their users’ password history, then this creates an additional security hole, as the company needs to allocate resources to protect this file. Also, even though users can no longer reuse prior passwords, alterations are still possible [42]. Zhang, Monrose, and Reiter [51] identify that we can easily predict new passwords from old ones when password ageing policies force updates.

The reason given for introducing a minimum password age is to prevent users from bypassing the password expiry system by entering a new password and then changing it right back to the old one [30]. However, if an attacker gains access to a user’s account and changes their password, then the user will be unable to change it again until the required number of days have elapsed, or with an administrator’s help. Ten pieces of advice recommended changing passwords if a compromise is suspected. This can be inconvenient for users not affected by the compromise, and also for those who are. If there is a breach at the server, then the users were not at fault, yet still they must choose a new password.

From anecdotal evidence, we know the advice to change your password regularly is widely hated by users [20]. Seven pieces of the advice we collected encouraged the use of password expiry, while only four pieces of advice discouraged it. This is despite research suggesting that the security benefits are minimal [9, 51]. This implies the inconvenience to users is worth less to organisations than the minimal security benefits. Or, do organisations want to be seen to be enforcing strong security practices, and forcing expiry is just one way of doing this?

More discussions of the advice collected can be found in Reference [31]. However, the circulation of contradictory advice could give an insight into why organisations and users might have trouble following proper authentication recommendations.

The finalised set of costs was reached through an iterative process that involved assigning costs to each of our 79 advice statements and adapting the cost-categories until we reached a set of cost-categories that made it possible to describe the range of advice impacts. This finalised list of cost types, as shown in Table 3, was then presented to user and administrator participants in our study for feedback.

User versus organisation costs. We wished to differentiate between costs borne by the user and costs borne by the organisation. We believe it will be interesting to know who bears most of the costs: the user or the organisation. In addition, what might be a small cost for an organisation could be a large cost for a user. We, therefore, separated cost categories to be user- and organisation-specific, e.g., user computing power separate from organisation computing power.

Minor costs. In our analysis, we acknowledged the need to distinguish the extent to which a cost occurs. For example, both Enforce restrictions on characters and Make digital and physical back-ups require organisation time. But little organisation time is needed to enforce a composition restriction, and a lot of effort is needed to digitally and physically back up work. It is not within the scope of this model to have a full grading of the costs, but we do acknowledge a difference between small or partially felt costs and more substantial costs. In our tables, a minor cost is identified as ○ and a more substantial cost is identified as ●.

Periodic costs. We also notice that advice such as Make digital and physical back-ups and Keep email up-to-date and secure require continued action. This significantly increases the costs. We acknowledge three types of costs: one-off costs usually relating to setup or account creation, costs that occur at every login, and periodic costs that occur repeatedly over different time frames. To identify these costs, we use a super-scripted symbol. Costs at login are denoted by an at symbol: and periodic costs are denoted by a sun: .

Positive costs. Finally, some pieces of advice reduced the burden on the organisation or user. These pieces of advice we acknowledge as “positive costs” and, as such, they are represented with the positive symbol: +.

Once satisfied that we had a provisional set of cost categories for users and organisations, we began our user study. We created five surveys aimed at administrators and five surveys aimed at end-users. Both survey types used a multiple-choice format where participants were asked to identify the costs they associated with a selection of advice statements. We randomly directed participants to one of the five relevant surveys, each containing questions about a subset of advice. This allowed us to ask about the large number of advice statements we had collected without overburdening participants. However, to ensure that even with a low number of sample participants we would still receive some meaningful analysis, we did include some overlap in the end-user surveys. Every end-user was asked to identify costs for the following advice statements:

We chose these as representative of typical current advice. Note that the advice “Do not allow users to paste passwords” was not collected as part of our research yet we include it here. We chose to add it to our survey as a “canary.” This advice has been discussed online as a piece of bad advice that, unfortunately, became common practice, but has no discernible security ramifications [24]. This advice will act as a test case for our methods.

While we might have liked to include overlap in the administrator surveys, too, the breadth of advice related to administrators was too great to allow redundancy. In fact, we were forced to remove some advice from the administrator survey to keep the estimated completion time within 10 minutes. Neither users nor administrators were remunerated for completing the survey, so keeping it within a shorter time frame was important for encouraging meaningful responses.

Example survey question. In the survey, participants were asked to indicate the severity and frequency with which they experience costs and inconveniences as a result of authentication advice. We also asked for comments about whether they approved of the given piece of advice. Figure 1 shows a graphic providing users with an explanation of how to complete a survey question. Users were also given the following example to help them understand the survey:

Administrators were given a similar description but were asked to assess which of the organisation cost categories the advice affected. Full versions of the questions and answers for both the user survey and the administrator survey are available on GitHub [31].

Participants were chosen using a snowball sampling technique. For the administrator survey, we reached out directly to our contacts in systems administration or security management roles. We asked these people to take part in the survey and encouraged them to recruit future subjects from among their acquaintances. Similarly, we reached out to a selection of non-technical acquaintances, asking them to complete the user survey and encouraging them to circulate it to others. Both surveys were circulated to the general population via social media.

The study was approved by our university Ethics Review Board and no personal identifying information was collected about participants. The only demographic information collected asked end-users to rate themselves on their internet security knowledge. This was to ensure that our sampling techniques did not skew our end-user results towards highly technical or security-aware individuals. We received 44 participants for our end-user survey and 37 participants for our administrator survey. In Figure 2, we plot the number of users who assigned themselves to each Internet security rank. Users were told to “Select a rank on the below scale from 1 (very poor) to 5 (very knowledgeable) to describe your internet security knowledge.” No users described their knowledge as very poor but after that, we seem to have a reasonable spread across the remaining ranks with an expected lower number of people selecting 5 (very knowledgeable).

When collecting the survey results, we use the following rules for identifying what costs the participants agreed exist for each piece of advice (i.e., did participants believe a cost major, minor, positive, or non-applicable):

Users and administrators were also asked about their approval of the advice statements. The results of this are represented in the table for both users and administrators. A ✔ represents that the majority of respondents approved of the advice.✘ indicates that the majority disapproved, and■ represents that the majority of respondents were neutral about the advice. Occasionally, respondents were split between different approval ratings. This is also indicated in the table. For example, if users were split between approve and neutral, then we represent this as✓/■.

In Table 4, we present the costs that users and/or administrators associated with each piece of advice. As mentioned, we could not include every piece of advice in the surveys in an attempt to reduce the size of the survey. In the table below, we highlight the corresponding row in the table in blue (see the organisation side of the row “Enforce maximum length”) to show that this advice was not included in the survey. At times, even when the advice was not directly included in the survey, we indicate obvious costs that exist. For example, for the “Enforce maximum length” advice, the administrative costs could be extrapolated from the “Minimum password length” costs.

The costs associated with implementing and following advice are shown in Table 4. The most selected user cost category was increased risk of forgetting, and the most selected organisation cost category was user education required. It is important to note that while organisation time is required to develop and run user education, user time is also necessary. Therefore, each time we see a user education cost, we recall that this also involves a burden for users.

There were 200 costs identified across all the pieces of advice for the organisation and 109 costs identified for users. Of these, the users identified that 28% of them were major costs. Administrators identified that 16% of the costs to the organisation were major. 37.5% of these major organisation costs were attributed to user education. Of the 200 organisation costs, 44% were repeating costs, i.e., they were felt at each login or periodically. Of the users’ 109 costs, 46% were repeating. However, nearly half (49%) of the repeating organisation costs were attributed to user education, a cost that will be felt by users as well. Similarly, of the user repeating costs, 40% were related to the cost of forgetting, as many attributed this as occurring at login. The cost of forgetting is a cost that is felt by the organisation as well, since the users’ only solution is often to contact the organisation’s help desk.

Of our 11 cost categories, help desk time, time to implement, user education, and all the user costs excluding computing power, involve a human directly. In these categories, advice is adding to the emotional burden or workload of the individuals involved. Humans are impacted by 87% of all the costs identified. In the user survey, we witness strong emotional responses towards the advice we asked users to analyse. We received responses such as “I am sick of passwords and logins and they are making me less productive as I have to look up the passwords so often!!” Users only agreed on three pieces of advice that have a positive impact on a cost category. These are: “You should use a password manager to store your passwords,” “Write your password down safely,” and “You should alter your password and then reuse it on other websites or computers.” All three of these had a positive impact on memorability.

For each piece of advice, we asked users and administrators to indicate whether they approved of the advice, felt neutral about it, or disagreed with it. In the 6th and 13th columns of Table 4, we indicate the opinion that represented the reactions of the majority of respondents in each category.

While we took the majority answer as the mutual consensus for Table 4, it is important to note that opinions on advice were divided. We found that for 32% of the advice, there was at least one administrator respondent who disagreed with others on whether advice was valuable. More starkly, for 71% of the advice considered by end-users, there was at least one user-respondent with a different opinion from others. While a 32% difference could be put down to differences of opinion on practice or different interpretations of the advice statement, the 71% disagreement among end-users shows a significant lack of consensus.

Table 5 shows examples where a lack of consensus was prominent. For both administrators and end-users, we indicate how many respondents agreed with the given advice statement, were neutral about it, and disagreed with it. For example, 6 administrators agreed with the advice “Never reuse passwords” and 1 was neutral about it. However, for the same piece of advice, 19 users approved of “Never reusing passwords,” but 9 were neutral and 13 disapproved. One user said, “I experience regular frustration and lose valuable time at work trying to login to numerous platforms with different passwords.” The majority opinion for each group is highlighted and indicated in the decision column. Notice that in some places, the users and administrators come to a different consensus about whether a piece of advice is valuable. For example, users believe they should be allowed to store password hints, but administrators disagree.

Throughout Table 5, we see evidence of differences in opinion between users and administrators. Users were conflicted about whether typing URLs was good advice, however, administrators unanimously (excluding neutral responses) disapproved of it. Administrator respondents said “It might be nice in theory, but impossible to implement and enforce in practice” and “There may be occasions when this is relevant but typo-squatting also poses a risk besides phishing.” This advice might have security benefits, but it introduces a new attack vector and also results in a significant time loss for anyone who follows it.

The advice “Include specific characters in your username” divided administrator opinions and users strongly disagreed with it. In a 2007 research paper [14], Florêncio et al. determined that rather than making the secret password more complex, complexity could instead be added to the username. Users can choose and record a long username and this will likely offer effective protection from a bulk guessing attack; the same attack that a complex or long password is hoping to combat. One administrator disapproving of this advice said, “good for passwords, over complicates usernames as password should be secure, usernames are often generated based on firstname.lastname etc.” One end-user respondent commenting on the advice said, “Stupid and pointless” and another said, “Can’t see a good reason for this, but with a password manager it wouldn’t be too painful.” If this policy was being implemented in an organisation, then these responses emphasise the need for it to be coupled with effective user education and validation.

Administrators were generally in favour of altering a password before reusing it at another site. But most users surveyed did not approve of the advice. One user who disagreed with the advice said, “Impossible to remember what password goes with what site.” A different user who disagreed said, “Any method to your password creation makes it less secure.” One administrator commented that ensuring alterations before password reuse should be a strict policy that is reinforced by user education.

End-users were torn on the advice, stating that it should not be possible to paste your password when logging in. The comments given emphasise the differing opinions. One user said: “If someone is trying to copy/paste a password, they’re probably beyond help.” While another user said: “This is horrendous advice that leads to problems using password managers. It encourages using crappy passwords.”

Many users were willing to follow advice if it seemed beneficial. For example, for the advice “regularly change passwords,” one user commented: “Undoubtedly sensible, undoubtedly annoying.” There is substantial evidence in the responses [31] that user education has been effective. Generally, users’ responses echoed the advice that was given in historic security documents [7]. However, few users and administrators showed an awareness of new recommendations that have superseded the legacy advice [19]. For example, the 2017 NIST advice recommends not introducing restrictions on the characters allowed in passwords. Yet, most users and administrators indicated that they agreed with forcing the use of certain characters in passwords. Note that this new NIST advice has been available and circulating since 2017, and our user survey was completed in October 2020.

Overall, understanding users’ approval of advice is a valuable stepping stone to improved user education and mutual respect between users and policy enforcers. As Adams and Sasse show, users will simply employ less secure workarounds if security policies do not allow them to do their job effectively [1].

Unauthorised access has many avenues, and one piece of advice is unlikely to protect against them all. For example, requiring passwords of length 12 might help against a password-guessing attack but will do nothing to protect against a phishing attack. To represent how “beneficial” a piece of advice is, we would like to know in which ways the piece of advice protects against unauthorised access, if at all. To this end, we require a concise list of the different methods an adversary could use to gain access to a protected account. The NIST 2017 Digital Identity Guidelines [19] includes a table of “Authenticator Threats.” Given that this NIST 2017 authentication document has been a highly regarded [10] and thoroughly researched document, we accept these as a concise list of the attack vectors against authentication. Below, in our Table 6, we list the attacks that NIST identified (Table 8-1 in Reference [19]). Included are the name, details, and example for each different attack type. In places, we diverge slightly from the official text provided by the NIST document to simplify the descriptions and examples.

As with costs, which could be larger or smaller, the benefits provided by different advice also vary. The benefits of advice, therefore, should reflect the probability of each attack being successful. Given some baseline chance of each attack, we reflect on whether a piece of advice increases or decreases the chance that an attack is successful. We show the result of this analysis for the 79 advice statements in Table 7. We use the symbols \(\color {red}{\mathbf {\Uparrow }}\) and \(\color {green}{\mathbf {\Downarrow }}\) to indicate an increase or decrease, respectively, in the probability of success for the attack type. \(\color {red}{{\bf \uparrow }}\) and \(\color {green}{\mathbf {\downarrow }}\) indicate less significant increases or decreases in the probabilities of success for the attacks. An underline, , indicates that the improvement is not directly enforceable. This could be because the advice is too vague and therefore there is ambiguity on how an organisation or user might follow it. Or simply that an organisation ca not bind their users to follow the advice. In supplementary material, we include all explanations for why we believe the chances of certain attacks are impacted by the advice [31]. When determining this assignment of benefits, we took input from stakeholders and experts, including giving presentations at specialist conferences/workshops [21, 22, 32, 47], statistics from our university IT department, and insights from our university’s information security manager.

Throughout Table 7, we mark each piece of advice as either increasing or decreasing the chance of compromise. However, there is an interesting question about what baseline the advice increases or decreases the chance of compromise relative to. We first thought of the baseline as being a passive version of the advice. For example, if the advice is: “Email up-to-date and secure” or “Encrypt passwords,” then the baseline we measure the improvements from is doing nothing. This made sense for a lot of the advice statements, but not all. For example, the advice “Do not store hints” is already a passive action, but we do not want to measure its effects against itself. Our second consideration for the baseline was the opposite of what the advice stated. For example, for the advice “do not store hints,” we consider the difference in attack threat between this and “do store hints.” Or maybe less strictly, “you are allowed to store hints.” This appears to be a logical comparison for most pieces of advice. The downside is that the opposite can, in a few cases, be too “strict.” For example, the opposite of “do not include names” and “don’t repeat characters” are “include names” and “repeat characters,” respectively, both unnatural pieces of advice. Two pieces of advice had ambiguous opposites. The opposite of “write down safely” could either be “don’t write down” or “write down, but not safely.” In this case, we chose to consider the opposite as “don’t write down.” Similarly, we choose the opposite of “alter and reuse passwords” to be that reuse is allowed even without altering. For our purposes, we choose to identify the increase or decrease in the chance of attack by comparing it to the opposite of the specific piece of advice. However, in a real-world situation, a proposed policy could simply be compared in light of the current/previous policy in place. No matter what baseline method is chosen, the discussions provided in our supplementary material [31] should be relevant when assessing benefits.

Below, we take two advice categories and provide our discussion describing the benefits that were identified for each advice statement in that category. The two categories are password managers and generated passwords. For a discussion of all other advice statements in Table 7, see Reference [31].

Overall, we identified 177 positive benefits (decreasing the chance of attack) and 26 negative benefits (increasing the chance of attack). 69% of the negative benefits related to the user advice and 31% of the negative benefits related to the organisation advice. A more even split of 43% to 57%, respectively, exists for the positive benefits.

We begin by assigning points to each cost type using an ordinal scale. We assign 2 points to a major cost that reoccurs at each login, 1.5 to a major periodic cost, and 1 to a major one-time cost. A minor login cost is assigned 1, a minor periodic cost 0.75, and a minor one-time cost is 0.5. A positive cost can account for \(-\) 2. Though this is a crude scoring system, it provides insight into both the most costly and least costly advice. Figures 4(a) and 4(b) depict the 12 most costly pieces of advice and the 6 least costly pieces of advice for the organisation and the end-user, respectively.

In Figure 4(a), most of the least costly pieces of advice to the organisation relate to the composition of passwords. For example, “Take initials of a phrase and use this as your password.” Advising this does not bring any costs to the organisation. Many of the most costly pieces of advice for the organisation involve back-end processes; for example, “Implement Defense in Depth,” “Monitor and Analyse intrusions,” and “apply access controls.” Introducing two-factor authentication shows up in both the user and the administrator’s most costly advice. Similarly, the advice to change passwords regularly occurs in both top 12 lists. Whereas, the advice “Don’t choose remember me on your computer” has high costs for users but low costs for the organisation. Most of the most difficult pieces of advice for users relate to password creation, memorising, and changing. For example, “Don’t use dictionary words in your password,” “Password hints should not be stored,” and “Store history to eliminate password reuse.” This last piece of advice specifically relates to when regular password changes are enforced. It ensures a user does not choose a previous password. The piece of advice “Don’t use dictionary words in your password” is particularly interesting. The advice was given by 16 different sources, yet we know from leaked password databases that users primarily choose word-based passwords [49]. This depicts how impotent giving advice can be if the costs appear to not outweigh the benefits from a user’s point of view. Shay et al. find that the “use of dictionary words and names are still the most common strategies for creating passwords” [42]. Notice that the last piece of advice in the user chart Figure 4(b) has a negative cost. This is because allowing users to use any ASCII characters actually makes their life easier. Of course, most security policies and rules will come at a cost. What will be interesting is to see whether the benefits outweigh these costs.

For benefits, we use a similar ordinal scale: a large decrease in attack risk ( \(\color {green}{\mathbf {\Downarrow }}\) ) counts for 2 points, a minor decrease to attack risk ( \(\color {green}{\mathbf {\downarrow }}\) ) is 1 point. A small increase to risk ( \(\color {red}{{\bf \uparrow }}\) ) is \(-\) 1 point, and a large increase of risk ( \(\color {red}{\mathbf {\Uparrow }}\) ) is \(-\) 2 points. Note that a comparison of benefits alone can only provide limited meaning. Some attacks are more important to protect against than others, and advice can protect against attacks to varying degrees. However, for this analysis, a simple comparison of how many attacks a piece of advice protects against and whether it offers minor or major protection could be interesting. We, therefore, attempt a loose ranking of the advice according to the assigned benefits. “Security beneficial” is a simple indication of whether the advice protects against the different attacks from Table 6.

Using the above scoring methodology, Figure 5 shows the 12 most security-beneficial pieces of advice and the 6 least security-beneficial pieces of advice. The advice that this classification identifies as the most beneficial seems to be in line with what we might imagine. Also, notice that there is an overlap between the 12 most beneficial pieces of advice and the 12 pieces of advice that had high organisation costs. For example, “Apply access controls,” “Implement defense in depth,” and “Monitor and analyse intrusions” appear in both lists.

The 6 least beneficial pieces of advice actually result in negative benefits. This means they increase the likelihood of an attack occurring. Looking through the 6 pieces of advice, we can understand why. For three of these advice statements, rather than increasing security, their goal seems to be to create a better user experience. For example, “write [your password] down safely,” “offer to display password,” and “generated passwords must aid memory retention.” The other three pieces of advice have negative benefits, because they create a new attack vector or make an attack more likely. Distributing passwords by envelope means they can be physically intercepted. Enforcing a maximum length puts an upper bound on the length of passwords and makes guessing attacks easier. Finally, though storing backups is an important secure practice, it rarely actually protects against attacks, instead, it mitigates the harm done if an attack takes place. Therefore, purely in terms of protection from attacks, this ranks poorly. In addition, having both physical and digital copies creates extra data that must now be protected, and physical protection is now a factor. This is a good example of how our simple points system does not give the whole picture of whether a security practice should be employed.

We are interested in the tradeoffs between the costs and the benefits. Is high-cost advice balanced by high benefits? Or are users paying high usability costs for small increases to security? To analyse costs versus benefits, we plot each piece of advice on a graph. This allows us to visualise how benefits compare to costs. The benefit score is shown on the x-axis, and the cost score is on the y-axis. Advice that falls in the bottom right quadrant (green area) is high-benefit and low-cost advice. Similarly, the advice that falls in the upper left quadrant (red area) is low-benefit and high-cost advice. The worst advice will have low benefits and high costs. Figure 6 shows the results of this plot for a selection of advice.²

High-cost and low-benefit advice. Let us look first at the high-cost and low-benefit advice. Reassuringly there is not too much advice in this quadrant. The only piece of advice that falls in the red area is “change your passwords regularly.” This was high-cost advice for both the end-user and the organisation. Research has also shown that regular password changes have few security benefits and that they can do more harm than good [9, 51]. The NIST advice explicitly states that regular password changes should not be enforced [19]. One user in our user study wrote “I hate this! The only solution I’ve come up with is to increment a number in the password each time. So inconvenient and frustrating, especially when combined with other bad password advice.”

The next piece of advice closest to the top left (high-cost and low-benefit) quadrant is the advice “Use SMS or call-based 2-factor authentication.” While we have generally come to accept that two factor authentication (2FA) offers security benefits, SMS text and call-based authentication have security flaws. Text and phone calls are not protected by encryption, and phone numbers can be easily spoofed. In the initial version of the NIST Digital Guidelines, SMS-based two-factor authentication was to be deprecated [18]. However, this decision was overturned [19]. Susceptibility to spoofing explains why this advice receives a lower security ranking than 2FA using an app or specialised device in our plot. However, the low ranking of 2FA in general reflects a limitation of our model. Despite offering major decreases in attack probability in the important areas of phishing and online guessing, 2FA using a phone introduces four new attack vectors: physical theft, eavesdropping, side channel attack, and endpoint compromise. Since our model does not have probabilities or more graded weightings, it thinks that these new attack vectors outweigh the benefits. This is only an issue for the 10 pieces of advice that have both positive and negative security impacts, however, it is important to keep in mind.

Low-cost and low-benefit advice. In the bottom left quadrant, we can see low-cost and low-benefit advice. Enforcing a maximum password length falls directly into this category. This advice had no positive security value. It is an example of advice that compromises usability for no increase in security. Unfortunately, this practice is still enforced by organisations. In 2014, Saini assessed the policies of 23 different websites and 8 enforced maximum lengths of 40 characters or less on passwords [40]. Three of these limited the length to just 16 characters and one limited it at 12 characters. In the course of our study, we found that some websites only reveal their limit on password length after the user had attempted to use a longer password [33].

High-benefit advice. “Administrator accounts should have extra protection,” “Every user in an organisation must have their own account,” and “Implement technical defenses” all fall solidly within the bottom right quadrant. “Apply access control systems” comes at a higher cost but offers the strongest benefits of any advice. All the advice in this quadrant is shown in Table 8 under Good advice. Notice that most of the high-benefit advice is under the control of the organisation and not the end-user. It seems that if an organisation can put proper controls and secure systems in place, then they can ensure a much higher level of security more effectively than placing the burden on the user.

Some benefit and some cost advice. The advice in the blue diagonal is more difficult for us to form conclusions about. Anything on the positive x-axis offers security benefits but inevitably comes at a cost. For this advice, a more detailed quantitative analysis would be necessary to determine whether the benefits outweigh the costs.

For example, one expensive piece of advice is “Use 2-factor authentication using an app or special device.” However, it offers increases in security against online guessing and phishing attacks, two of the most common attack types. It is likely that ensuring the benefits outweigh the costs will depend on its implementation and the needs of the specific organisation and users. These nuances are something our current model ca not uncover. Similarly, the piece of advice “Use a Password manager” lies at (benefits=4,costs=2). It was well-regarded by users in our study. It can protect against three attack types, and most of the costs it incurs are to the organisation. A password manager greatly reduces the users’ memory load and by extension, a user can use as long, random, and complex of a password as they wish. However, as with many of the pieces of advice, the value of a password manager will lie in how users utilise it. If a user uses a password manager and continues to reuse a common password choice across multiple sites, then many of the potential benefits will not materialise.

In summary, we see that, in many cases, it is difficult to distinguish whether the costs outweigh the benefits of security advice at a glance. Most advice had some negative impact on users or the organisation. This difficulty in assessing the tradeoff could be evidence of one reason why users and organisations often follow advice that researchers have shown ineffective. It seems, an “at a glance” observation, even by a security professional, might not always be possible for many of the pieces of advice we collected. Interestingly, we would have expected the high-benefit advice to correlate with high costs. However, when we plotted a trend line for the full scatter plot of advice, this was not the case. The trend line had a negative slope of \(m=-0.0373\) . The fact that the high-cost advice does not map to high benefits implies that the advice we force users and organisations to follow does not necessarily result in positive returns for their effort. Interestingly, if we restrict to organisation costs, then the trend line has a positive slope, while restricting to user costs gives a negative slope. This reinforces our earlier observation that organisational efforts seem to result in a better cost/benefit tradeoff.