Keywords

1 Introduction

Design options abound in user experience (UX) design. Thus, UX designers constantly face decisions about whether to do things in one way or another. These decisions concern the design product as well as the work process, and they ultimately determine whether the designs succeed or fail. While many studies have examined how designers collaborate [18, 23], generate ideas [7, 25], and acquire information [9, 19], we focus on how they make decisions.

While models of decision-making conventionally depict it as a rational process of defining the problem, identifying decision criteria, developing alternatives, and selecting the best alternative, this process is rare in practice [26]. In practice, professionals often make decisions on the basis of intuition, experience, analogy, and the like [2, 10, 12, 13, 30]. Klein’s [13] recognition-primed decision (RPD) model has become a prominent conception of how experienced professionals make decisions. We take the RPD model as our starting point and add a focus on mental models, which have long been an important notion in design [17, 22]. In addition, we heed Allen’s [2] advice to study decision-making together with information seeking.

This study investigates, based on observation and interviews, how UX designers use the mechanisms of RPD, information seeking, and mental models in making UX design decisions. Furthermore, we identify the UX design layers in which these mechanisms are used. The study contributes insights about how UX designers arrive at their decisions and, on that basis, discusses implications for decision-support systems for designers.

2 Related Work

Klein [13] developed the RPD model on the basis of studies of command-and-control (C2) staff, such as fire fighters. C2 staff must be able to react rapidly and flexibly in dynamic, high-stake situations. Rational decision-making is ineffective under such conditions because it is too slow. Instead, Klein [13] found that C2 staff mainly makes decisions through pattern recognition and mental simulation. Pattern recognition is the ability to recognize analogies between the current situation and previously experienced situations, without explicitly stating these analogies beforehand. It turns experience into an action-oriented ability. Mental simulation is the process of consciously enacting a sequence of events [16]. It enables the actor to mentally try out an explanation or idea to learn how well, or poorly, it matches the current situation.

While UX designers’ decision-making has not received much research attention, several researchers have investigated decision-making in other design fields, including engineering design [1, 4, 8] and software design [3, 30]. Designers often make tentative decisions during design processes [4, 8]. It is not until criteria emerge and consequences are clarified to a satisfactory level that designers would make final decisions [8]. This way, decisions remain tentative until the designers have gained confidence in the decision-making process that forms the basis for the decisions [4, 8]. In addition, researchers have extended and refined the RPD model. For example, Ahmed et al. [1] found that experienced designers rely on intuition and pattern recognition by referring to past designs. Dwarakanath and Wallace [4] found that designers use the RPD model by evaluating an alternative as soon as it is generated. Furthermore, it is only in the early design phases that they generate different alternatives and compare them with criteria; during detailed design their decision-making process becomes more implicit [4]. Zannier et al. [30] found that in addition to mental simulation, designers also turn to mental models when they face complex questions. In the naturalistic decision-making community, mental models are defined as “a person’s beliefs about causal relationships” [14, p. 167]. Nielsen [21] similarly states that mental models are based on belief rather than fact.

Multiple researches have investigated the role of information seeking in decision-making [12, 20, 27]. One noteworthy finding is that C2 staff often seeks information to justify their decisions, rather than to make them. For example, Mishra et al. [20] found that emergency-response commanders tend to look for information that provides post hoc justification for their decisions. This behavior accords with Allen’s [2] information-behavior modes and extends Wilson’s [29] problem-solving model of information seeking. Outside of C2 settings, people also seek information to justify their decisions. For example, Soelberg [27] found that people often look for justification for their decisions, rather than for information to help them arrive at their decisions in the first place. In Soelberg’s [27] study the decision makers spent weeks on justification before they were ready to act on their decision (about which job to choose); in the study by Mishra et al. [20] the more experienced emergency-response commanders acted near immediately and often did not seek justification until they were retrospectively asked for it. Regarding designers, Girod et al. [6] identified that the designers who used informal decision-making methods sought more external information. Informal decision-making methods emphasize subjective assessment over evaluation matrices and numerical scales. Although these methods appear similar to naturalistic decision-making, informal decision-making is not, at least not necessarily, based on the expertise that comes from years of experience. The use of informal decision-making and external information often resulted in less effective decision-making because the designers spent less time on defining criteria than seeking information [6].

Dwarakanath and Wallace’s [4] finding that designers’ decision-making changes from early to detailed design creates a need for a categorization of the elements in the design process. Based on analyses of website design, Garrett [5] categorizes UX elements into five layers. Ordered from abstract to concrete, these layers are strategy, scope, structure, skeleton, and surface. The layers are interdependent. For example, strategy design frames scope design, but scope design also has an influence back on strategy design. Table 1 gives the definitions of the five UX layers.

Table 1. The definitions of the UX layers [5]
Full size table

3 Method

We conducted 113 h of field observation and five interviews in two digital product teams in a logistics company in Denmark. The first team was designing a decision-support system for company-internal trade managers, and the second team was designing a cargo-monitoring and information-sharing platform for the company’s customers. Both teams were agile teams and used a Kanban board – an agile project-management tool. The Kanban boards showed all tasks as user stories. A sample task title was “As a user, I want to do… so I can…” Each task description specified criteria that the team members needed to achieve when designing the product.

In the teams, we observed product owners (PO1 and PO2), UX designers (UX1 and UX2), and a user researcher (UR2). We collectively refer to these team members as designers. The product owners were included as designers because they often made UX design decisions. During the field observation we sat in on team meetings and also observed the designers’ informal discussions with each other and with other people on site. As part of the field observation we occasionally asked designers for explanations of their decisions. After the field observation we interviewed PO1, PO2, UX1, UX2, and UR2 about how they arrived at their decisions. Each interview lasted about an hour. Prior to the field observation and interviews, the company and the designers gave their informed consent to participate in the study.

The field observation was documented in written notes, the interviews were audio-recorded and transcribed. In analyzing the data we first identified the decision points, using Klein et al.’s [15] definition of decision points. Four cues were used in identifying the decision points:

  • Explicit verbal cues, such as “I had to decide…”

  • A designer considered multiple alternatives and then proceeded according to one of them

  • A designer made a judgement that affected the outcome of the design process

  • A designer proceeded in one way in a situation where another team member might have proceeded in another way.

As the second step of the analysis, we identified which UX design layer each of the decision points was about. We used Garret’s [5] five layers, see Table 1. For example, when a designer decided to include certain information on the interface to make it available to the users at that point in the dialogue, the designer was making a decision about the structure layer. Third, we analyzed how designers reached their decisions by distinguishing among three mechanisms: RPD, information seeking, and mental models. We identified RPD by its two main components: mental simulation and pattern recognition [13]. For mental models we applied the definitions of Klein [14] and Nielsen [21]. Table 2 gives the definitions of these decision-making mechanisms along with an example. Fourth, we used open coding to analyze in more detail how the designers used information seeking and mental models to make decisions.

Table 2. The decision-making mechanisms
Full size table

4 Results

After excluding 31 decision points for which we could not determine the decision-making mechanism, we had 48 decision points for analysis. At only two of these 48 decision points (both in the strategy layer) did the designers generate multiple alternatives before making a decision. Thus, the widely touted process of rational decision-making was rare.

4.1 RPD

The designers used RPD consistently from scope to skeleton design, see Fig. 1. Overall, 23% of the mechanism instances were RPD. During scope design the designers mainly used pattern recognition; during structure and skeleton design they mainly used mental simulation.

Fig. 1.
figure 1

Percentage of RPD, information seeking, and mental models in decision-making at the five UX design layers. Because the designers used two mechanisms at eight of the decision points, the total number of mechanism instances was 56.

Full size image

The designers pattern recognized on the basis of personal experiences, not on the basis of information they learned from other sources. They for example did not pattern recognize on the basis of user behavior because they, apart from UR2, seldom personally observed user behavior during user-research sessions. Instead, they recognized and made use of analogies with Gmail, Proto.io, and other systems they often used themselves. As an example, one of the designers decided to provide their customers with both old and revised system versions because other companies did so: “Like google, for example. They allow you to switch between the new beta inbox design or, if you want, the old Gmail design.” The reason why the designer used Google as an analogy was its product quality and large user base. She also pointed out that it is important to show other team members the results of user research in order to convince them that a decision is right.

Regarding mental simulation, the designers used it in two ways: to simulate user-computer interactions and to simulate the consequences of modifying user-interface elements. The designers tended to do the former based on their experience of interacting with similar systems and the later through technical considerations. For example, in a meeting, PO1 immediately simulated how users would use filters for searching by using a wireframe UX1 had designed: “PO1 pointed at the wireframe UX1 had drawn on her notebook and said that if users use a filter to search, then the other filters won’t work.” PO1 later described the simulation of user-computer interactions by saying that “you know just as much as other users from experiences that make sense […] It’s a lot easier to empathize with your users than it is to have a technical understanding of how the back-end infrastructure should work.” This example shows that PO1 thought that simulating user-computer interactions is easier than simulating technological solutions because he can empathize with users but does not possess engineering knowledge.

With respect to simulating the consequences of modifying user-interface elements, we observed an instance in which UX2 decided that a certain element should be larger. When we later interviewed UX2, he explained that the interface element had to be enlarged to account for the possibility that the interface language was switched to a language in which the text occupied more space: “When looking at those tiles, you immediately switch to development perception […] Some languages require more space than others to express whatever it is you’re trying to express. What will happen if you switch to another language?” In general, we observed fewer instances of the second type of simulation than of the first type.

4.2 Information Seeking

Information seeking was a frequent decision-making mechanism. A total of 39% of the mechanism instances were information seeking, distributed across all five layers (Fig. 1). The designers sought information the most during structure design. Their Kanban policy stated that they had to conduct user research after user-interface design and for that reason they conducted user-feedback sessions during structure design. Starting from structure design, they also sought information, such as design criteria, in documents:

“UR2 told UX2 that PO2 thought there were too many call-to-action buttons, but UX2 explained and showed UR2 the user story with design criteria. He read a user need aloud and pointed at the screen.”

In looking for design inspiration we observed that designers sometimes tried out design features on websites that appeared not to be directly related to the design they were doing. Furthermore, they sometimes shared inspiring examples with their team via an internal communication tool.

The company culture created frequent opportunities for the designers to acquire information from other designers. For example, the weekly UX meetings provided opportunities for the designers to share their designs and get feedback. None of the 48 decisions changed as a result of these UX meetings. However, in one instance we noticed that two designers had different mental models for the same decision. The meeting improved both designers’ understanding of the decision point:

“In weekly UX meetings, designers show and talk about what they are working on. UX2 shows the UX team his interaction design […] UX1 asks why UX2 has chosen to use the users’ name, rather than their email, for login. ‘I think it was to create something’, UX2 replies. UR2 joins the discussion and says ‘no’; it is more about users who do not want their emails to be given to the company.”

In almost half of the information-seeking instances the designers sought information to justify their decisions. That is, they already had a decision in mind when they asked other team members for their opinion. There were two reasons for this behavior: to bolster their personal confidence in the decision and to create team ownership of the decision by talking about it. UX1 stated that receiving opinions from her colleagues made her feel more confident: “You either get validation – they agree that it is a good idea – or they question you and make you question whether it is a good direction […] That makes you feel a little bit more comfortable about the direction you are going”. We did observe instances where designers changed their decisions on the basis of feedback. In all these instances the feedback came from users, not from other designers. However, the majority of the feedback confirmed ideas and decisions rather than suggested new designs.

4.3 Mental Models

Mental models were frequent during strategy, scope, and surface design (Fig. 1). A total of 38% of the mechanism instances were mental models. We identified three kinds of mental models: (1) causal relationships, (2) empathy, and (3) simple statements.

The causal-relationship mental models were similar to Klein’s [9] definition of mental models. For example, PO1 decided to create a new user-authentication system because it would enable users to find all the information they were allowed to access in one place. In another instance, PO2 decided not to establish a traditional partnership because it would be too much effort. He had previously had his own business and knew from that experience that: “Traditional partnerships require a lot of time and energy, and you have to do the same things again and again.”

Like causal relationships, empathy was also a frequent kind of mental model, particularly during strategy and scope design. The designers imagined themselves as users and made decisions that would improve their own user experience. For example, in one instance PO2 decided that the product should be the users’ best friend. He said: “if I put myself in their shoes, what I would really want is to have this really knowledgeable guy helping me, being friendly, and making my workdays easier.”

Lastly, simple statements were the least frequent kind of mental model. Some of these mental models were expressed as knowledge about principles for good design. For example, a designer chose a sans-serif font because “serif is not good on the screen.” Most simple-statement mental models were however expressions of information derived from the designers’ personal experience. As an example, UX2 and UR2 were designing a notification feature and needed to decide whether to add a pop-up screen to explain to the users why they received the link contained in the notification. UX2 reasoned (using mental simulation) that they should add the pop-up: “If a sender just pastes the link instead of providing additional information, then the user will not have the context.” UR2 countered this reasoning with a simple statement derived from his user research, namely that he had “the impression from our customers that users would be provided with a context.”

5 Discussion

The UX designers use a mix of pattern recognition, mental simulation, information seeking, and mental models in their decision-making. While pattern recognition and mental simulation are restricted to the three middle UX design layers, mental models are primarily used in the most abstract and most concrete layers. The absence of mental simulation during strategy design is understandable because this layer does not have clear user-interaction goals, thereby making it difficult for the designers to match the outcome of mental simulations against desired goals. Hence, when asking for a colleague’s opinion, a designer received confirmation for her decision but also the feedback: “You have to come up a decision and figure it out [whether it is a good decision].” The designers’ preference for mental models in the abstract design layers indicates, we surmise, that strategy and scope decisions are more readily made using causal relationships and empathy. The frequent use of causal relationships also suggests that the designers may lack the experience base necessary for pattern recognition [1, 24]. The importance of empathy in UX design decisions extends previous RPD research [13,14,15]. Consistent with RPD research the UX designers rarely generate multiple alternatives before making decisions, and they only do it during strategy design.

Information seeking is spread across decisions in all five UX design layers. It is about evenly divided between information seeking to reach a decision and information seeking to justify a decision that has already been reached. The high incidence of information seeking for justification is consistent with previous research [2, 20, 27]. In fact, the experienced emergency responders studied by Mishra et al. [20] appear to have an even higher incidence of information seeking for justification. Justification serves to increase the designers’ confidence in their decisions but has little impact on the quality of their decisions. The high incidence of justification is also similar to Weick’s [28] description of sensemaking as a retrospective process. Weick asserts that actors make sense of an event by looking back at it to see what they have experienced. This way, sensemaking may inform future decisions, but it lags behind the decision-making process that produced the current event. Like other designers [9, 19], the UX designers often seek information from colleagues, rather than in documents. Consulting other people is a straightforward way of acquiring feedback on designs and ideas, either as informed opinion or as impetus for creative discourse.

We see two implications of our results for decision-support systems. The first is a shift of decision support away from the documentation of the unfolding decision-making process. To support UX designers in pattern recognition they must experience a rich variety of examples. Klein [14] proposes decision games as a vehicle for C2 staff to play out decisions and experience their consequences in dynamically evolving situations. However, UX design does not possess situational dynamics similar to those of C2 work. We contend that UX designers are more in need of experiencing a curated collection of design features that exemplify good solutions to different design problems. A decision-support system built around such a collection will both sensitize designers to problems they should consider and to alternative ways in which these problems may be solved depending on the context. The system may be used for training as well as for real design projects. The key idea is to shift the focus of decision support from documentation to creative input into the decision-making process.

The second implication is that decision-support systems should approach UX design decisions as teamwork rather than individual work. We observed information exchanges among designers and also instances in which two designers used different criteria for the same decision point. Thus, a system that juxtaposes multiple designers’ considerations will support and align with current UX design practices. The system may also support activities such as timeouts, in which the designers briefly suspend their individual activities to meet and collaboratively discuss the design issues each of them are currently facing [10]. If such a system succeeds in strengthening the incentives to collaborate then it will increase the number of design criteria considered before turning to justification.

6 Limitations

Three limitations should be remembered in interpreting the results of this study. First, the two studied design teams are from the same company. Conventions, practices, and so forth may be different in other settings, such as design consultancies. Second, we acknowledge that much of decision-making is only indirectly accessible to observation. However, by observing meetings and informal conversations, we got data where the designers reasoned about their designs and design decisions [11]. Furthermore, we occasionally asked questions during the field observation and supplemented it with interviews. Third, we did not follow the two teams for the whole product development process. Observing the teams earlier or later in the process might influence the distribution of the decision-making mechanisms across the UX design layers.

7 Conclusion

This study makes three contributions. First, it shows how UX designers make decisions in practice. In addition to mental simulation and pattern recognition, UX designers also make decisions by seeking information and by using mental models. The designers’ use of these mechanisms is unevenly distributed across the different UX design layers. Second, UX designers seek information to justify their decisions as well as to reach them. In almost half of the information-seeking instances the studied designers sought information to justify decisions they had already more or less made. Third, UX designers’ mental models extend beyond their beliefs about causal relationships to also include other kinds of beliefs. The studied designers’ mental models consisted of causal relationships, empathy, and simple statements. In terms of implications, we propose that future work on decision-support systems for UX designers should focus on creative input rather than documentation and on teamwork rather than individual work.