2.1 Challenges of Designing AI Products
Advances in AI have enabled a new set of capabilities, opening up a new design space for interactive systems. However, designing AI-enabled interactions brings unique design challenges. Even simple AI applications (e.g. two-class classifiers) can make inference errors, leading to UX breakdowns and harm [
49]. A significant amount of research investigated the challenges of designing human-AI interaction, such as intelligibility [
1], explainability [
60,
98], user control [
62,
87], feedback [
89], error recovery [
41], and setting user expectations [
47,
53]. Parallel to these efforts, a growing body of work has focused on uncovering and mitigating inherent biases and potential harm of AI applications for developing fairer systems [
39,
57,
64,
94,
99]. Despite these efforts, it still remains difficult to anticipate the potential errors and harm a not-yet deployed AI system might cause [
104].
Another, complementary strand of research has investigated the practices and roles of industry practitioners who work on cross-functional teams. Researchers report that practitioners without a background in AI (e.g., designers, product managers, domain experts) have challenges in engaging data and AI [
24,
78]. Product professionals struggle to understand what AI can do, and cannot easily formulate business problems into data science problems [
54,
65,
72,
77,
78,
108]. From an interaction design perspective, AI presents unique challenges for ideating and rapid prototyping novel, implementable AI experiences [
24,
90,
104] (see [
104] for a review of AI’s design challenges). Designers mostly join projects towards the end for communicating the AI system’s output to end users, and are rarely involved in problem formulation [
21,
24,
105].
In addition to challenges specific for individual roles, there are many challenges in cross-disciplinary collaboration, mainly due to a lack of shared workflow, shared language, and shared expertise [
31,
50,
78]. Some researchers characterized this as a gap between AI and product expertise [
102,
105]: data science and AI teams struggle to elicit user requirements from product teams, and they tend to overlook how the AI system will generate value for users [
54,
72,
78]. Product teams (e.g., PMs, designers) tend to envision AI concepts that cannot be built or too high stakes to deploy an imperfect AI model [
24,
104]. Recent research investigating the high failure rate of AI initiatives shows that most AI product failures stem from miscommunications and misalignment between team members, especially in early phase problem formulation [
26,
46,
95].
A small but growing number of studies explored the emergent best practices for designing AI products. Successful practitioners formed close collaborations between data science and product teams [
54,
90,
103,
105], where product teams worked to sensitize data science and AI partners to human-centered approaches [
107]. Design teams in particular served as facilitators on AI teams to establish a shared understanding and alignment between different disciplines using interim artifacts and visualizations [
105,
107]. Several researchers highlighted that such artifacts and boundary objects [
56,
88] might play a key role in scaffolding cross-disciplinary collaboration among AI practitioners and stakeholders [
15,
90,
102,
105].
In terms of data and AI literacy, Yang et al. [
103] interviewed designers who successfully engaged
AI as a design material. These participants only had a high level understanding of AI (e.g. what a model and label is), and they worked with abstractions of AI capabilities and value propositions (e.g. predicting user intent to surface relevant actions). Another study found that design teams created internal AI resources, such as an AI capability matrix and a collection of examples, for educating themselves and engaging their clients in early phase ideation and problem formulation [
105]. Similarly, studies on data science and AI practitioners revealed that they worked to sensitize their cross-functional partners, such as PMs and domain experts, to AI concepts [
54,
72,
78]. The way they worked resembled design practice [
54]; they probed their collaborators to get at the underlying problem, to frame and reframe it as a well-defined data science problem [
54,
72,
76]. Finally, research showed that effective AI teams had an approach similar to startups [
83], where they would focus on a minimum viable product at the intersection of what is feasible with AI models and data, and what is desirable and viable for customers and users [
96,
103,
105,
107].
Current research efforts to aid practitioners in designing human-AI interaction broadly fall into four themes. First, researchers created resources to improve non-technical stakeholders’ literacy in data science and AI concepts [
37,
52], however there is no consensus on what constitutes a
good enough technical understanding [
104]. Second, researchers developed prototyping tools and design-oriented data exploration workflows with little or no code to allow first-hand experimentation with data and AI [
17,
27]. Third, researchers created resources, guidelines, and design patterns to provide best practices for addressing AI’s design challenges [
4,
5,
42,
75]. Fourth, researchers explored data and AI through design-led inquiry to share first-person accounts, as well as emergent solutions such as new methods, boundary objects and metaphors [
8,
15,
23]. Reflecting on these, some researchers argued that AI might require design processes beyond user-centered design; for instance, processes such as matchmaking [
11] that seek for potential value in existing AI systems and datasets [
20,
29,
101,
107].
2.2 Human-AI Interaction Guidelines
HCI literature has a rich history of providing design principles to improve the user experience and usability of interactive systems, typically in the form of guidelines [
6,
9,
41,
67,
87], checklists [
44,
81], and design patterns [
12,
18,
92].
Prior work investigating the application of guidelines in practice showed that practitioners often consider guidelines useful yet face challenges in selecting, prioritizing, or translating them for their work [19, 71, 91].Recently, building on the large body of research on interaction with intelligent systems, technology companies have proposed principles, guidelines, and design patterns for human-AI interaction to aid product professionals in designing AI products (e.g., [
4,
5,
42,
75]). These resources cover a comprehensive list of design considerations, such as explainability (e.g. “make clear why the system did what it did” [
4]), control (e.g. “give people familiar, easy ways to make corrections” [
5]), and feedback (e.g. “let users give feedback” [
75]). In addition to general guidelines, some resources focus on specific interactions (e.g. voice interactions [
3], chatbots [
68]) or considerations (e.g. ethics [
43]) (see a review in [
7]).
While there are more than a dozen human-AI guidelines, little work studied how practitioners use these guidelines and resources. Prior research focused on validating the effectiveness of guidelines in improving the user experience [
59], or studying guidelines comparatively to reveal the landscape of considerations [
45,
100]. A study mapping existing guidelines and resources to current product development processes pointed out that AI guidelines mostly help with late phase efforts (e.g. prototyping), and that resources for early phase are scarce (e.g. ideation) [
104]. Recently, a few researchers investigated how specific resources and guidelines are used by industry practitioners, such as fairness toolkits [
22] and conversational UI guidelines [
51]. In the same spirit, this work investigates if and how human-AI guidelines in [
75] are used, which we briefly introduce below.