Is It AI or Is It Me? Understanding Users’ Prompt Journey with Text-to-Image Generative AI Tools

The term "prompt engineering" was initially coined in the context of natural language processing for programming tasks [47]. It has since evolved to denote the meticulous process of selecting the right prompt to produce intended outputs with large language models [51]. Challenges related to prompting are abundant and span multiple dimensions. For instance, prompt-based prototyping for front-end interfaces involves challenges like task granularity, scoping the request, and dealing with users’ inaccurate mental models of the history and syntax of prompts [25, 26]. Liu et al. delved into the issue of abstraction matching in the context of code generation, exploring approaches to bridge differences in levels of detail and interpretation between user prompts and AI capabilities Furthermore, models’ unclear interpretation of prompts can lead to user frustration [26, 67].

Prior research on prompt-based interactions has examined factors influencing the effectiveness of prompts in large language models, such as the negative impact of lengthy instructions [42] or the role of word order in achieving accurate model responses [36]. Prompt-making tools with various design approaches like constraining vocabulary, incorporating menu-driven interfaces, and block-based programming [46] have been introduced to assist in the intent formulation as well as writing prompts [31]. On the model side, strategies such as examples, code-like formatting, and repetition have shown promise [26]. However, research on intuitive, in-the-wild prompting behaviors is still scarce. A noteworthy example is the work by Zamfirescu-Pereira et al. [68], which explores intuitive prompting behaviors with large language models with a conversation agent. Our research aims to build on this line of research by examining how users intuitively structure their prompts. In doing so, we aim to fill the gap in practical, often improvisational, methods employed by users in creative contexts.

In text-to-image tools, similarly, prompts serve as a critical means of communicating users’ ideas and intentions to the AI systems. Prior work has systematically evaluated the effectiveness of different approaches to prompting in diffusion models and proposed various prompt engineering guidelines and taxonomies [16, 32, 64]. For example, Liu and Chilton proposed a set of guidelines covering key aspects to consider when crafting prompts with text-to-image tools, such as prompt phrasing, seed, iterations, and style considerations [32]. Furthermore, Oppenlaender contributed a taxonomy of six prompt modifiers, commonly used within the online Promptist community including subject terms, style modifiers, image prompts, quality booster, repeating, and magic terms, offering additional insights into effective prompt crafting [44]. Sanchez similarly, developed a taxonomy of the semantic structure of prompts by examining Stable Diffusion prompts [50].

Recent research has increasingly focused on text-to-image tool integration into creative communities. Ko et al. focused on the visual art domain, highlighting both the design opportunities such as the need for domain-specific models and challenges like the lack of expression for describing the desired image [28]. Tholander and Jonsson investigated how generative machine learning could integrate into design activities such as ideation and sketching [57]. Additionally, Chang et al. examined the practices and motivations of text-to-image tool users in the art community. They highlighted the value that artists place on effective prompt templates and that the templates can be considered as art themselves [11]. This demonstrates a need to understand and align capabilities to user workflows.

Studies also have shown a growing text-to-image adoption in various industries like gaming, and design. Vimpari et al. [61] investigated the perception, adoption, and usage of text-to-image generation tools among game industry professionals highlighting that tools are used for inspiration, to support communication, or to compensate for lack of skills. Chiou et al. explored designers leveraging AI tools in ideation, specifically noting how word selection in prompts can influence artistic expression [12]. Furthermore, Chung and Adar [13] presented a tool for combining prompts to different areas of the artwork, outlining design compromises, and exploring socio-technical challenges associated with generative models such as ownership of the artwork.

Despite the valuable guidelines and taxonomies developed, prior studies often discuss these taxonomies in isolation from the users’ goals and motivations. We aim to bridge this gap, by taking a closer look at users’ prompt journey and their barriers, as well as the motivations behind their choices.

Interacting with generative AI tools poses unique challenges for users that can diverge from the broader human-AI guidelines often aimed at decision-making contexts [5]. Challenges of content generation with AI include the open-ended nature of generative outputs, lack of user agency, misalignment with users’ creative goals, and information overload [10, 35, 48]. These challenges are further compounded in specialized settings like public spaces [33], virtual environments [59], and game design [45], each bringing their specific challenges and requirements.

In response, frameworks specifically aimed at improving human-AI interaction in generative contexts have been developed. Grabe et al. outlined interaction patterns around curating, exploring, evolving, and conditioning for effective collaboration [20]. Similarly, Guzdial and Riedl’s turn-based co-creation model aims to structure the interaction, providing the user with a greater sense of agency in the creative process [21]. Weis et al. proposed six design principles for generative AI, emphasizing the importance of embracing imperfection, facilitating multiple outcomes, encouraging exploration and control, and providing clear mental models and explanations [62]. Furthermore, designing interactions that cater to users of varying technical skills has proven beneficial in enriching the generative AI experience [24].

Specific to text-to-image generation, Ko et al. [28] focused explicitly on text-to-image generators advocate practices like stakeholder participation, customizable and rewarding interfaces, control mechanisms to reduce risks, pedagogical scaffolding tied to learning goals, community data-driven model tuning, and validating outcomes. A similar emphasis on stakeholder collaboration is noted by Han and Cai [22]. Trovato et al. [11] suggest additional facets like embedding linked resources for aiding novelty and propose the idea of using prompt templates for exploration and validation. Additionally, Sanchez critiques existing support methods for prompting, arguing that they tend to homogenize prompting practices and fail to adapt to the diverse intentions of users [50].

Building upon existing research, our study aims to extend the discourse around supporting users’ needs at different stages of their prompt journey with text-to-image tools. We aim to highlight the nuanced ways in which individuals craft prompts for these tools, thereby improving our understanding of how to better support users’ interaction with generative AI tools.

Midjourney is a commercially available text-to-image AI tool with over 10 million users on their Discord server as of February 2023 [40], which makes it well-suited for research on how users interact with text-to-image AI tools. Midjourney is hosted on Discord, a popular social messaging platform, particularly in the gaming industry. The Midjourney Discord server offers several channels where users can generate and view images. These channels are highly active, with multiple prompts and output images appearing in almost real-time.

Midjourney offers multiple server commands that allow users to perform specific actions. For example, the command “imagine” initiates a box called prompt where users can input text. A few seconds later, a 2x2 grid appears containing four blurry images that refine and render in real-time, each representing a distinct variation of the entered prompt. Users can choose any of the four images and request new variations—which will generate four new images that are similar to the selected image, or the user may upscale one of the images to get a higher-resolution version of the image with more details.

In this research, we narrow our focus to Midjourney which relies on textual prompts and minimal additional parameter adjustments. We acknowledge the broad spectrum of capabilities and interfaces present in various text-to-image tools, including those with parameters or detailed image editing features. However, the decision to focus on Midjourney, in particular, was driven by its significant usage by the general public at the time of study in August 2022 [4, 27, 49] and the emergence of a dedicated user community around it.

We conducted 19 semi-structured interviews with Midjourney users. The interviews took place over two weeks in August 2022 and were conducted remotely.

When initiating prompts, our participants employed a variety of techniques to control the overall structure of prompts, which we categorized into five structures: Descriptive Sentences, Templates, Overview + Detail, Chunks, and Word Sequences. A summary of these structures, along with examples, is provided in Table  1.

Evaluating AI-generated images emerged as a standard practice among our participants, though we found minimal background influence on this process. Only four (P10, P12. P14, P17) out of the study’s 19 participants offered prior background influencing their image evaluation. For example, P10 mentioned, "being an artist myself, I know when something doesn’t look composed right or if it looks off or imbalanced," emphasizing the influence of artistic training on composition and balance. Similarly, P12’s background in photography led them to pay specific attention to depth of field, stating, "sometimes I’m going for a shallow depth of field, but most of the time I’m not."

While participants’ motivations and definitions of a successfully generated image varied, we identified two factors that influenced how participants approached image evaluation: the specificity of their objectives and the representational (i.e., "refers to art that represents something with visual references to the real world" [52]) or non-representational (i.e., "Work that does not depict anything from the real world" [52]) nature of their intended content. Importantly, these factors were not static. Participants’ goals could range from open-ended exploration to achieving specific outcomes, and their desired level of representation could shift between representational and non-representational based on their immediate needs and creative ideas. This adaptability highlights the dynamic nature of their engagement with Midjourney, where evaluation criteria were not fixed and evolved in tandem with their artistic intentions. Furthermore, we highlight the criteria participants often mentioned in terms of subject matter and visual design considerations to evaluate the generated image. In presenting these criteria, we view them as tools that participants employed to effectively navigate their evaluation approach. We note that the scope of our study includes a broad exploration of these criteria without linking them directly to the specific objectives or content types.

During iteration, after evaluating the output image, participants performed a diverse set of actions to modify and refine their prompts if they felt the current prompt was generating undesired results. Prompt refinement processes move participants closer to their ideal output image, often performed in steps by incrementally modifying the prompts and evaluating images. A detailed list of these prompt refinement processes can be found in Table  2.

One refinement strategy used by our participants is adding words to describe the content in more detail compared to the original prompt. During image evaluations, participants paid attention to visual qualities and subject matter in their outputs. Consequently, when specific visual elements or content were lacking in the image, participants elaborated their prompt. P18 shared an example of adding ’dark’ and ’bowl’ to refine their prompt: "I was trying to be more specific where the wood was darker. So, I said a dark brown wooden flat bowl plate." These additions reflect different prompting structure strategies (section 4.1), from adding chunks or including adjectives within a sentence.

Additionally, some participants (n=7) mentioned stepping back and omitting words as part of their refinement process. As P2 demonstrated one example iteration for us, they realized adding to the prompt was not working, so they changed their strategy: "shrinking it back down as you notice, the more I am adding, it is not helping. It is not making it better. So, I am going to go back down to simplicity." Some participants also used this strategy iteratively to understand the prompt and image relationship. For example, P17 explains that they removed a word to determine whether it contributed to the output image: "so then I said, let me take out the style of the artist and let me see if it [Midjourney] just recognizes ’Googie’ [an architectural style] in general."

Some participants (n=11) mentioned that word order, repetition, and exaggeration were effective methods to emphasize specific aspects of their prompts or accentuate a theme. Consequently, when the generated image missed or lacked something, they refined their prompt by re-ordering, repeating, or exaggerating words. By rearranging the phrases in the prompt, P2, who was experimenting throughout the interview, came closer to their desired visual qualities in a spider portrait: “having the descriptors at the front [of the prompt] and spider at the end seems to be making a difference.” While P2’s approach stemmed from their own trial and error, P12 adopted an exaggeration strategy observed from other users: "exaggerating, and that’s what I find in Midjourney [...] if you really wanted to make sure that the figure ’Bob Ross’ is going to be tall, really tall, and very, very tall or like, keep overstating that and I’ve even seen some people say that technique." In addition to these linguistic strategies, a few participants adjusted the weights as part of their refinement process. Adjusting weights can have effects similar to re-ordering or exaggerating, allowing users to subtly influence the model to focus on specific aspects of a sub-prompt (see section 4.1.4).

The degree of understanding of how parameters influence output images varied among participants. Some participants like P1 demonstrated a clear grasp, explaining how adjusting the aspect ratio impacts their generated image: "if you put a –9:16, that gives you much better portraits...But if I do it in the square format, there’s not enough [room]." Their statements llustrated an awareness of parameter effects. However, other participants, like P8, acknowledged using certain parameters without fully grasping their impact on the generated image: "you know, - -16 by 9, - - quality 25, whatever. I don’t know much difference [it makes] there or if it even really does anything."

Finally, refinement sometimes (n=16) involved leveraging Midjourney’s capability to generate diverse outputs from the same input. This technique, known as re-rolling or making variations, involves participants rerunning the same prompt, capitalizing on Midjourney’s inherent variability. P2 explained this approach, stating, "you can still put in the same prompt and it’s [Midjourney] gonna come up with something totally different." This approach was particularly useful at different refinement stages, with some, like P5, starting to explore variations after achieving a satisfactory base result: "I guess the first thing that I look for is, you know, the accuracy from my perspective of what I was after and then versioning from there" Novice users, in particular, tended to depend more on this method. For example, P3, a novice, utilized re-rolling without modifying their prompt, in hopes of achieving a more favorable outcome akin to "roll[ing] the dice and hop[ing] for a better number." As P3 explained, they used this method because they were unsure how else to refine: "so, I wouldn’t know what to add or if I needed to add pluses or take away the commas to make that work." This highlights how some participants, like P3, often rely on Midjourney’s stochastic nature to explore possibilities, especially when uncertain about how to adjust prompts effectively.

Building upon our initial inquiry, our second research question seeks to examine the challenges users experienced during their prompt journey. We surfaced two themes of challenges influencing the users’ prompt journey: challenges in aligning user intentions and generated outputs, and challenges in mastering prompt crafting knowledge.

Our findings point to a complex view of the prompt journey as both an individual and social experience. To learn about prompts our participants mostly engaged in trial and error, relied on familiar constructs (e.g., descriptive sentences), and demonstrated a strong inclination towards observational learning [37, 39]. Our participants, facing the complexities of crafting prompts and the overload of external information, turn to observing and mimicking the work of their peers, as highlighted by participants such as P4 and P16. This pattern of learning, aligning with Lave and Wenger’s [29] concept of social interactions in situated contexts, underscores the communal aspect of learning in complex environments. However, this community-based learning is not without its challenges, including uncertainty about discerning critical aspects of prompt creation. Additionally, it also raises ethical considerations about the originality and ownership of creative ideas as expressed by P8 and five others. The tension between learning from others and developing one’s unique style mirrors broader discussions in the fields of creativity and intellectual property [15].

In addition to individual prompt writing, our findings highlight the collaborative and social aspects of prompt creation in image generation with Midjourney. Practices like ‘prompt jamming’ and collective prompt development, as seen by participants (e.g., P6, P10, P13), illustrate a communal dimension to creativity where “people feed off of each others’ ideas" P13. This dynamic presents opportunities for fostering community-driven learning initiatives that discuss the "why" behind successful prompts and can enable users to develop their own effective strategies. This approach can also facilitate a more inclusive and supportive learning environment, encouraging users to share their insights and strategies more openly [6, 9].

Our findings highlighted that users approach text-to-image tools with a spectrum of goals, ranging from open-ended exploration to targeted outcomes. Participants (e.g., P9 and P10) whose exploration embraced unexpected results, while participants with defined creative aimed polished images to match their vision. We also observe the prioritization of different elements across users - some focus predominantly on overall visual style, others on precise decorative features or objects within each image. Furthermore, as shown in Figure 1, user priorities remain fluid, subject to shift even during a single iterative session. Additionally, in our study, we observed that participants do not conform to a uniform approach in crafting prompts. While drawing inspiration from community examples, they ultimately personalize and adapt their approaches based on individual preferences, goals, and assumptions about Midjourney. This suggests a tendency for users to gravitate towards prompt structures they find intuitive or effective, based on their unique intentions.

While existing text-to-image AI tools enable general-purpose image generation, they lack personalization to each user’s evolving goals. By recognizing the diversity across users and flexibility within individual users, future text-to-image systems could become more adaptive partners in realizing user-driven visions. Our findings highlight the challenge of misalignment between user goals and Midjourney’s interpretations. This often resulted in frustration, particularly when crucial elements were overlooked or unexpected visual blends occurred, as experienced by P14 and P11. The mismatch was further exacerbated by participants’ expectations of human-like comprehension from the AI, leading to confusion and prompt abandonment, as seen with P3, P5, and P16. These observations underscore the critical nature of clearly specifying goals for aligning AI systems, as emphasized by [17, 60]. Achieving alignment with natural language prompts is further complicated by the lack of rigid formal grammar, as noted by Fiannaca et al. [18]. This highlights the potential of prompt structures as tools for specifying user goals and guiding image generation. While efforts to decode the semantic structure of prompts are promising [18, 50, 66], they may not fully capture user intentions and personal preferences in more complex prompts in text-to-image generation.

Inspired by our participants, we provide two example scenarios illustrating how prompt structures could guide image generation. While these scenarios are grounded in our findings, further research is required to assess their effectiveness in accurately interpreting user intentions and generating satisfying visual outcomes:

While the AI model drives text-to-image generation, tool design plays a critical role in aligning users’ expectations with the AI’s capabilities. Our findings have shown, that participants (e.g., P3 and P5) sometimes expect Midjourney to interpret prompts like humans, leading to failed iterations. Following established human-AI interaction guidelines [5, 34, 45], text-to-image tools should explicitly communicate both strengths and limitations, emphasizing the inherent imperfection and unpredictability of results [56, 59, 63]. Predicting the need for explanations based on user behavior [14] could provide users with timely and relevant information, particularly when frustrated or confused, ultimately improving understanding and experience. Addressing specific challenges like mismatched human-AI language logic can further bridge the gap through features like prompt conversion, representative examples, and clarifying the model’s language limitations compared to humans [54].