3654777.3676399

ProtoDreamer: A Mixed-prototype Tool Combining Physical
Model and Generative AI to Support Conceptual Design

Hongbo Zhang Pei Chen∗ Xuelong Xie
Zhejiang University Zhejiang University Zhejiang University
Hangzhou, China Hangzhou, China Hangzhou, China
hongbozhang@zju.edu.cn chenpei@zju.edu.cn xuelongxie@zju.edu.cn
Chaoyi Lin Lianyan Liu Zhuoshu Li

Zhejiang University Zhejiang University Zhejiang University
Hangzhou, China Hangzhou, China Hangzhou, China
chaoyilin@zju.edu.cn liu_lianyan@zju.edu.cn lizhuoshu@zju.edu.cn
Weitao You Lingyun Sun

Zhejiang University Zhejiang University
Hangzhou, China Hangzhou, China
weitao_you@zju.edu.cn sunly@zju.edu.cn
Figure 1: A prototype process supported by ProtoDreamer. (a). A designer conceived and prototype the design of a beach
vehicle through embodied interaction with tangible objects and physical materials. (b). The designer utilized ProtoDreamer to
rapidly explore high-fdelity schemes. He captured a image of the physical model and inputted design requirements through
voice: “This is a Cyberpunk-style beach vehicle, the green part is the light and the blue part is a cockpit with transparent glass ”. (c).
Upon generating a satisfactory candidate scheme, the designer proceeded to mask local areas for refnement and optimization.
(d). After the prototype has been realized through 3D modeling and printing as the frst version, the designer further added
physical materials on the frst version model for rapid, on-site iteration via ProtoDreamer. (e). ProtoDreamer’s user interface.
∗ The
ABSTRACT
corresponding author.
Prototyping serves as a critical phase in the industrial conceptual
Permission to make digital or hard copies of all or part of this work for personal or design process, enabling exploration of problem space and identif-
classroom use is granted without fee provided that copies are not made or distributed cation of solutions. Recent advancements in large-scale generative
for proft or commercial advantage and that copies bear this notice and the full citation models have enabled AI to become a co-creator in this process. How-
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or ever, designers often consider generative AI challenging due to the
republish, to post on servers or to redistribute to lists, requires prior specifc permission necessity to follow computer-centered interaction rules, diverging
and/or a fee. Request permissions from permissions@acm.org.
from their familiar design materials and languages. Physical proto-
UIST ’24, October 13–16, 2024, Pittsburgh, PA, USA
© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. type is a commonly used design method, ofering unique benefts
ACM ISBN 979-8-4007-0628-8/24/10 in prototype process, such as intuitive understanding and tangible
https://doi.org/10.1145/3654777.3676399
UIST ’24, October 13–16, 2024, Pitsburgh, PA, USA Zhang et al.
testing. In this study, we propose ProtoDreamer, a mixed-prototype communication modes. To overcome these challenges, AI research
tool that synergizes generative AI with physical prototype to sup- has explored various controlled conditions for generative mod-
port conceptual design. ProtoDreamer allows designers to construct els, including pose [69], semantic map [9, 80], bounding box [82],
preliminary prototypes using physical materials, while AI recog- etc. These approaches ofer more ways for designers to concretely
nizes these forms and vocal inputs to generate diverse design al- communicate with AI, but most of them still require to conform
ternatives. This tool empowers designers to tangibly interact with computer-centered rules. For example, designers need to utilize
prototypes, intuitively convey design intentions to AI, and continu- semantic maps according to correspondence color rules. In this con-
ously draw inspiration from the generated artifacts. An evaluation text, HCI research is increasingly focusing on designer-centered
study confrms ProtoDreamer’s utility and strengths in time ef- interactive AI. For example, in PromptPaint [12], the interaction
ciency, creativity support, defects exposure, and detailed thinking between artists and their painting mediums is emulated through
facilitation. a graphic panel to assist in mixing prompts to express fuzzy con-
cepts. Popblend [71] adheres the divergent and convergent thinking
CCS CONCEPTS in generative process to achieve inspiration stimulation. The HCI
• Human-centered computing → Interactive systems and community strives to enable designers to collaborate with AI using
tools; • Applied computing → Computer-aided design; • Com- their familiar design material, language, and procedure, moving
puting methodologies → Artifcial intelligence. beyond computer-centered interaction rules.
Physical prototype is a universal and common design material
KEYWORDS used by designers in the industrial conceptual design stage [33]. It
leverages tactile materials such as pasteboard and clay for rapid
prototype, creativity support, generative AI, large-scale model
and tangible ideation, aiding designers in embodied sense-making
ACM Reference Format: and swift structuring [24]. Cognitive studies afrm that body move-
Hongbo Zhang, Pei Chen, Xuelong Xie, Chaoyi Lin, Lianyan Liu, Zhuoshu ments are integral to thinking, with interactions with tangible
Li, Weitao You, and Lingyun Sun. 2024. ProtoDreamer: A Mixed-prototype models boosting creativity through body-based cognition [24, 73].
Tool Combining Physical Model and Generative AI to Support Conceptual Additionally, design studies highlights the distinct advantages of
Design. In The 37th Annual ACM Symposium on User Interface Software and physical prototype method in conceptual design: it ofers richer
Technology (UIST ’24), October 13–16, 2024, Pittsburgh, PA, USA. ACM, New
sensory stimuli [6, 23], improves understanding of design prob-
York, NY, USA, 18 pages. https://doi.org/10.1145/3654777.3676399
lems [34, 47], and facilitates team communication [8]. The tan-
gibility of physical prototype is especially valuable in function
1 INTRODUCTION and structure testing, exposing defects and highlighting unrealistic
The conceptual design is a fundamental and essential stage in the prototype assumptions [29]. Given these inherent advantages of
industrial design process [24]. Designers embark on transforming physical prototype, there is considerable interest in mixed tools that
vague notions into concrete ideas by delineating the product’s over- combine virtual and physical space, despite the growth of digital
arching outline and layout in conceptual design [83]. Prototyping is design support tools [5, 62]. These tools aim to integrate the intu-
critical in the industrial conceptual design process, ofering design- ition and tangibility of physical prototype with the high-precision
ers a way to reduce mental loads and augment creativity through simulation capabilities of digital prototyping, thereby harnessing
iterative representations that evolve over time [78]. This practice their collective advantages of both domains [66].
has proven benefcial for exploration of design space [17, 26], iden- This study introduces ProtoDreamer, a mixed-prototype tool in-
tifcation of design problem [28, 76], supplementation of design- tegrating physical models with generative AI to support industrial
ers’ mental model [49, 68] and discovery of unexpected phenom- conceptual design. Figure 1 presents a prototype process applying
ena [31, 42, 48]. ProtoDreamer. Supported by the proposed tool, designers can en-
Enhancement of Artifcial Intelligence (AI) in generative capa- gage with tangible materials for embodied ideation and concretely
bilities is transforming the paradigm of prototype and conceptual articulate design intention. Subsequently, design schemes aligning
design. Large-scale text-to-image generative models such as Mid- with given intention can be generated, which allows to be further
journey [13], Stable Difusion [53], and DALL-E [51] have shown regenerated and refned locally. On the one hand, ProtoDreamer
profciency in applying common-sense knowledge and providing supports designers to express design intentions to AI with familiar
design schemes like human designers [71], enabling AI to be a design language and intuitive interaction, enhancing the shared
co-creator in prototyping process [15, 22]. This AI involvement in mental model between human designers and AI agents [22]. On
prototyping expands human designers’ logic and scale due to its the other hand, designers derive conceptual design inspiration not
generative variability [20, 72], thus showing potential in supporting only from embodied interaction with physical models but also from
creativity in conceptual design [56]. However, the challenge re- the observation of incessant and random generated artifacts.
mains for AI to efciently and accurately comprehend designers’ in- In this study, a formative study was constructed initially, and 20
tention while generating outputs that align with their expectations. designers was invited to conduct conceptual design tasks in a mixed-
Text-guided generative models necessitate clear text descriptions prototype environment. Their attitudes toward the integration of
from designers to express their ideas, yet converting ambiguous physical materials and generative AI were gathered. The fndings
design intentions into precise text remains difcult. Additionally, served as essential guidance for the design and development of
simple text prompts often fall short of capturing what designers ProtoDreamer. An evaluation study with another 20 designers was
imagines in their mind [12], underscoring the need for more natural also conducted, which proved the usability and creativity support
ProtoDreamer UIST ’24, October 13–16, 2024, Pitsburgh, PA, USA
of ProtoDreamer in industrial conceptual design. The results indi- sensemaking, reducing cognitive load and stimulating creativity.
cated that ProtoDreamer presented distinctive strengths including The embodied interaction utilizes interplay between mental and
intuition, time efciency, and strong expressiveness. This paper physical representations to facilitate the idea exploration and re-
makes following contributions: fnement [24, 67]. Second, the physical model increases the number
of sensorial stimuli, such as touch, sight, and smell [6]. It makes
• Revealing designers’ challenges in the mixed-prototype en-
the design environment more intuitive and interactive, promoting
vironment combining physical model and generative system,
reasoning and facilitating the search for better solutions [24]. Third,
and proposing design goals for developing mixed-prototype
the tangible quality of physical prototype supports function test and
tools through formative study.
defect exposure. Prototype assembly and dissection is considered
• Building a novel mixed-prototype system supporting indus-
a deep-level interaction, providing broader visual cues, enhancing
trial conceptual design. ProtoDreamer allows designers to
structure and mechanism understanding [63]. Owing to the irre-
intuitively communicate with generative AI and concretely
placeable benefts aforded by physical prototypes in conceptual de-
convey design intention in their familiar design language,
sign stage, there is a burgeoning exploration into mixed-prototype
fostering the shared mental models in co-creation system.
tools that integrate tangible models with digital technologies. The
• Conducting evaluation study, verifying the efectiveness of
HCI community attempted to augment traditional physical pro-
ProtoDreamer in industrial conceptual design. Moreover,
totype through integrating digital ability, such as version control,
we discussed the infuence of the mixed-prototype method
accurate editing, complex calculation, and digital simulation [29].
combining physical model and generative AI on industrial
For example, Monteiro et al. [45] developed TeachableReality, pro-
conceptual design and its application space.
totyping tangible augmented reality with everyday objects. Suzuki
et al. [60] presented RealitySketch, a sketching system which com-
2 RELATED WORK bined digital sketch with physical object in real world. Tao et al. [62]
2.1 Generative AI in Conceptual Design devised a novel mixed system that enables designers to prototype
Generative AI refers to computational techniques that automately through a physical printing pen alongside fne guidance ofered
create new, plausible media [46]. Recently, generative AI is pro- by tablet computers. In this paper, we endeavor to synergistically
gressively becoming a collaborator with human across various integrate the traditional physical prototype method with generative
creative design domains, including but not limited to industrial AI in a mixed-prototype tool, combining their collective strengths
design [22, 38], graphic design [64], diagram design [50], UI de- to support industrial conceptual design.
sign [55], urban design[14], and fashion Design [74]. Generative AI
brings distinct advantages to industrial conceptual design process. 2.3 Generative AI-assisted Design System
Specifcally, the participation of generative AI supports creativity Guidelines
in conceptual design [56]. As generative AI produces artifacts as
output, rather than focusing on decisions, labels, classifcations, With the development of large-scale models, HCI researchers have
or decision boundaries [72], generated outputs opens an imagi- endeavored to formulate human-AI interaction guidelines aimed
nary space through the infnite generation, efectively expanding at mitigating the interaction complexities while concurrently en-
designer’s creative exploration boundaries [20]. Kirsh [32] also in- hancing their usability [2, 37, 72, 79]. Justin et al. [72] put forward
dicated that the cognitive velocity and depth of the design process six principles according to generative AI characteristics. They in-
may be enhanced when designers shift their observational focus dicated that generative systems should enable to explore a space
from their own actions to those executed by computational systems, of possible outcomes based on users’ query, provide version man-
which might transcend their inherent thinking logic and scale. In agement to track outputs and corresponding parameters, and allow
addition, as generative AI systematically produces candidate design both users and AI to edit candidate artifacts, etc. Amershi et al. [2]
schemes in real-time. [65], its participation enhances prototype also proposed 18 generally applicable design guidelines for human-
efciency. While generative AI has introduced subversive possibili- AI interaction. They pointed out that it is essential for AI system to
ties, designers often perceive it as a complex design medium due ignore undesired services, engage in disambiguation, and encour-
to challenges in communication with AI [16, 77]. In this context, age granular feedback. Some studies also explored how to interact
enhancing the usability of AI-assisted design tools hinges on en- with generative AI efectively by prompting engineering. Liu and
abling designers to engage with generative AI both intuitively and Chilton [37] presented design guidelines which can help users pro-
efciently in their familiar way. duce better outcomes from text-to-image models. Zamfrescu et
al. [79] explored how to use natural language to steer generative AI
and proposed prompting strategies. However, owing to the nascent
2.2 Physical Prototype and Mixed-prototype in nature of generative AI, limited endeavors have employed the sug-
Conceptual Design gested interaction principles within practical systems. In this paper,
The physical prototype is a commonly used method in industrial a generative system was designed and developed in accordance
conceptual design. It serves as an unambiguous representation of with the established guidelines. We concerned “How can designers
design concepts and is employed for various purposes like explor- better use AI” and “How can AI better understand designers”, which
ing shapes, communicating ideas, and verifying functional struc- includes helping designers to interact with AI through their habits
tures [29]. The physical prototype has distinct merits in support- while helping AI agent to comprehensively understand designers’
ing conceptual design. First, the tangibility promotes embodied creation intention, respectively.
Figure 2: (a) The constructed mixed-environment for formative study. 1: provided physical materials and tools; 2: physical
model made by participant; 3: UI of developed system for formative study; 4: a camera capturing the physical model during
prototype process; 5: a video recorder recording the design process. (b) The developed system based on Stable Difusion web UI
for formative study. 6: prompts area; 7: captured image; 8: parameter area controlling the generation; 9: design scheme.
3 USER STUDY 1: A FORMATIVE STUDY environment efectively facilitate conceptual design? 2) What are the
In this formative study, we constructed a mixed-prototype envi- challenges in conceptual design under support of the mixed-prototype
ronment that integrates physical prototype and generative AI. We environment? Three researchers in this study analyzed all tran-
conducted an experiment with 20 designers to assess their attitudes scribed texts using afnity diagrams [44], extracting challenges and
and natural workfows in prototyping, aiming to identify challenges requirements.
and requirements in such a mixed setting.
3.1 Participants
3.4 Findings
Twenty designers (12 female and 8 male, aged 25 to 30) with indus-
Nearly all participants acknowledged the advantages of the mixed-
trial design background were recruited. Eleven participants hold
prototype environment. Employing thematic analysis, three re-
professional positions as designers, whereas nine are currently pur-
searchers aimed to gather positive perspectives and attitudes to-
suing design education. All participants possess at least three years
ward its implementation through a hybrid process of deductive
of practical experience in design. They are familiar with physical
and inductive coding [61]. After resolving disagreements through
prototyping methods and generative AI tools, thus they can focus
discussion, 10 codes related to the advantages of proposed mixed-
on design tasks rather than tool learning.
environment were obtained: Easy to prototype, Time-efciency, High
fdelity, High feasibility, Intuitive perception, Structured thinking,
3.2 Experimental System and Environment
Detailed thinking, Defects exposure, Creativity support, and High
To conduct the formative study in a mixed-prototype setting, we accomplishment. These fndings corroborate the participants’ fa-
established a laboratory environment (Figure 2a) and also built a vorable views on the mixed-prototype environment and system,
system upon the open-source Stable Difusion web UI [3] (Figure thereby providing a foundation for further research. Moreover, we
2b). The Stable Difusion v2.1 [58] was adopted in the system. Our also summarized designers’ challenges(C) during design process.
setup included a computer and a camera within a controlled space C1: I need more mobility and portability. Four participants
stocked with diverse physical materials. During the prototype phase, (P2, P7, P11, P12) mentioned that the fxed camera limits the viewing
designers could create and test appearance and structure of design angle of image capture. Four participants (P1, P7, P10, P19) indicated
concept through physical materials. They can also convey design that the fxed environment with large equipment limits the spatial
intention to generative system through captured images and typed freedom of prototyping. Therefore, adopting portable and movable
prompts to cooperate with the AI agent. equipment to improve the mobility of the mixed-prototype system
is essential to expand its application scope.
3.3 Task and Procedure C2: I cannot convey my design intention in detail to AI. Par-
In the formative study, participants engaged in a 60-minute concep- ticipants mentioned challenges in conveying design intention in
tual drone design task within a mixed-prototype environment. We detail through physical models, especially the design details (P1,
provided a specifc design theme to allow designers to have a similar P13, P12). The accurate comprehension of a designer’s intention
ability to prototype as they usually conduct in real cases [35]. After by AI agent and subsequent controlled generation that aligns with
the prototyping task, a semi-structured interview was conducted. this intention, are crucial in human-AI co-creation. Frequent mis-
We focused on two main questions: 1) Can the mixed-prototype understandings from AI agent might reduce users’ trust, patience,
Figure 3: The fow chart using ProtoDreamer. Thin arrows indicate the fow sequence and thick arrows indicate the information
transmission between designers and AI system.
and confdence of human-AI cooperation. For example, three par- DR2: Enabling designers to convey design intention in detail
ticipants (P6, P10, P12) indicated that they do not want to continue based on physical materials. Enabling designers to correspond
cooperation when AI tampers with their design many times. specifed model details with local requirements or restraints, thus
C3: I face challenges in facing prompts and parameters. achieving detailed intention transmission and generation control.
Participants complained that they spent a lot of time and energy DR3: Simplifying interaction and reducing the difculty in
on tool using rather than design. Specifcally, Participants reported adjusting prompts and parameters. Optimization of the human-
they are “at a loss” when facing complex parameters (P2) and “don’t AI interaction exempts designers from the minutiae of parameters
know how each parameter afects design” (P13). P7 mentioned chal- and nebulous prompts, allowing them to concentrate on the con-
lenges in writing prompts, especially prompts related to design ceptual design and selection of outputs that meet their approval.
quality and style. Participants emphasised that they “care more DR4: Supporting local optimization and editing based on
about design outputs than the prompts and parameters” (P8, P10, generated outputs. Supporting designers to continue to optimize
P13). It is crucial to reduce the difculty of prompt writing and the part of the generated scheme, including regenerating and ad-
parameter adjustment through the interaction that designers are justing the style, proportion, or angle.
familiar with, which might enhance the AI usability.
C4: I want to modify generated schemes further. Some partic- 4 PROTODREAMER
ipants (P7, P14, P19) expressed contentment with the overall quality
4.1 Interaction Flow
of the output yet conveyed dissatisfaction with specifc elements,
expressing a desire for localized regeneration. P14 and P16 hoped We provide a user journey to introduce the capabilities and inter-
to adjust the shape, angle, and proportion of the generated design action fow of ProtoDreamer (Figure 3). A designer conducted a
scheme. P20 indicated that “Local editing and regenerating is not an conceptual design for the innovative drone. At frst, the designer
alternative to editing the generated results in Photoshop, but a data- didn’t have any ideas, so she built some rough physical materi-
driven detail optimization in my design”. By implementing localized als casually, exploring the proportion and structure of the drone
constraints, the generated output can be more refned and compre- through embodied interaction with these materials (Figure 3a). She
hensive, and better aligned with the designer’s anticipations [56]. conveyed design intention to AI by designated colors and corre-
sponding semantics (Figure 3b~c). After AI understood her intention
and generated corresponding images (Figure 3d~f), she obtained
and browsed several candidate schemes immediately and chose a
3.5 Design Requirements satisfactory one (Figure 3g). Then she attempted to further refne
In response to the fndings from the formative study, we summa- the output for local amplifcation through adding anchors and ar-
rized four design goals for developing a mixed-prototype tool. These rowheads (Figure 3h~i). Under the support of ProtoDreamer, she
goals not only address the specifc needs identifed by participants explored the basic structure of the prototype by constructing and
in the formative study but also correspond to established principles testing the tangible model, while concurrently drawing detailed
in human-AI interaction research [2, 37, 72, 79]. inspiration from her interactions with the generative system.
DR1: Integrating the mixed-prototype system on a mobile
device to enhance portability. Improving mobility and portability 4.2 Detailed Design
by integrating the mixed-prototype system into mobile devices, 4.2.1 Implementing a Web Application Compatible with Mobile
thus enhancing the application scope of the prototype tool. Devices (for DR1). ProtoDreamer is designed as a web application
a 3 3
b
Create Scheme Refine Scheme Create Scheme Refine Scheme
Prompt
a realistic drone, modern style
7 Mask Local Area
6
the red part is a solar panel, and the green Local Refine Prompt
part is light strips
There are solar panels on both sides.
4
Scheme Style 8
Add Refine Anchor Delete All Anchor
9
Generate
5
2 Generate 10
1 1
Figure 4: The user interface of ProtoDreamer. (a). The Create Scheme Page. 1: Prototyping Canvas; 2: Capturing Button; 3:
Functional Navigation; 4: Text Prompt Area; 5: Scheme Style Folder. (b). The Refne Scheme Page. 6: Add Mask Button; 7: Brush
Adjustment Tool; 8: Text Prompt for Local Mask Area; 9: Add or Delete Anchor; 10: Anchor and Arrowhead for Editing.
for accessibility on mobile devices such as tablet computers. The and enhance the continuity of the segmented feature, morphologi-
built-in rear cameras of these devices facilitate direct capture of cal dilation is employed. Firstly, a structuring element, commonly
physical models The user interface of ProtoDreamer is presented a basic shape like a square or circle, is determined. The dimension
in Figure 4, containing two modes for global and local generation. and form of this element dictate the extent of the expansion. Sec-
In the “create scheme” mode (Figure 4a), there is a canvas to present ondly, the binary mask undergoes an expansion process using the
captured image and show the design schemes generated by AI. predefned structural elements, enlarging the mask’s white region.
Upon achieving an initial satisfactory scheme, users may opt to Sequential Inpainting. Based on the obtained binary mask,
select “refne scheme” for further local refne (Figure 4b). regions designated by diferent colors are generated one-by-one
using the inpainting model of ControlNet [81]. In addition, the text
controlling the inpainting-generated content is extracted from the
4.2.2 Enabling Detailed Generation Control via Colors of Physical user-provided overarching prompt.
Materials (for DR2). In the mixed-prototype setting, designers efec-
tively convey the general form and structures via physical models
and textual descriptions, but AI still faces challenges in compre- Algorithm 1: Iterative Sequential Inpainting to obtain
hending these cues. For example, designers might use ad hoc paper Matrix-based Display of Generative Outcomes.
cutouts to symbolize components like “charging ports” or “indi- Input: Initial image �� ; color-designated regions � 1 and � 2 ; local
descriptions � 1 and � 2 for � 1 and � 2 respectively; � control weight
cator light” during design. Current AI approaches process images parameters {� 0 , � 1 , · · · , ��−1 } .
and textual inputs separately, resulting in suboptimal localization Output: Candidate scheme matrix A ∈ R�×� .
and interpretation of specifc design elements in physical models. 1 for � from 0 to � − 1 do
�
ProtoDreamer addresses this by allowing the nuanced specifcation 2 ��− 1 ← �� (�� , � 1 , � 1 , �� ) ,where � denotes time step during
difusion process
of design intention through color-based physical materials. Unlike // Inpaint the color-designated region � 1 based on the
the fxed color-content relationships in semantic mapping methods, local description � 1
ProtoDreamer lets designers freely associate colors with specifc 3 for � from 0 to � − 1 do
� �
requirements, thereby enriching structured thinking and precise in- 4 ��−2� ← �� (�� , � 2 , � 2 , � � ) // Inpaint the
color-designated region � 2 based on the local
terpretation. We realized this via a two-step extraction of specifed description � 2
color regions and a sequential inpainting procedure (Figure 5). 5 end
Region Segmentation based on HSV Color Space. The ex- 6
� �
add ��−2� to A
traction of designated color regions is performed within the Hue, 7 end
Saturation, and Value (HSV) color space. Captured images of physi- 8 return candidate scheme matrix A
cal models are converted to HSV, facilitating color diferentiation
based on chromaticity and luminance. Following this conversion,
predefned ranges for HSV are applied to segment the targeted color 4.2.3 Providing Matrix-based Generation to Present the Influence of
regions. This isolated specifc color region becomes the foreground, Weighting Parameters (for DR3). Both prior research and our forma-
while the remainder forms the background, yielding a binary mask tive study reveal challenges faced by designers in predicting AI re-
that highlights the area of interest. sponses to parameters adjustments [22]. In response, we introduce
Morphological Dilation. The initial binary mask may present a matrix-based generation and display mode. This method vividly
noise, manifesting as small gaps or holes. To address this problem illustrates generative outcomes under varying weight conditions,
Figure 5: Technical architecture diagram of realizing detailed generation control via colors of physical materials, including a
two-step extraction of specifed color regions and a sequential inpainting procedure.
to an output more closely aligns with the local prompt within the
masked area. For instance, the third image in the second row shows
the result of generating a solar panel at � 1 with a control weight
� 3 = 1.0 followed by the creation of a light at � 2 with � 2 = 0.85.
Our system recommends employing two colors to signify diferent
semantic intentions in a prototype. For designs requiring additional
localized function modules, the method detailed in section 4.2.5
may be utilized.
4.2.4 Providing Prompts Style Folders (for DR3). Current text-to-

image generation models necessitate designers to acquire special-
ized prompt engineering skills for obtaining high-quality design
schemes. This often involves providing not only basic design de-
scriptions but also various abstract terms and style references, such
as “pure background”, “high quality”, or “multi-details”, to enhance
generation quality. In light of this, a plethora of guidelines and
tips have been developed within the creative community, aimed
at assisting designers and artists in crafting efective prompts that
align with logic and rules of AI systems. Most guideline advocate
Figure 6: A matrix-based presentation example to demon- for a structured approach to prompt input. However, a key element
strate the infuence of weight variation for two separate re- in improving human-AI cooperative systems is to align AI with
gions on the overall design schemes. the user’s natural workfow, rather than adhering to a computer-
centered logic. To address this, we design “scheme style folder”
within ProtoDreamer (Figure 4 a5). This function ofers an array of
thereby ofering designers with detailed control and immediate unpre-selected prompts alongside stylistic images, akin to a range of
derstanding of the efects of parameter changes. It aims to relieve camera flters. Designers can thus efortlessly produce stylistic ac-
designers from the burden of meticulously considering the specifc curate proposals with minimal input or even a single keyword. This
meanings and infuential scopes of diverse parameters during the designer-centered approach supports designers in goal-oriented ex-
prototyping process. Consequently, designers can redirect their ploration instead of opening the black-box aimlessly [22]. Examples
focus entirely to the creative aspects of design. Specifcally, we of design schemes generated using the style folder are shown in Fig-
present results underscoring the varying impact of control weights ure 7. The number and types of candidate styles can be continuously
across color-designated regions on overall design quality. This is updated and customized by users.
achieved through iterative sequential inpainting, each iteration ap-
plying distinct control weights to designated regions (see Algo.1). 4.2.5 Supporting Local Modification and Editing (for DR4). Through
In Figure 6, we show how weight variations in two distinct re- the aforementioned steps, designers obtain a series of candidate
gions infuence the design outcome. Control weights, represented design schemes. Suitable options can be selected and subjected to
as {� 1, � 2, � 3 } are adjusted in each iteration of ControlNet. As the localized modifcations by clicking the “refne scheme” button (Fig-
weight value of increases, the inpainting efect intensifes, leading ure 4b). These changes might encompass the incorporation of new
Figure 7: A drone design using the generation control of style

folder for an example.
Figure 8: ProtoDreamer’s refne ability.
design elements or the tweaking of existing geometries. We intro-
duce two modifcation modes, local regeneration and local detail
adjustment, supported by ControlNet [81] and DragDifusion [57],
to the user interface. ProtoDreamer employs the pre-trained image-
respectively. The designer initiates adjustments by targeting a spe-
to-image difusion model, Stable Difusion XL 1.0 [59] to achieve
cifc area and inputting local requirements or constraints, or by
initial image generation. For sequential inpainting and localized
altering its dimensions and orientation via anchor points and move-
regeneration, the system utilizes the ControlNet inpaint model [39],
ment arrows (Figure 8).
which is compatible with Stable Difusion v1.5. Furthermore, the
system integrates DragDifusion [57] for precise local detail adjust-
4.3 Design Iteration
ments. The developed ProtoDreamer is hosted on a local server
We conducted a series of pilot tests to interactively enhance the sys- equipped with a GTX 3090 GPU.
tem design. First, we integrate voice input functionality to simplify
operation. Participants mentioned that voice interaction allowed
for hands-free operation, thus enabling a more immersive interac-
5 USER STUDY 2: AN EVALUATION STUDY
tion with physical materials. Second, we increase functional notes We address the following research questions in evaluation study.
aligned with designers’ cognition. We add intuitive notes to de- RQ1: Is ProtoDreamer usable? How does ProtoDreamer support
signers to clarify the functionality of each interactive component conceptual design?
and how to respond after operation. Third, we enable designers to RQ2: Does ProtoDreamer efectively enhance creative ideation
adjust the infuence of physical models on generation results. We during the prototyping phase?
observed that designers expect AI to play varied roles throughout RQ3: What advantages does ProtoDreamer ofer over alternative
the conceptual design phase. Specifcally, when a designer’s cre- prototyping tools?
ativity reaches an impasse, they look to AI to act as a creator for
inspiration, generating schemes distinctly divergent from physical 5.1 Participants
models. While when the designer has specifc concepts, they expect We recruited another 20 designers (8 males and 12 females, average
AI as an executor to refne a scheme based on physical models. Con- age is 28.8) with backgrounds in design. To ensure these individu-
sequently, we design an interactive component to enable designers als have enough insight in evaluation study, we mainly recruited
to modulate the infuence of physical models on generated schemes professional designers and senior design students (MA and Phd)
according to targeted needs. who had a minimum of three years experience. Participants in this
study were recruited through online publicity and design forum.
4.4 Implementation We also screened participants who familiar with prototyping skills
ProtoDreamer is developed based on Stable Difusion Web UI [3], a and generative AI tools so that they could focus on exploring de-
Gradio library-based browser interface designed for Stable Difu- sign rather than on tools learning. All participants signed a consent
sion [53]. The back-end server of ProtoDreamer invokes various form approved by our institution. This study has no other ethical
difusion models to generate images, which are then relayed back or privacy impacts.
Figure 9: The presentation of the designer’s satisfactory design outputs under the support of ProtoDreamer. The note P3-A3
meas that this is the 3rd iteration of the 1st design concept made by P3.
5.2 Task and Procedure support of Protodreamer, two researchers reviewed the design pro-
Each participant underwent an individual orientation, beginning cess video with participants after the prototyping period, observed
with a 5-min informed consent process and a subsequent 20-min what they said when thinking and did while working. Researchers
demonstration. This preliminary phase aimed to acquaint them with coded the design process and resolved diferences through discus-
ProtoDreamer and the Think-aloud method. This was followed by a sion. To analyze the design output of ProtoDreamer, we paid special
one-week prototyping period. Participants had unrestricted access attention to the captured physical model image and the generated
to ProtoDreamer, enabling them to explore various design concepts scheme extracting from the execution log of the authoring process.
without limitation on time or the number of ideas pursued. The use
of Think-aloud method was a requisite during the design process.
Following the prototyping phase, participants reviewed their design 6 RESULTS AND FINDINGS
schemes and prototyping video footage with the researchers. Then,
they were asked to complete several questionnaires and engaged
6.1 RQ1: ProtoDreamer Is Highly Usable in
in interviews. Supporting Industrial Conceptual Design.
6.1.1 Overall Design Outputs in the Evaluation Study. Participants
learned the functions of ProtoDremer and applied them in the
5.3 Data Collection and Analysis prototype process without difculty. The accumulated total usage
Our study entailed a comprehensive collection of participants feed- time was approximately 26 hours and 39 design concepts were
back, including video recordings of screen and workspace activity, produced. The average prototyping duration of each design concept
execution logs, design outcomes, questionnaire, and interview. To and each iteration were 40 min. Figure 9 shows some prototypes that
evaluate the objective feedback, the System Usability Scale (SUS) [7] participants were satisfed with. Both physical materials (Figure 9
and Creativity Support Index (CSI) [19] were applied to measure a, b, e, f), and commonplace objects from working surroundings
the usability and creativity support of ProtoDreamer and Stable (Figure 9 c, d) were used by designers to complete the prototyping
Difusion WebUI. Specifcally, CSI includes six factors: Collabora- processes, lowering the threshold of prototyping and bringing more
tion, Enjoyment, Exploration, Expressiveness, Immersion, and Results possibilities for creation.
Worth Efort. Similar to previous research methodologies [52], our
collaboration index quantifed the level of human-AI cooperation
instead of human-human cooperation. All interviews in the evalua- 6.1.2 System Usability Scale (SUS) of ProtoDreamer. The SUS re-
tion were transcribed. Two researchers discussed the transcribed sults are presented in Figure 10. The average total SUS score of
text together. Besides, to study the interaction process under the ProtoDreamer was 82.50 (�� = 6.12). The acceptability of the SUS
Table 1: CSI results of ProtoDreamer in the evaluation study.
Avg. Factor Avg. Factor Avg. Weighted

Factor
Counts (SD) Score (SD) Factor Score (SD)
Collaboration 2.50 (0.81) 17.40 (1.11) 43.50 (14.91)
Enjoyment 0.70 (0.46) 16.20 (1.33) 11.20 ( 7.41)
Exploration 3.20(0.75) 15.90 (1.97) 51.10 (15.32)
Expressiveness 4.10 (0.54) 17.90 (1.22) 73.60 (11.30)
Immersion 0.80 (0.75) 13.20 (1.47) 10.40 ( 9.28)
Results Worth Efort 3.70 (0.78) 18.60 (1.36) 68.80 (14.56)
Figure 12: Average weighted factor score for each of six fac-
tors on CSI.
Figure 10: Results of the SUS questionnaire.
6.1.3 Dissatisfied Cases Presentation. To comprehensively demon-

strate the design diversity of participants, we showcase some dissat-
isfed cases (Figure 11). We categorized the causes of dissatisfaction
into three perspectives: AI’s lack of structural knowledge, AI gen-
eration instability, and poor shooting quality. First, AI faces chal-
lenges in understanding the structural design in physical models
and successfully converting it into digital solutions. For example,
P7 designed a drone with a telescopic and lifting structure. P7
tried many times, but AI cannot comprehend and refne it success-
fully (Figure 11a). Second, due to the generation mechanism, the
color information in the physical model may interfere with the
results. For example, P10 built a colorful airplane with physical
materials (Figure 11b), the designer failed to get a minimalist and
realistic drone design after repeated attempts due to the complex
color. The randomness and instability during the generation also
occasionally lead to poor-quality outputs. Due to the prompt of
“mechanical arm”, AI mistakenly generated “human hand” in Fig-
ure 11c. Third, the interaction in the application of ProtoDreamer
Figure 11: Dissatisfed outputs that are randomly selected has an impact on the output results. For example, the poor shooting
from design process. quality of physical models, including low background contrast, dark
light, obvious shadow, inappropriate perspective angle, will reduce
the co-creation satisfaction (Figure 11d).
result was “acceptable” and its adjective rating was “Good” accord-
ing to the standard proposed by Bangor [4]. Besides, the usabil-
6.2 RQ2: ProtoDreamer Supports Creativity
ity score and learnability score were 84.22 (�� = 6.13) and 75.63 Efectively during Prototyping.
(�� = 16.99) respectively. Specifcally, most participants perceived 6.2.1 Creativity Support Index of ProtoDreamer. The average CSI
the system to possess a signifcantly high level of availability and ex- score for ProtoDreamer was 86.20 (�� = 4.83), which indicated
hibited robust functional consistency, enabling them to accomplish excellent support for creative work. Table 1 and Figure 12 show the
conceptual design tasks efciently. results of average factor counts, average factor score, and average
Figure 13: Extracted four support modes of PrtoDreamer during assisting ideation.
weighted factor scores for each of the six factors on the CSI. The modes of using ProtoDreamer to support conceptual design from
CSI results highlighted ProtoDreamer’s strengths in augmenting the perspective of design lead, time distribution, and prototype
expressive capabilities and elevating overall satisfaction. Besides, changes and presented corresponding examples (Figure 13). Firstly,
the score of collaboration and exploration indicated that the col- designers deploy ProtoDreamer as a Translator. In this mode, design-
laborative engagement with ProtoDreamer during the conceptual ers with clear concepts devote substantial time to develop physical
design facilitates divergent thinking and creative expansion. How- prototypes and materialize their ideas, then use ProtoDreamer to
ever, the immersion and enjoyment factors were not signifcant with translate captured images of prototypes into high-fdelity digital
ProtoDreamer’s usage. This phenomenon might be attributed to visuals. This phase chiefy involve physical prototype modifcation,
ProtoDreamer’s reliance on unconventional design materials and with digital alteration limited to the fnal “adding skin” stage. Sec-
cross-domain interactions. The incorporation of generative AI, an ondly, designers use ProtoDreamer as a Stimulator. Lacking initial
unfamiliar and challenging design material, alongside the unpre- inspiration, they quickly build a rough model, allowing AI to assume
dictability, demands a steep learning curve for designers [72]. This the design initiative and seeking further navigation through adjust-
requirement for adapting to generative AI interfaces may lessen en- ing input parameters. In this mode, the prototype is mainly altered
joyment associated with the design process. Moreover, the demand in the digital space. Thirdly, as an Explorer, ProtoDreamer aids in
for designers to alternate between constructing physical models the entire conceptual design process. Here, designers continuously
and virtual editing could reduce immersion due to frequent domain scrutinize, splice, and dissect the shape and function of current
switching. physical prototype, testing and verifying its structure. While AI
contributes detailed inspiration for each physical state. This mode
6.2.2 Support Modes of ProtoDreamer During Assisting Ideation. features sequential and alternate physical and digital changes of
Through design process coding, especially paying attention to the prototype. Furthermore, designers employ ProtoDreamer as a
designers’ behaviors and prototypes’ changes, we extracted four
Refner, using its “Refne Function” to optimize details on a rela- this combination behind the ProtoDreamer to the industrial design
tively satisfactory output, with changes primarily in virtual local process through our results. First, when compared to other proto-
details. These four modes occur randomly in conceptual design, type tools, designers pointed out that the ProtoDreamer is good at
often in diverse combinations and multiple applications within a providing schemes efciently and accelerating the design iteration.
single design concept. As the long-time cost is the inherent disadvantages of physical
prototype [29] while the AI owns the ability to ofer a large number
of schemes based on its generative variability [72], their combina-
tion can add data-driven details to low-fdelity physical models,
accelerating the prototyping process. Second, the participation of
generative AI in physical prototype enhances the design expres-
siveness , as well as promoting idea exploration. The CSI results
indicated that the expressiveness provided by ProtoDreamer is sig-
nifcantly strong. Similarly, participant reported that “ProtoDreamer
can help me refne my physical model with high quality. It inspired
me with many details based on my physical model scheme”. Besides,
designers also consider the combination of physical model and AI
provides a easier way to externalize ideas and express schemes,
lowering the threshold of industrial conceptual design, which can
be also proved by the result worth efort index in CSI. Third, the tan-
gibility and AI participation jointly promotes the mixed-modality
Figure 14: Advantages of ProtoDreamer compared with com- prototyping methods and tools. Designers can perceive diferent
monly used prototype tools modalities of design information in ProtoDreamer. In contrast to
mere visual prototypes like sketches, tangibility of physical model
ofers a more accurate sense of scale and texture, allowing for
6.3 RQ3: ProtoDreamer Exhibits Distinctive continuous examination, verifcation, and manipulation, thereby
Advantages in Comparison to Conventional promoting structural thinking and highlighting incorrect assump-
Prototype Tools. tions. [24, 63]. The randomness of AI helps designers capitalize on
As ProtoDreamer represents a novel integration, it presents a chal- the outcome as probabilistic and ofers generated schemes based on
lenge to identify comparable baselines or existing tools for com- each physical confguration [21]. Therefore, ProtoDreamer provides
parative analysis. Therefore, we asked designers to compare their the potential implementation path for multimodal prototype tools.
experience of ProtoDreamer with the prototype methods or tools
commonly used in their design work, including sketching, mod- 7.1.2 Newly Challenges Introduced in Combination of Physical
eling, traditional physical prototyping and so on. Specifcally, 10 Model and Generative AI. Although ProtoDreamer received positive
strengths of mixed-prototype system identifed through thematic responses from designers in the evaluation study, we engaged in a
analysis in our formative study (Section 3.4) were used to construct critical discussion regarding the newly introduced challenges, in or-
a questionnaire to verify distinctive advantages of ProtoDreamer der to give potential users or industrial designers a more extensive
in a wider range (Figure 14). According to our results, we summa- application guidance. On the one hand, the randomness and un-
rized the obvious advantages of ProtoDreamer in defect exposure, predictability of AI might lead distraction in the conceptual design.
structure thinking, and detail thinking. For example, P3 reported During the evaluation study, designers occasionally got unsatisfac-
that “When designing and bonding the joints of physical components, tory AI output, which may be due to poor input information (such
I will consider the design of structural details more deeply. It will as poor shooting quality, unclear prompts, etc.). Once designers
make me think about the infuence of stability and balance of the encountered unexpected feedback, they became confused because
product, avoiding some unrealistic ideas”. Besides, due to the ability they didn’t know what happened in the black box and felt difcult
to produce ample solutions within a design space of generative in maintaining focus on the creative process. This distraction might
AI [65], the ease, time efciency, and sense of accomplishment have contribute to lower enjoyment and immersion score in CSI results.
been improved efectively. P18 mentioned that “My sketching ability On the other hand, the generation of a high-fdelity scheme during
is not very strong. ProtoDreamer allows me to express my ideas in a the preliminary conceptual design phase might potentially result
simple manner, enhancing my sense of accomplishment and providing in creativity fxation, as it discourages the ongoing exploration
me with a better creative state”. of alternative solutions [10]. For example, in the Refner mode of
ProtoDreamer (e.g., Figure 13D), designers spent more time per-
7 DISCUSSION fecting details instead of engaging in the exploration of diverse
possibilities, which might limit their ideas to a narrow space prema-
7.1 Design Support Ofered by the Combination turely. Therefore, we suggest designers alter their prototype in the
of Physical Model and Generative AI physical world and the virtual world repeatedly to explore more
7.1.1 The Combination of Physical Model and Generative AI Presents possibilities when applying ProtoDreamer or this kind of mixed-
Distinctive Contribution. ProtoDreamer is the integration of physi- prototype method. It is also recommended that designers constantly
cal prototype method and generative AI. We discuss the support of adjust capturing angle and input parameters, and generative more
times in order to alleviate design distraction and fxation caused by This data-enabled random addition might surpass designers’ sub-
generation randomness and similar schemes. jective logic and opinions and bring unexpected design concepts
In addition to concerns in generation, our mixed interaction and inspiration.
mode might add limitations related to designers’ manual labor. In
participants’ feedback, excellent high-fdelity schemes were mostly
produced based on well-made and detailed physical models, which
7.2.3 Human-AI Co-prototyping Experience Influenced by Design-
requires designers to pay more labor and time. Therefore, although
ers’ Understanding of Scope, Limitations, and Applicable Scenarios
the embodied interaction with physical materials can promote em-
of AI Agent. In observation of our study, designers sometimes har-
bodied thinking, structural ideation, and verifcation test, Proto-
bored unrealistic expectations towards AI agent, leading to a grad-
Dreamer might be more inefcient than the pure digital generative
ual erosion of confdence and trust. For example, P3 only provided
systems from the production efciency perspective.
a simple keyword with the expectation that the AI agent would
yield intricate details in return. P10 wants a generated scheme in
7.2 Designer-AI Co-creation in ProtoDreamer minimalist style based on a colorful physical model (Figure 11b).
7.2.1 Tangible Interaction Supports Concrete Communication in Therefore, efective human-AI co-prototyping necessitates not only
Designer-AI Co-creation. The design process is not only result- user-friendly system designs but also the incorporation of educa-
oriented but also process-oriented. The goal of conceptual design tional and guiding measures for designers, ensuring that designers
is not to generate high-quality design schemes but to constantly understand the capacity scope and boundaries of AI partner [2].
refne ideas and fnd better solutions. Therefore, HCI studies extend We summarized some directions to efectuate this objective after
beyond merely facilitating creative processes via generative results. practice. First, face to target end-designers and reduce using pro-
It encompasses an ongoing cycle of refnement and iteration of ideas fessional technical vocabulary. P5 reported that “I am a little at
through dynamic interaction with generative AI [11, 75]. From our a loss when faced with ‘Prompt’, but when it is changed to ‘Design
process observation and user interview, we found that tangible in- requirements and restrictions’, I can better understand it”. Second,
teraction in ProtoDreamer supports concrete communication with make clear how to interact with component in user interface and
AI. In ProtoDreamer, the physical model serves as an anchor point what they can do. Sufcient annotations need be provided for inter-
and shared base for AI agent, expressing design intention more intu- active components to efectively communicate the implications and
itively and concretely to AI, while enabling AI follow the designer’s impacts of user interactions. Third, instructional guidance phase
original intention by understanding the rich details on physical plays a pivotal role in enhancing users’ comprehension and grasp
models. Few designers mentioned challenges related to detailed of capacity of AI.
intention share in the evaluation study. P5 indicated that “Physical
prototype contains abundant information including shape, scale, and
color, which supports me to communicate with AI intuitively and ef-
ciently”. P8 also reported that “Color materials can bind position and 7.2.4 Ever-changing Role of AI Agent Confuses Designers and Dis-
semantic information, expressing requirements to AI in detail, without rupts Design Rhythm. Through the observation of human-AI co-
the help of coordinates and directions”. Previous study demonstrated creation in this study, we found that ProtoDreamer can assume a
that multi-modal and non-verbal communication could support variety of cooperative roles according to AI’s contribution, such as
AI partners in developing better shared understanding of design “create new”, “extend”, and “refne” [52]. However, due to the un-
goals [22], which also proved the efectiveness of applying tangible certainty inherent in the generation technology, the ever-changing
models in ProtoDreamer to promote human-AI communication. AI role will cause problems for designers. P9 reported that “when I
am in the fnal process of fne adjustment of the candidate scheme, AI
7.2.2 Generative AI Participation Promotes Design Creativity. We will still change the color matching and basic functions of the design,
discuss the AI-enhancing-creativity beyond ProtoDreamer system. which disrupts my control and thinking of the design rhythm”. Some-
First, generative AI can promote design ideation and enrich design one also indicated that “I hope AI can clarify its role and execute
reasoning. Designers are presented with the opportunity to contin- corresponding endeavors in accordance with my intended trajectory,
uously obtain a diverse range of outputs by adjusting prompts sim- rather than embarking upon a random and unpredictable course”.
ply, providing designers with an infnite number of design schemes Although we provided interactive components to adjust the infu-
for design reasoning and concept exploration. Second, generative ence weight of physical model control generation in ProtoDreamer,
AI can take over tedious and laborious expression tasks [56], en- it still remains insufcient for designers to comprehend AI’s abil-
abling designers to focus on the most important thing in conceptual ity and contribution. As comprehending the scope, background,
design, the “exploration”[27]. For example, in the refne mode of and knowledge of collaborators holds pivotal signifcance in co-
ProtoDreamer, designers can control the AI generation to change creation [18, 54], it is necessary to clarify and present designers
the scheme scale and shape by simply placing anchors and arrows. with a range of distinct AI roles, defning the role types, cooperation
This delegation not only saves time but also reduces the cogni- modes, and contribution forms in human-AI co-prototyping system.
tive load on designers, allowing them to think more freely and For example, we can provide specifc AI roles, such as Translator,
innovatively. Third, generative AI might ofer insights escaping Stimulator, Explorer, and Refner, to end users directly, improving
designers’ relevance sense and awareness through providing data- the specifc ability and usability of each AI role through prompt en-
enabled understandings [20]. Participants reported many times that gineering or fne-tuning, instead of letting users face an unknown
AI supplemented the surprising details based on physical models. and changeable AI.
Figure 16: Application space in facilitating design explo-

ration. (a). Exploring the shape design of an intelligent
speaker based on various combinations of rough physical
models under the support of ProtoDreamer. (b). Exploring
and expressing various confgurations and states of a luggage
structure under the support of ProtoDreamer.
in swift and comprehensive exploration during the conceptual de-

sign phase. This is illustrated in Figure 16a1, where designers use
physical materials to create and modify diverse preliminary shapes.
Figure 15: Application space in lowering threshold of design. These shapes can then be quickly transformed by AI (Figure 16a2),
(a) An ordinary user applies ProtoDreamer to change LEGO uncovering a multitude of possibilities in a short time. Moreover,
model shapes and styles. (b) Utilizing ProtoDreamer to im- ProtoDreamer supports testing diferent product confgurations.
prove the efcient communication of customer’s intention For example, in Figure 16b, designers can experiment with difer-
in the process of customized jewelry design. (c) A prototype ent states and confgurations of a suitcase by rearranging tangible
process of Intelligent Motion Capture Gloves designed by an materials, fostering in-depth consideration of structure and details
engineer independently under the support of ProtoDreamer. through interactive, physical engagement.
7.3.3 Supporting On-site Testing and Iteration: “Fostering situation-
based action and thinking”. The mobility of ProtoDreamer allows
7.3 Potential Application Space of
for prototype iteration process to transcend traditional design stu-
ProtoDreamer dio boundaries, supporting on-site design activities. With Proto-
7.3.1 Lowering Threshold of Design: “Everyone is a designer”. Pro- Dreamer, designers can efectively conduct prototype testing and
toDreamer ofers a user-friendly and low-barrier approach for pro- iteration in actual settings and environments (Figure 17a), utilizing
totyping, enabling individuals without specialized painting or mod- materials present in these real-world contexts. This on-site applica-
eling skills to materialize their ideas into prototypes. For example, tion of ProtoDreamer not only circumvents the spatial dimension
ordinary users can use ProtoDreamer to innovate LEGO gameplay, compression and conversion typically associated with prototyp-
unleashing more potential for imaginative ideation and creativity ing [1], but also promotes context-specifc action and thinking by
space (Figure 15a). Customers can also participate in customized enabling direct design iteration with the real scenarios and confg-
design process through ProtoDreamer, conveying their visions to urations of the targeted product [40].
designers efectively, thereby enhancing both communication ef-
ciency and overall satisfaction (Figure 15b). Besides, engineers and 7.3.4 Enhancing Team Communication and Design Collaboration:
technicians can independently undertake modeling design tasks “Empowering collaboration for seamless design”. Tangible object
based on technical principles (Figure 15c). facilitates spatial reasoning and reduce cognitive load during collab-
orative design [24, 30, 43]. With ProtoDreamer, tangible prototypes
7.3.2 Facilitating Design Exploration and Deliberation: “Unleashing enable design team members to constantly create, test, explain, or
possibilities for prototyping”. During the design process, prototyp- negotiate schemes, endowing them with new meanings [36]. Be-
ing serves not only as a mean to express schemes but also as a tool to sides, given the often interdisciplinary nature of design process [70],
navigate design problem space and solution space through imagin- ProtoDreamer supports team members to intuitively and freely ad-
ing, discussing, and shaping potential states [25, 41]. ProtoDreamer just the prototype through physical materials. Engineers and even
presents a unique platform for professional designers to engage clients without any design skills can also actively participate in
Third, the research duration scale and participant sample are

relatively limited. A more realistic evaluation could apply Proto-
Dreamer in actual design workfows, involving large-scale design
teams over extended periods. We intend to release ProtoDreamer
to the wild for a broader understanding of usability and utility.
9 CONCLUSION
We propose ProtoDreamer, a mixed-prototype tool combining phys-
ical model and generative AI to support industrial conceptual design.
ProtoDreamer allows designers to construct preliminary prototypes
using physical materials, while AI recognizes these forms and vocal
inputs to generate diverse design alternatives. In an evaluation
study, designers were able to create a satisfactory design scheme
under the support of ProtoDreamer. In addition to the usability and
learnability of the system being verifed, the creativity support in
prototyping process had been confrmed, especially in perspectives
of improving expressiveness and worth of efort. Designers showed
a strong preference for the mixed-prototype modes and indicated
its distinct strengths in time efciency, defects exposure, and struc-
tured and detailed thinking. Our work and fndings provide efective
Figure 17: (a). Application space in supporting on-site design. AI-assisted tools to support design and ofer important insights and
Designers used ProtoDreamer to test and explore the design suggestions for human-AI co-creation based on large-scale models.
of Head-mounted Night Light based on existing helmet prod-
ucts. (b). Application space in enhancing design collaboration. ACKNOWLEDGMENTS
The interdisciplinary design team applied ProtoDreamer to This work was supported by the National Key Research and Devel-
enhance design communication with a Fruit Picking Manip- opment Program of China (2022YFB3303301). The authors would
ulator in the meeting room. like to thank Bowen Xu for assistance in system development,
Mingyan Wang, Tuotuo Zheng for assistance in fgures production,
as well as all the reviewers for their constructive opinions.
creating and testing activities, contributing to reaching team con- REFERENCES

sensus for design goals (Figure 17b). Design team members are [1] Remi Alkemade, Fons J Verbeek, and Stephan G Lukosch. 2017. On the Efciency
of a VR Hand Gesture-based Interface for 3D Object Manipulations in Conceptual
able to directly articulate their requirements and swiftly access Design. International Journal of Human–Computer Interaction 33, 11 (2017), 882–
generated design schemes aligned with the current iteration, thus 901.
[2] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi,
facilitating efcient communication and decision-making. Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019.
Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI conference
on human factors in computing systems. 1–13.
8 LIMITATIONS [3] AUTOMATIC1111. 2022. Stable Difusion Web UI. https://github.com/
AUTOMATIC1111/stable-difusion-webui
We discuss the limitations of our study that need to be addressed [4] Aaron Bangor, Philip T Kortum, and James T Miller. 2008. An Empirical Evaluation
in future research. of the System Usability Scale. Intl. Journal of Human–Computer Interaction 24, 6
First, the way to capture the information of physical model is (2008), 574–594.
[5] Loris Barbieri, Agostino Angilica, Fabio Bruno, and Maurizio Muzzupappa. 2013.
single. We only achieve intention share through capturing the im- Mixed Prototyping with Confgurable Physical Archetype for Usability Evalua-
age of the physical model by camera. It might not only reduce the tion of Product Interfaces. Computers in Industry 64, 3 (2013), 310–323.
[6] Johan J Broek, Woulter Sleijfers, Imre Horváth, and Alex F Lennings. 2000. Using
dimension of information, but also be afected by shooting quality, Physical Models in Design. In Proceedings of CAID/CD’2000 conference. Citeseer,
such as clarity, shooting angle, shooting environment and back- 155–163.
ground. In future research, we plan to capture the 3D data of the [7] John Brooke. 1996. Sus: A “Quick and Dirty’Usability. Usability evaluation in
industry 189, 3 (1996), 189–194.
physical model directly through the tangible user interface or 3D [8] Bradley Camburn, Vimal Viswanathan, Julie Linsey, David Anderson, Daniel
reconstruction technology. Jensen, Richard Crawford, Kevin Otto, and Kristin Wood. 2017. Design Prototyp-
Second, more comprehensive quantitative research is needed ing Methods: State of the Art in Strategies, Techniques, and Guidelines. Design
Science 3 (2017), e13.
to fully explore the underlying factors that contribute to the fa- [9] Pei Chen, Yangkang Zhang, Zejian Li, and Lingyun Sun. 2022. Few-Shot Incre-
cilitation and augmentation prototype process. The comparative mental Learning for Label-to-Image Translation. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition. 3697–3707.
analysis against existing tools, as a benchmark, can be conducted [10] Peiyao Cheng, Ruth Mugge, and Jan P.L. Schoormans. 2014. A New Strategy to
to discern the efectiveness of ProtoDreamer, bolstered by robust Reduce Design Fixation: Presenting Partial Photographs to Designers. Design
statistical analysis. Experiments with strictly controlled conditions Studies 35, 4 (2014), 374–391.
[11] DaEun Choi, Sumin Hong, Jeongeon Park, John Joon Young Chung, and Juho
(such as design tasks and design time) should also be carried out to Kim. 2023. CreativeConnect: Supporting Reference Recombination for Graphic
provide a more complete and in-depth efciency report. Design Ideation with Generative AI. arXiv preprint arXiv:2312.11949 (2023).
[12] John Joon Young Chung and Eytan Adar. 2023. PromptPaint: Steering Text- 2023 ACM Designing Interactive Systems Conference. Association for Computing
to-Image Generation Through Paint Medium-like Interactions. In Proceedings Machinery, 1955–1977.
of the 36th Annual ACM Symposium on User Interface Software and Technology. [39] Lllyasviel. 2023. controlnet-sd1.5-inpaint. https://huggingface.co/lllyasviel/
Association for Computing Machinery, New York, NY, USA, Article 6, 17 pages. ControlNet-v1-1. Last accessed on August 15, 2023.
[13] Midjourney Company. 2022. Midjourney. https://www.midjourney.com. [40] Rachael Luck. 2018. What Is It that Makes Participation in Design Participatory
[14] Hai Dang, Frederik Brudy, George Fitzmaurice, and Fraser Anderson. 2023. World- Design? Design Studies 59 (2018), 1–8.
Smith: Iterative and Expressive Prompting for World Building with a Generative [41] Mary Maher and Hsien-Hui Tang. 2003. Co-evolution as a Computational and
AI. In Proceedings of the 36th Annual ACM Symposium on User Interface Software Cognitive Model of Design. Research in Engineering design 14 (2003), 47–64.
and Technology. Association for Computing Machinery, Article 63, 17 pages. [42] David Mathias, Chris Snider, Ben Hicks, and Charlie Ranscombe. 2019. Acceler-
[15] Nicholas Davis, Chih-Pin Hsiao, Yanna Popova, and Brian Magerko. 2015. An ating Product Prototyping Through Hybrid Methods: Coupling 3D Printing and
Enactive Model of Creativity for Computational Collaboration and Co-creation. LEGO. Design Studies 62 (2019), 68–99.
Creativity in the digital age (2015), 109–133. [43] Ashwin Michaelraj. 2009. Taxonomy of Physical Prototypes: Structure and Valida-
[16] Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design tion. Ph. D. Dissertation. Clemson University.
Innovation: Challenges for Working With Machine Learning as a Design Material. [44] Bill Moggridge and Bill Atkinson. 2007. Designing Interactions. Vol. 17. MIT press
In Proceedings of the 2017 CHI conference on human factors in computing systems. Cambridge.
278–288. [45] Kyzyl Monteiro, Ritik Vatsal, Neil Chulpongsatorn, Aman Parnami, and Ryo
[17] Steven Dow. 2011. How Prototyping Practices Afect Design Results. Interactions Suzuki. 2023. Teachable Reality: Prototyping Tangible Augmented Reality with
18, 3 (2011), 54–59. Everyday Objects by Leveraging Interactive Machine Teaching. In Proceedings of
[18] Valentina Fantasia, Hanne De Jaegher, and Alessandra Fasulo. 2014. We Can the 2023 CHI Conference on Human Factors in Computing Systems. Association for
Work It Out: An Enactive Look at Cooperation. Frontiers in psychology 5 (2014), Computing Machinery, Article 459, 15 pages.
874. [46] Michael Muller, Lydia B Chilton, Anna Kantosalo, Charles Patrick Martin, and
[19] Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Greg Walsh. 2022. GenAICHI: Generative AI and HCI. In CHI conference on
Biskjaer, and Peter Dalsgaard. 2019. Mapping the Landscape of Creativity Support human factors in computing systems extended abstracts. 1–7.
Tools in HCI. In Proceedings of the 2019 CHI Conference on Human Factors in [47] Jacob Nelson and Jessica Menold. 2020. Opening the Black Box: Developing
Computing Systems. 1–18. Metrics to Assess the Cognitive Processes of Prototyping. Design Studies 70
[20] Elisa Giaccardi. 2019. Histories and Futures of Research Through Design: From (2020), 100964.
Prototypes to Connected Things. International Journal of Design 13, 3 (2019), [48] Kevin N Otto et al. 2003. Product Design: Techniques in Reverse Engineering and
139–155. New Product Development. Tsinghua University Press.
[21] Elisa Giaccardi and Johan Redström. 2020. Technology and More-than-human [49] Rivka Oxman. 2004. Think-maps: Teaching Design Thinking in Design Education.
Design. Design Issues 36, 4 (2020), 33–44. Design Studies 25, 1 (2004), 63–91.
[22] Frederic Gmeiner, Humphrey Yang, Lining Yao, Kenneth Holstein, and Nikolas [50] Lihang Pan, Chun Yu, Zhe He, and Yuanchun Shi. 2023. A Human-Computer
Martelaro. 2023. Exploring Challenges and Opportunities to Support Designers in Collaborative Editing Tool for Conceptual Diagrams. In Proceedings of the 2023
Learning to Co-create with AI-based Manufacturing Design Tools. In Proceedings CHI Conference on Human Factors in Computing Systems. 1–29.
of the 2023 CHI Conference on Human Factors in Computing Systems. 1–20. [51] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Rad-
[23] Gabriela Goldschmidt and Maria Smolkov. 2006. Variances in the Impact of Visual ford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot Text-to-image Generation.
Stimuli on Design Problem Solving Performance. Design Studies 27, 5 (2006), In International Conference on Machine Learning (ICML). PMLR, 8821–8831.
549–569. [52] Jeba Rezwana and Mary Lou Maher. 2022. Understanding User Perceptions,
[24] Matheus Galvão Gomes, André Ogliari, Rodrigo Bastos Fernandes, and Collaborative Experience and User Engagement in Diferent Human-AI Interac-
Karuliny Oliveira Marques. 2022. Evaluation of Physical Models as Creative tion Designs for Co-creative Systems. In Proceedings of the 14th Conference on
Stimuli in Conceptual Design of Products. Design Studies 81 (2022), 101119. Creativity and Cognition. 38–48.
[25] Wendy Gunn and Jared Donovan. 2016. Design and Anthropology. Routledge. [53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
[26] Tim Hess and Joshua David Summers. 2013. Case Study: Evidence of Prototyping Ommer. 2022. High-resolution image synthesis with latent difusion models. In
Roles in Conceptual Design. In Proceedings of the 19th International Conference Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
on Engineering Design (ICED13), Vol. 1. 19–22. 10684–10695.
[27] Rong Huang, Haichuan Lin, Chuanzhang Chen, Kang Zhang, and Wei Zeng. 2024. [54] R Keith Sawyer and Stacy DeZutter. 2009. Distributed Creativity: How Collective
PlantoGraphy: Incorporating Iterative Design Process into Generative Artifcial Creations Emerge from Collaboration. Psychology of aesthetics, creativity, and
Intelligence for Landscape Rendering. In Proceedings of the CHI Conference on the arts 3, 2 (2009), 81.
Human Factors in Computing Systems. Association for Computing Machinery, [55] Xinyu Shi, Ziqi Zhou, Jing Wen Zhang, Ali Neshati, Anjul Kumar Tyagi, Ryan
Article 168, 19 pages. Rossi, Shunan Guo, Fan Du, and Jian Zhao. 2023. De-Stijl: Facilitating Graphics
[28] Matilde B Jensen, Christer W Elverum, and Martin Steinert. 2017. Eliciting Design with Interactive 2D Color Palette Recommendation. In Proceedings of the
Unknown Unknowns with Prototypes: Introducing Prototrials and Prototrial- 2023 CHI Conference on Human Factors in Computing Systems. 1–19.
driven Cultures. Design Studies 49 (2017), 1–31. [56] Yang Shi, Tian Gao, Xiaohan Jiao, and Nan Cao. 2023. Understanding Design
[29] Lee Kent, Chris Snider, James Gopsill, and Ben Hicks. 2021. Mixed Reality in Collaboration Between Designers and Artifcial Intelligence: A Systematic Litera-
Design Prototyping: A Systematic Review. Design Studies 77 (2021), 101046. ture Review. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2,
[30] Mi Jeong Kim and Mary Lou Maher. 2008. The Impact of Tangible User Interfaces Article 368 (oct 2023), 35 pages.
on Spatial Cognition During Collaborative Design. Design Studies 29, 3 (2008), [57] Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and
222–253. Song Bai. 2023. DragDifusion: Harnessing Difusion Models for Interactive
[31] Takashi Kiriyama and Takashi Yamamoto. 1998. Strategic Knowledge Acquisition: Point-based Image Editing. arXiv preprint arXiv:2306.14435 (2023).
A Case Study of Learning Through Prototyping. Knowledge-based Systems 11, [58] Stabilityai. 2023. Stable Difusion v2.1. https://huggingface.co/stabilityai/stable-
7-8 (1998), 399–404. difusion-2-1. Last accessed on July 26, 2023.
[32] David Kirsh. 2013. Embodied Cognition and the Magical Future of Interaction [59] Stabilityai. 2023. Stable Difusion XL 1.0. https://huggingface.co/stabilityai/
Design. ACM Transactions on Computer-Human Interaction (TOCHI) 20, 1 (2013). stable-difusion-xl-base-1.0. Last accessed on August 15, 2023.
[33] Carlye A. Lauf, Daniel Knight, Daria Kotys-Schwartz, and Mark E. Rentschler. [60] Ryo Suzuki, Rubaiat Habib Kazi, Li-yi Wei, Stephen DiVerdi, Wilmot Li, and
2020. The Role of Prototypes in Communication Between Stakeholders. Design Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and
Studies 66 (2020), 1–34. Visualizations in AR through Dynamic Sketching. In Proceedings of the 33rd
[34] Gay Lemons, Adam Carberry, Chris Swan, Linda Jarvin, and Chris Rogers. 2010. Annual ACM Symposium on User Interface Software and Technology. Association
The Benefts of Model Building in Teaching Engineering Design. Design Studies for Computing Machinery, 166–181.
31, 3 (2010), 288–309. [61] Jon Swain. 2018. A Hybrid Approach to Thematic Analysis in Qualitative Re-
[35] Jing Liao, Preben Hansen, and Chunlei Chai. 2020. A Framework of Artifcial search: Using A Practical Example. Sage research methods (2018).
Intelligence Augmented Design Support. Human–Computer Interaction 35, 5-6 [62] Ye Tao, Shuhong Wang, Junzhe Ji, Linlin Cai, Hongmei Xia, Zhiqi Wang, Jinghai
(2020), 511–544. He, Yitao Fan, Shengzhang Pan, Jinghua Xu, et al. 2023. 4Doodle: 4D Printing
[36] Stephen W Littlejohn and Karen A Foss. 2009. Encyclopedia of Communication Artifacts Without 3D Printers. In Proceedings of the 2023 CHI Conference on Human
Theory. Vol. 1. Sage. Factors in Computing Systems. 1–16.
[37] Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering [63] Christine A Toh and Scarlett R Miller. 2014. The Impact of Example Modality
Text-to-image Generative Models. In Proceedings of the 2022 CHI Conference on and Physical Interactions on Design Creativity. Journal of Mechanical Design
Human Factors in Computing Systems. 1–23. 136, 9 (2014), 091004.
[38] Vivian Liu, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2023. 3DALL- [64] Michihiko Ueno and Shin’ichi Satoh. 2021. Continuous and Gradual Style Changes
E: Integrating Text-to-Image AI in 3D Design Workfows. In Proceedings of the of Graphic Designs with Generative Model. In 26th International Conference on
Intelligent User Interfaces. 280–289.

[65] Josh Urban Davis, Fraser Anderson, Merten Stroetzel, Tovi Grossman, and George
Fitzmaurice. 2021. Designing Co-creative AI for Virtual Environments. In Cre-
ativity and Cognition. 1–11.
[66] Jelle Van Dijk. 2018. Designing for Embodied Being-in-the-World: A Critical
Analysis of the Concept of Embodiment in the Design of Hybrids. Multimodal
Technologies and Interaction 2, 1 (2018).
[67] Vimal Viswanathan, Olufunmilola Atilola, Nicole Esposito, and Julie Linsey. 2014.
A Study on the Role of Physical Models in the Mitigation of Design Fixation.
Journal of Engineering Design (2014), 25–43.
[68] Vimal Viswanathan and Julie Linsey. 2011. Understanding Physical Models in
Design Cognition: A Triangulation of Qualitative and Laboratory Studies. In 2011
Frontiers in Education Conference (FIE). IEEE, S3F–1.
[69] Jiashun Wang, Xueting Li, Sifei Liu, Shalini De Mello, Orazio Gallo, Xiaolong
Wang, and Jan Kautz. 2023. Zero-shot Pose Transfer for Unrigged Stylized 3D
Characters. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition. 8704–8714.
[70] Lihui Wang, Weiming Shen, Helen Xie, Joseph Neelamkavil, and Ajit Pardasani.
2002. Collaborative Conceptual Design—state of the Art and Future Trends.
Computer-aided Design 34, 13 (2002), 981–996.
[71] Sitong Wang, Savvas Petridis, Taeahn Kwon, Xiaojuan Ma, and Lydia B Chilton.
2023. PopBlends: Strategies for Conceptual Blending with Large Language Models.
In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.
1–19.
[72] Justin D. Weisz, Michael Muller, Jessica He, and Stephanie Houde. 2023. Toward
General Design Principles for Generative AI Applications. https://doi.org/10.
48550/arXiv.2301.05578
[73] Margaret Wilson. 2002. Six Views of Embodied Cognition. Psychonomic bulletin
& review 9 (2002), 625–636.
[74] Di Wu, Zhiwang Yu, Nan Ma, Jianan Jiang, Yuetian Wang, Guixiang Zhou, Hanhui
Deng, and Yi Li. 2023. StyleMe: Towards Intelligent Fashion Generation with
Designer Style. In Proceedings of the 2023 CHI Conference on Human Factors in
Computing Systems. 1–16.
[75] Zihan Yan, Chunxu Yang, Qihao Liang, and Xiang’Anthony’ Chen. 2023. XCre-
ation: A Graph-based Crossmodal Generative Creativity Support Tool. In Proceed-
ings of the 36th Annual ACM Symposium on User Interface Software and Technology.
1–15.
[76] Maria C Yang. 2005. A Study of Prototypes, Design Activity, and Design Outcome.
Design Studies 26, 6 (2005), 649–669.
[77] Qian Yang, Nikola Banovic, and John Zimmerman. 2018. Mapping Machine
Learning Advances from HCI Research to Reveal Starting Places for Design Inno-
vation. In Proceedings of the 2018 CHI conference on human factors in computing
systems. 1–11.
[78] Robert J Youmans. 2011. The Efects of Physical Prototyping and Group Work
on the Reduction of Design Fixation. Design Studies 32, 2 (2011), 115–138.
[79] JD Zamfrescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang.
2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design
LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in
Computing Systems. 1–21.
[80] Yu Zeng, Zhe Lin, Jianming Zhang, Qing Liu, John Collomosse, Jason Kuen, and Figure 18: SUS Results of ProtoDreamer and SD WebUI.
Vishal M Patel. 2023. Scenecomposer: Any-level Semantic Image Synthesis. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
22468–22478.
[81] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Con-
trol to Text-to-Image Difusion Models. https://github.com/lllyasviel/ControlNet- APPENDIX1. PARTICIPANT INFORMATION
v1-1nightly
[82] Yangkang Zhang, Chenye Meng, Zejian Li, Pei Chen, Guang Yang, Changyuan The participant information in the evaluation study is showed in
Yang, and Lingyun Sun. 2023. Learning Object Consistency and Interaction Table2.
in Image Generation from Scene Graphs. In Proceedings of the Thirty-Second
International Joint Conference on Artifcial Intelligence (IJCAI). 1731–1739.
[83] Ning Zou, Qing Gong, Qing Chai, and Chunlei Chai. 2023. The Role of Virtual APPENDIX2. A SUPPLEMENTED ANALYSIS
Reality Technology in Conceptual Design: Positioning, Applications, and Value.
Digital Creativity 34, 1 (2023), 53–77. During the rebuttal stage, we supplemented a comparative experi-
ment. We contacted 19/20 participants in User Study 2 and invited
them to use the StableDifusion WebUI interface to complete design
task. All back-end server implementation is same to ProtoDreamer
to ensure fairness. Aiming to ensure participants to complete simi-
lar design tasks, we summarized all product categories they created
in User Study 2 as text, and they were asked to choose one or
more as design tasks. After design, they completed SUS and CSI
questionnaires. For the data analysis, the Shapiro-Wilk test was
run at a signifcance level of 0.05 for normality test. The t-test for
normal distribution while Mann-Whitney U test for non-normal
distribution.
Table 2: Overview of participants in the evaluation study. The average CSI score of ProtoDreamer and baseline was 86.20 (�� =
4.83) and 69.90 (�� = 4.87), in which there was a signifcant dif-
ID Age Gender Domain Occupation ference (� (37) = −10.357, � < 0.001). Figure 19 show the average
P1 27 F Industrial Design Student / Phd
P2 25 M Computational Design Designer weighted factor scores for each of the six factors on the CSI. Specif-
P3 26 M Industrial Design Designer ically, ProtoDreamer presented signifcant higher performance in
P4 24 M Industrial Design Student / MA
P5 27 M Industrial Design Designer
collaboration (� = 98.0, � = 0.009), exploration (� = 39.0, � < 0.001),
P6 23 F Industrial Design Student / MA and expressiveness (� = 54.0, � < 0.001) than the baseline.
P7 35 F Mechanical Engineering Designer
P8 24 F Industrial Design Student / MA
P11 26 M Industrial Design Student / MA
P12 24 F Industrial Design Student /MA
P16 27 M Mechanical Engineering Engineer
P18 25 M Mechanical Engineering Engineer
P19 25 M Industrial Design Designer
P20 26 F Computational Design Student / Phd
The SUS results are presented in Figure 18. The average total SUS
score of ProtoDreamer and baseline was 82.50 (�� = 6.12) and 50.92
(�� = 6.08), in which there was a signifcant diference (� (37) =
15.869, � < 0.001). Besides, the usability score was 84.22 (�� = 6.13)
for ProtoDreamer while 54.93 (�� = 5.94)for baseline, in which
there was a signifcant diference (� = 380.0, � < 0.001). The
learnability score was 75.63 (�� = 16.99) for ProtoDreamer while
34.87 (�� = 11.88) for baseline, in which there was a signifcant Figure 19: Average weighted factor score for CSI. *indicates
diference (� = 364.5, � < 0.001). the signifcant diference.

3654777.3676399

Uploaded by

Copyright:

Available Formats

3654777.3676399

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3654777.3676399

Uploaded by

Copyright:

Available Formats

ProtoDreamer: A Mixed-prototype Tool Combining Physical

Model and Generative AI to Support Conceptual Design

Chaoyi Lin Lianyan Liu Zhuoshu Li

Weitao You Lingyun Sun

4.2.4 Providing Prompts Style Folders (for DR3). Current text-to-

Figure 7: A drone design using the generation control of style

Table 1: CSI results of ProtoDreamer in the evaluation study.

Avg. Factor Avg. Factor Avg. Weighted

6.1.3 Dissatisfied Cases Presentation. To comprehensively demon-

Figure 16: Application space in facilitating design explo-

in swift and comprehensive exploration during the conceptual de-

Third, the research duration scale and participant sample are

creating and testing activities, contributing to reaching team con- REFERENCES

Intelligent User Interfaces. 280–289.

You might also like