Introduction
In 1979, Jane Darke pleaded for embracing subjective aspects into the study of design processes (Darke, Reference Darke1979). Darke's plea was answered, resulting in research work which closely examined design from the perspective of the designer. One subject of study in which subjective factors were acknowledged was the study of interpretation.
Goldschmidt, for example, has demonstrated how architectural design students construed the same task in very different manners when designing a small residential unit that needed to conform to the idea of a “cube” (Goldschmidt, Reference Goldschmidt1988). Other key examples can be found in the work by Schön, who proposed that different designers inhabit different “design worlds”, shaped by their individual knowledge and design experience (Schön, Reference Schön1988), or in Bucciarelli who pointed out the existence of “object worlds” reflecting unique practices and interpretive perspective in engineering design (Bucciarelli, Reference Bucciarelli2002).
It is difficult to overstate the importance of studying interpretation for deepening our understanding of human design processes. The pervasiveness of interpretation in design is reflected in the well-known schematization of design as “seeing-moving-seeing” (Schön and Wiggins, Reference Schön and Wiggins1992), considering that interpretation facilitates seeing, which both precedes and follows “moving” (i.e., action).
Our work aims to further these efforts of (to borrow Darke's phrase) “looking into the designer's head” (Darke, Reference Darke1979, p. 43), by attempting to visualize the designer's thoughts and considerations. Existing approaches for mapping design processes provide us with tools for describing design cognition at various levels of analysis. Linkography, for example, offers a powerful toolkit for global analysis of linkages between actions, which can be applied for complete design processes (Goldschmidt, Reference Goldschmidt2014), while the method developed by Prats et al. allows for a more local analysis of specific transformation between shapes (Prats and Garner, Reference Prats and Garner2006).
Furthering these efforts, we propose a new method for visualizing design through the eyes of the designer. Compared with previous work, our method enables to increase the resolution of observation, thus enabling to elicit further insights into how designers view design situations. Our method was developed on the basis of a simple design task, devised for closely observing interpretation-related events. The task focuses on designing a miniature Japanese rock garden (JRG), for reasons that are clarified later (see the “Method” section). We then utilized recent frameworks for modeling design from a situated perspective (Gero and Kannengiesser, Reference Gero and Kannengiesser2004) and drew inspiration from discourse analysis (Fauconnier, Reference Fauconnier1994) for proposing a new approach to visualizing mental content in design. The proposed approach enables a high-resolution visualization and analysis of interpretive events in design, and thus enhances our ability to study and understand design processes.
Aims and scope
We aim to enable researchers to map, visualize and better understand events in design activity. On the basis of Gero's fundamental distinction between functions, behaviors, and structures in design (Gero, Reference Gero1990), this work takes a “structure” perspective. A structure is generally defined as an artifact itself or its representation and is consequently the most concrete entity in Gero's ontology. Our choice to focus on the interpretation of structure is thus justified by its potential to reduce the complexity of the phenomenon under consideration, taking into account the tremendous richness of interpretation processes (as they are tightly interlinked with language).
Objectives
Main objectives set for this study are: (1) observe and document designers as they engage in interpretation activity, (2) track the manners in which simple objects were assigned with interpretations, and (3) propose a method for visualizing such events, while considering the designer's perspective of the situation.
Significance
This work takes a step forward in visualizing design processes, by proposing a systematic way to trace momentary events, as a reflection of the designer's view of the situation. In this, it goes beyond current approaches for design visualization, by enabling to create high-resolution pictures of the designer's understanding of semantic relations.
Potential readers who may benefit from this work are: (1) design-cognition theorists, who are interested in realistically modeling the ways in which designers view design situations; (2) developers of intelligent CAD systems, who wish to integrate interpretive capabilities into future design support systems; and (3) design practitioners and educators, who aim to better understand how we engage in interpretation in real-time.
Background
Metaphor and imagery in design interpretation
Tversky et al. have divided interpretation in design into two sub-processes: construction of mental representations and attribution of meaning to these (Tversky et al., Reference Tversky, Suwa, Agrawala, Heiser, Stolte, Hanrahan, Haymaker and Lindemann2003). As metaphors enable designers to mentally capture complex relations within short, memorable descriptions (Hey and Agogino, Reference Hey and Agogino2007), they intuitively facilitate both of the above sub-processes, and are thus powerful interpretive tools. Indeed, Lakoff and Johnson have elevated the status of conceptual metaphors from a mere decorative linguistic device to a central aspect of human thinking and understanding (Lakoff and Johnson, Reference Lakoff and Johnson2003). The basic principle at work is as follows: by invoking a metaphor (consider their famous example of “time is money”), we are forming a mapping between two domains (time and finance). This enables us to further understand one in terms of the other (e.g., similarly to money, time can also be “spent”). Since such mappings enable to fluidly encode and decode information which is viewed as meaningful from one's perspective, they enhance designers’ capability to cope with design problems in real-time (Casakin, Reference Casakin2006).
When utilizing metaphors, we depart from the literal meaning of a phrase to convey another, non-literal, meaning (Searle, Reference Searle and Martinich1990). These non-literal (intended) meanings can be thought of as our personal interpretation of a certain state of affairs (whether real or imaginary). Since such departure requires us to draw on our imaginative capacities and look beyond the literal meaning, there is a tight link between metaphors and mental images which “accompany” them (and thus plays a cognitive role in interpretation). Consider a team of architects who metaphorically refer to a building as “the beating heart” of their project – in doing so, they project an interpretive image onto a certain part of the design, which then shapes the way the project is understood and thus navigated.
One form of imaginative ability which stands out from design interpretation literature is that of “seeing as” – we look at one object and interpret it as another (Goldschmidt, Reference Goldschmidt1991). For example, a thin 2D rectangle can invoke a mental image of a tall building. Since these cognitive events involve “looking through” the given object (what Liddament described as epistemic transparency; Liddament, Reference Liddament2000), they are associated with metaphoric thinking in design (as we do not take the visual object in front of us “literally”). In this study, we do not limit ourselves to a single type of cognitive phenomenon such as “seeing as”. Instead, we attempt to trace the generation and elaboration of interpretations by focusing on the content of mental images which facilitate these (and other) interpretive events.
A useful starting point for discussing mental imagery is its basic characterization as a “quasi-observational” activity, in which one experiences something which is not present as a perceptual object (Sartre, Reference Sartre1940). However, as imagery refers to a wide range of cognitive phenomena (Richardson, Reference Richardson1969) which facilitate numerous abilities (Stevenson, Reference Stevenson2003), it is essential to further elaborate on our view of imaginative activity. More specifically, Kosslyn has notably advanced the view of imagination as a “screen” onto which mental representations are projected (Kosslyn, Reference Kosslyn1980). Imagining thus consists of a subject “reading-off” information which is temporarily represented in working memory (Dahl et al., Reference Dahl, Chattopadhyay and Gorn1999) for a certain purpose. Such mental reading enhances our ability to think and act, for instance, by facilitating inference-making without the need to manipulate logical terms (Johnson-Laird, Reference Johnson-Laird1983). This immediacy, along with the intuitive nature and power of metaphor, implies of a strong link between metaphoric thinking, mental images, and real-time performance in designing.
Based on the above, we hypothesize that inquiring into such images which appear on the mental “screen” (and thus offer useful information for the designer) can contribute to our knowledge of interpretation activity. We attempt to trace the content of such images by analyzing the ways metaphorical descriptions are externalized in the form of physical objects. Arcangeli has distinguished “seeing-imagining” (close your eye and imagine such a building) from “hearing-imagining” (repeat this with your tune of choice) and from other types of imagery (Arcangeli, Reference Arcangeli2020). This paper focuses on the mental content of imagery, regardless of the modality in which it is generated and experienced by designers. As established by Goel, studying how such content is externalized and utilized in the design process is important for our understanding of design cognition (Goel, Reference Goel1995).
Visualization as a way into the designer's head
Without a doubt, visualization is an effective means for understanding complex processes and events, and as such plays an important role in studying the (often rich) mental world of designers. One notable design visualization method, which takes the view of design as an information-processing activity, can be found in Akin (Reference Akin1989). On the basis of design protocols from architectural design activity, Akin has demonstrated how behavior in design can be described as a sequence of states and transformations between them. States reflect pertinent information dealt with by the designer at a given moment in time, and transformations indicate changes to this information. The diagrams formed by Akin expose the existence of multiple, somewhat concurrent, relations between entities, which shape the way designers engage in problem-solving activity when designing. A more recent method developed by Goldschmidt consists of the creation of “linkographs” – non-directed graphs which link “design moves” (Goldschmidt, Reference Goldschmidt2014). Simply put, design moves are actions/decisions taken by the designer. Linking design moves enables to visualize and trace the structure of the design process, in terms of how one acts based on past events. From a different perspective, Sun et al. have proposed a visualization method termed “creative segment”, which describes sketching in design as a tree with “design paths” for branches. Designers are said to both extend certain paths (by continuously working on a specific idea), add new paths, and revisit these from time to time. The resulting tree structure serves as a visual summary of the process as a whole. As such, it facilitates inquiry into designing – for example, by reflecting how inspirations are combined into a design outcome (Sun et al., Reference Sun, Xiang, Chai, Wang and Huang2014). Finally, taking a different approach, Prats et al. have proposed to trace designers’ thinking by visualizing the progression sketches and identifying transformations between shapes. Their method enables to extract and formalize such transformations as shape rules, which can be used for computational generation of shapes in designing (Prats et al., Reference Prats, Lim, Jowers, Garner and Chase2009). In addition to the above, certain works take an integrated approach – for example, Bilda et al. have used a coding scheme for imagery along with linkographs to examine designer's performance in order to reevaluate the widely accepted view that sketching is essential for conceptual design (Bilda et al., Reference Bilda, Gero and Purcell2006).
We aim to extend this arsenal for inquiry into mental content by offering a high-resolution approach for its visualization and analysis. Our approach thus focuses on the elaboration of mental content at specific moments in time in high detail, as to enable to draw important insights of fleeting information, which would otherwise remain unnoticed.
Situated cognition as an approach for modeling interpretation
Drawing on the situated approach to cognition (Clancey, Reference Clancey1997; Clark, Reference Clark1998; Wilson and Clark, Reference Wilson, Clark, Aydede and Robbins2009), Gero and Kannengiesser have formulated the situated function–behavior–structure framework (situated FBS) (Gero and Kannengiesser, Reference Gero and Kannengiesser2004), as an extension of Gero's original formalization of design in the FBS framework (Gero, Reference Gero1990). In the situated FBS, design is described using a series of transformations between structures (artifacts or their parts), behaviors (what these can do), and functions (what they are meant for). Furthermore, three design worlds are defined: external (outside of the designer), interpreted (an internal reflection of the former), and expected (an internal representation of future intentions). Since it acknowledges the interpretive aspect of designing, we find this framework useful for this study and embrace its distinction between the external and internal worlds. Additionally, we use “structure” to refer to the complete set of entities used by our designers as design representations and further define a “design element” or simply “element” as a single indivisible part of the structure.
Given its general nature, however, this framework demands further elaboration for our task of visualizing interpretive processes in designing. An important aspect of interpretation is the assignment of metaphoric meanings to the artifact (or its representation), a practice which enables conceptual control over the design (Casakin, Reference Casakin2006). Accordingly, it shapes designers’ perspective, as well as the way in which they devise their courses of action. As the transformations in the situated FBS occur between sets of abstract entities (which are independent of the values assigned to them in practice), visualizing such interpretive aspects demands the introduction of additional conceptual tools, which specifically deal with mental content and semantic aspects of designing.
For organizing such mental content, we draw inspiration from Fauconnier's mental space theory (Fauconnier, Reference Fauconnier1994). Mental spaces are virtual environments created by speakers via “space builder phrases”, which establish contexts for conversation. By uttering the phrase “yesterday at the mall”, we can build a mental space for hosting additional details, which may appear in the following conversation. For example, we can say that “John lost his wallet”, which will add both John and his wallet to the space. In this sense, a mental space can be thought of as an abstract container for entities introduced in discourse. Mental spaces enable to visualize interrelations between such entities and are thereby effective for disambiguation of discourse, even in complex cases of conflicting subjective perspectives (Liu and Gao, Reference Liu and Gao2010). In this paper, we describe the designer's internal world using such spaces, which consist of: (1) physical objects serving as design elements, (2) their groupings, and (3) their interpretations.
JRG design and interpretation
JRGs are traditional landscape creations which date back to the Heian period (from ~794 CE), originally designed as spaces for observation or contemplation (Slawson and ZoÌen, Reference Slawson and ZoÌen1991). Despite their description as “gardens”, JRGs often contain little or no vegetation, and instead consist of an arrangement of several rocks, placed carefully on a bed of sand. The rocks commonly have a symbolic value and are assigned with interpretations in varying degrees of complexity (Mansfield and Richie, Reference Mansfield and Richie2009). For example, a single rock may represent a turtle or a crane (these are, in turn, represent longevity), while several rocks may be joined to form miniature “mountain ranges” (possibly reminiscent of the mythological Mount. Sumeru), etc.
We view this tight relation between JRGs and interpretation as an invitation for utilizing it as a context for studying interpretation practices in design. In this paper, we regard interpretation as the assignment of a linguistic description to a physical entity, which metaphorically associates it with another entity. In Figure 1, for example, the middle rock is commonly assigned with the description of a “turtle”, which associates it with a real turtle (in this case, on the basis of resemblance in form).
Fig. 1. A “turtle” rock at the famous garden of Ryoan-ji in Kyoto, Japan (photograph by the authors).
Method
Approach
Interpretation, as we have construed it, occasionally occurs in design in a natural manner. However, to collect rich empirical data, we have resolved to devise a task which will emphasize the interpretive dimension of design activity (in this aspect, we share our aim with Prats et al., Reference Prats, Lim, Jowers, Garner and Chase2009). JRGs were identified as a promising context for observing interpretation-related phenomena, for various reasons. Fundamentally, the ambiguous forms used in such gardens naturally invite interpretation. Additionally, since JRGs are meant for viewing, function largely coincides with interpretation (Berthier, Reference Berthier and Parkes2000; Mansfield and Richie, Reference Mansfield and Richie2009). This renders such gardens literally as great “sandboxes” for experimenting with seeing and action, without the burden of strict technical requirements, which greatly complicate such processes.
Considering the loose nature of our task, several clarifications are due in terms of its characterization as a design activity. Primarily, drawing on the three viewpoints on the nature of design by Hay et al., we take the view of design as exploration. By this view, designing can be seen as a process in which the problem and its solution co-evolve, as the designer explores (and redefines) the problem and solution spaces (Hay et al., Reference Hay, Duffy, McTeague, Pidgeon, Vuletic and Grealy2017). Moreover, keeping in mind our aim to capture the designer's perspective during interpretation, a key methodological difference with previous works (see the "Background" section) is our level of analysis – drawing on the terminology in Stempfle and Badke-Schaub (Reference Stempfle and Badke-Schaub2002), and this study takes a “nano-perspective” and examines how the contents of interpretation can be represented and visualized at specific moments in time. Design sessions in this study are thus best seen as sources for extracting interpretation-related events, rather than complete accounts of how design problems are solved from start to finish. This is somewhat similar to creativity studies introduced by Finke et al., where subjects are required to invent a device by synthesizing several given shapes, without the context of a design problem or a specification of functional requirements (Finke et al., Reference Finke, Smith and Ward1996). Indeed, such studies have proved fruitful in advancing our understanding of cognitive activity in designing.
Considering our aims discussed above, JRGs were found advantageous for our purposes at the analysis phase as well. By comparison with sketching activities, for example, which have been extensively used in studying interpretation (see, e.g., Goldschmidt, Reference Goldschmidt1991; Schön and Wiggins, Reference Schön and Wiggins1992; Suwa and Tversky, Reference Suwa and Tversky1997; Suwa, Reference Suwa2003; Prats and Garner, Reference Prats and Garner2006), JRGs consist of a limited number of discrete elements with clear boundaries, which allow a close analysis of visual interpretation-related phenomena (e.g., see Van Tonder and Lyons, Reference Van Tonder and Lyons2005). While sketching offers a wide range of expression and ambiguity (which are highly important when studying interpretation), utilizing sets of clearly defined forms as design materials has enabled to carefully track how specific objects are matched with interpretations. Therefore, an analysis of designer's utterances when referring to such forms enables to draw sound inferences regarding the external referents of internal content.
Lastly, we point out that, despite the loose nature of the activity, the task does impose a few constraints – the designers are instructed to restrict their designs to the tray, to use elements from a limited set which is pre-determined, and to not stack them (see the “Task, environment, and setup” section). This provides a general framework for the activity, which does not impede the emergence of interpretive events (as it enables one to freely explore and devote attention to various aspects, beyond the need to fulfill strict requirements).
Task, environment, and setup
A task of designing a JRG using a miniature model was devised. In each session, a team of two designers worked together to design a “garden” according to their personal preferences. The “site” consisted of a small wooden tray (38 × 26 cm) covered with a thin bed of white sand (a small rake was available as well). A collection of small rocks in various shapes, sizes, and colors were supplied as design materials (23 in total). Subjects were requested to confine their designs to the limits of the tray, and refrain from stacking rocks (in accord with the fundamental guidelines of traditional practices; Slawson and ZoÌen, Reference Slawson and ZoÌen1991; Takei and Keane, Reference Takei and Keane2001) (Fig. 2).
Fig. 2. Task environment and setup.
To generate design protocols, subjects were asked to voice out their internal dialogue and design in a “think-aloud” style. Each session was preceded by a brief orientation, during which subjects attempted to think-aloud. We avoided modeling and instead simply prompted subjects for more details, if speech seemed rather succinct or non-descriptive (Gibson, Reference Gibson1997; Charters, Reference Charters2003). Occasionally, when lapsing into long periods of silence during the session, subjects were prompted to share their thoughts, but were neither instructed on how they should go about doing so, nor were evaluated on their performance. Evidently, the teamwork setting enabled to elicit key utterances in a rather natural manner, since the subjects had to communicate by speech when negotiating the task.
All participants hold a bachelor's degree in a visual design discipline (Architecture, Industrial Design, etc.) at minimum, with some holding a Master's degree as well. Participants were selected by their design background and work experience from two groups: novice designers (less than 2 years of practical experience) and professional designers (more than 5 years of practical experience); under the assumption that this will enable us to observe interpretation at different degrees of complexity. To facilitate communication, invited subjects were asked in advance to find a partner of an equivalent skill level that he/she would like to collaborate with.
Sessions were capped at 90 min, and the subjects were free to use their time for producing one/multiple designs. Actions and utterances were recorded using two video cameras (top and front). In addition, an observer who moderated the session took notes regarding additional aspects of the activity which seemed suggestive, such as bodily movements, facial expressions, etc.
Six couples (12 individuals) were observed in total. One session is taken as a case study for close observation and used for demonstrating our proposed method for the visualization of mental content in design. In addition, processed data from the other sessions, which reflect similar interpretive practices, are summarized as a single figure per session.
Processing and analysis
Sessions were transcribed from the audio recordings. To establish a rich documentation, our transcriptions included literal utterances (by the subjects and by the observer) and the observer's written comments. Images presenting various states of the physical model were extracted from each video. Each design move, defined as an operation on knowledge (Goldschmidt, Reference Goldschmidt2014), was represented by a single image. The audio-visual data were synthesized into text-image sequences, as a unified documentation of the design session.
Processed data were analyzed to track the assignment of interpretations to physical objects by: (1) reviewing and segmenting the data into short and meaningful episodes; (2) identifying events of assigning or modifying interpretations, which may be explicit (pointing at a rock and saying: “this is a mountain”) or implicit (declaring that the sand is “a pool of water”, which enables to infer that the rocks on it represent objects found in water); (3) using the above to write a verbal description of the events in the session, as a summary of the process; (4) identifying points in the description where surprising events occurred; (5) attempting to graphically place the entities involved in the event within mental spaces, as representative of the designer's perspective at the relevant time; and (6) examining the changes in the designer's perspective before and after the event, by relating the structure, the gestalt grouping imposed on it, and its metaphorical interpretation. Note that, as means for verification, conclusions drawn from our analysis are often not only with utterances but also with visual data (such as bodily gestures like pointing, etc.) (Sweetser, Reference Sweetser2007; Murphy et al., Reference Murphy, Ivarsson and Lymer2012).
Results
In this section, we provide an account of an episode from a chosen design session, in which we can closely observe how designers engage in interpretation and assign meanings to design representations (as a basis for demonstrating our visualization method in the following section). The account is accompanied by figures representing key events from the session (Harry and Neal's forest). Following this, outlines for the other design sessions are given (interpretive events in other design sessions).
Notice that each figure contains a numbered sequence of utterances and corresponding images of the structure at that time. When an important utterance seems ambiguous, we have marked the verbal expression in a bold font and the corresponding part in the image with a white line. Occasionally, figures are accompanied by a graphical explanation, as a useful visual summary.
Harry and Neal's forest
Harry and Neal are both novice architects who hold a Master's degree. Harry has interned in architectural offices both in Germany and in Japan (2 years in total), while Neal has interned for 1 year in a Japanese architectural firm.
Their session starts with a general discussion of the theme. Neal wishes to avoid traditional JRGs, which he is personally less fond of. Preferring gardens, which contain more greenery (“natural garden”), he reaches for “something which reminds me … (of a) green thing” and grabs a rock which he could see as a tree-top. By placing this “tree-rock” in the garden space, he concretizes the foundation for their design. This is followed by a brief failed attempt of seeing-as (“this really looks like (just) a rock”), to which Harry responds by proposing to reorient it to stand vertically, which would enable seeing it as another tree. This prompts the immediate identification of a third rock which can fulfill the same role by Neal (“this too”). Having determined the first interpretation, and projected it onto several elements, an interpretive world is about to emerge. Harry proposes to progress by placing “them” randomly, that is, he does not refer to a specific rock, but rather to the number of “tree-rocks” they have found. When three rocks are positioned in the garden space, Neal suddenly blurts out in surprise – “that area looks like a forest” (Fig. 3).
Fig. 3. Case 1; emergence of a global interpretation of a “forest”.
The interpretation of a “forest” is followed by a series of adjustments, during which Harry concludes that “it is very interesting to put them vertically”. Accordingly, he changes the orientation of the cornerstone of the garden to match the orientation of the other elements. Examining two “tree-rocks”, Harry further notices a height difference between them, which he interprets as a being a “small tree” and a “bigger tree”. He does not seem content with simply acknowledging these differences, but goes on to explain their existence – “it's close to a bigger one, so the seeds from the bigger one fell down, and then the smaller tree grew close”. The interpretive world is endowed with a dimension of time, and its elements are further linked on the interpretive level (Fig. 4).
Fig. 4. Case 1; deepening the interpretation by considering the dimension of time.
In a further attempt to tie the different elements together, a more detailed unifying interpretation emerges. Ingeniously, Harry proposes that a bird carried the seeds of a tree from one place to the other, which explains the spatial disparity of the tree groupings (more in the “Visualization of mental content in interpretation” section). Following this, the subject then reports – “for some reason right now, I suddenly see…a path here or a street”. This observation, which seems to have resulted from focusing on the groupings and negative space, serves to encourage further elaboration of the interpretation, by segmenting the space into meaningful units or subspaces (tree groups, paths, etc.).
Interpretive events in other design sessions
From the following figures, it can be seen that the assignment of metaphorical meanings as an interpretive device was utilized in the other sessions as well. For example, in one case a subject refers to a few rocks as “towers” (Fig. 5, photo 7); in another case, a group of rocks stands for a “lotus flower” (Fig. 7); similarly, we can see how a rock is referred to as a “shark's fin” (Fig. 6, photo 9), etc.
Fig. 5. Case 2; a castle with a fountain.
Fig. 6. Case 3; a beautiful fish.
Visualization of mental content in interpretation
We propose to approach the study of interpretation as an activity closely tied with the generation of metaphoric images, in which mental content is organized. This approach sheds some light on the manners in which designers view and navigate design situations when working with design representations. We represent mental content lying at the heart of interpretation to enable visualizing various aspects of the designer's thinking at certain moments in time. Our modeling and visualization approach is demonstrated (using the episode in “Harry and Neal's forest”) and then utilized as a means for getting insights into design cognition.
Into the designer's head
Interpretive descriptions assigned during interpretation activity both reflect and affect how the designer sees the structure, while alterations to the structure may cause changes in the description as well. In this sense, tracking changes in the structure and its interpretation can yield important information regarding the designer's thoughts during the design process.
Figure 10 provides a simplified example of modeling the designer's perspective, using our proposed approach, which draws on Fauconnier's mental spaces (Fauconnier, Reference Fauconnier1994) and on Gero & Kannegiesser's situated FBS framework (Gero and Kannengiesser, Reference Gero and Kannengiesser2004). Two spaces are drawn: M1 (external reality) and M2 (metaphoric interpretation world); these are contained in the external and interpreted worlds of the situated FBS framework, accordingly. Recalling the event in Case 1 where the designers have seen the space as a “forest”, we model it as follows: since in external reality several rocks were placed on the tray prior to the appearance of the “forest”, we placed three elements in M1 which represent these (each corresponding with a single rock). As each rock was interpreted as a “tree”, we connected each element in M1 with its interpretive counterpart in M2 (each representing a single tree in the designer's imagination). Furthermore, since the trees were finally seen as a forest, they are grouped within the metaphoric interpretation world and linked to a forest entity. Finally, the fact that the designer had “seen” the forest implies a gestalt grouping of the physical elements, noted by grouping the elements in M1, as well as by linking this group to the forest in M2.
Fig. 7. Case 4; lotus flower with fresh leaves and ones which “already grew”.
Fig. 8. Case 5; welcoming an unexpected element as a boundary for the design.
Fig. 9. Case 6; alien structures or islands.
Fig. 10. Visualizing the designer's perspective using the proposed modeling approach.
The link between an element and its interpretation was named as an “interpretive link”. These links connect physical elements with their imaginary counterparts. Establishing interpretive links may be done by directly naming an element metaphorically (“this is a tree”). Interestingly, such links are not merely referential, but generative as well – the “tree” did not exist as an entity in the metaphoric interpretation world before the rock was assigned with this description. Another way to establish interpretive links is by describing an element's role in the design (“this part serves as a mediator”). In both of these cases, establishing interpretive links shapes the designer's perception of the original object and its contextual space.
Finally, a clarification is due with respect to the entities populating these spaces. Two types of entities are presented in Figure 10: physical elements (white) and interpretive ones (gray/black). The latter are often associated with entities which are external to the design (denoted in parentheses). Furthermore, they are occasionally elaborated into specific kinds – for example, “tree” is later characterized as a “pine tree” or simply “pine”.
Metaphoric interpretation worlds
By using the above modeling approach, we can form maps or diagrams (such as Fig. 10), which can be regarded as “pictures” of the designer's mental content, at certain moments during the design process. What can these pictures tell us about the designer's mental world? Fundamentally, these summarize and visualize: (1) the metaphoric entities which the designer is conceptually employing to understand the structure at a certain moment; (2) certain relations these bear to the structure or its representation. As such, they enable us to make basic and important observations regarding how designers engage in interpretation.
First, as shown in the “Harry and Neal's forest” section, establishing interpretive links populates the metaphoric interpretation world with entities (in this case, “trees”). It is important to notice that the contents of this world should not be seen as an arbitrary agglomeration of independent items. Rather, they are entities chosen based on properties of the structure (e.g., based on the form resemblance between pine trees and elongated rocks), which are organized into meaningful wholes (the “trees” make up a “forest”). In a similar manner, the “leaves” in Case 4 make a “lotus flower” (Fig. 7), etc.
Second, the designer's mental content is not simply projected as a premade interpretation onto the structure. Instead, interpretations seem to be dynamically constructed on-the-fly, as a response both to one's own thoughts and actions and to the changing structure. Continuing with Case 1, after an initial interpretation was assigned (”tree”) by focusing on a single rock (”this looks like a tree-top”), the subjects resolved to freely place several rocks in the space. This was immediately followed by seeing the design as a “forest” (Figs. 3–6). The emergence of this forest, which served as the global interpretation (or setting) of the design, may be further elaborated as follows: first, an interpretation was assigned to an individual element. Then, several elements were placed in the space, inheriting this interpretation, and consequently (also considering their similarity) perceived as a group. Finally, a unifying interpretation was proposed. The global interpretation of the “forest” was thus generated as a consequence of (1) assigning previous interpretation of “trees”; (2) re-perceiving the physical elements under a different gestalt, which demanded adapting the interpretation by relating the parts (trees) with a new whole (forest). This episode supports Suwa and Tversky's suggestion that shifts in focus are important drivers for design (Suwa and Tversky, Reference Suwa and Tversky1997). In this case, a shift in observation in terms of groupings seems associated with the organization of the metaphoric interpretation world as a whole.
Third, rather than strictly bounded or well defined, designers’ interpretations seem flexible and open for further structuring. Surprisingly, the designers in Case 1 resolved to creatively expand the interpretive framework, by detaching from the physical dimension and introducing interpretive entities which do not have any external representation (tree-seeds and birds). This helped in enriching their metaphoric interpretation into a narrative (which is interesting in itself from a perspective of creativity in design), but more importantly, in strengthening its relation with the structure, via its elaboration. The placement of elements in the space was further justified by a chain of causation, in which both the physical and the purely mental played a part. Such flexibility can be observed in Case 5 as well, in the way that the designers serendipitously structure their design, as a response to an accidental line on the sand – the line is embraced and interpreted as a “boundary” between two areas of full and void, thus serving as a dividing line for the setting, which otherwise remains largely undefined (Figs. 6–8).
Finally, a certain type of conflict seems to play a central role in structuring the metaphoric interpretation world, motivating designers to coordinate their perceptions and interpretations. More specifically, when the expectation of designers from elements conflicted with self-generated rules, the designers’ interpretive faculties were triggered into action. An intriguing example of this is evident in the above sample from Case 1: the subjects were building a “forest”, where each rock was interpreted as “tree”. Furthermore, two kinds of trees were identified: “pine” and “opposite of pine” (deciduous). One subject noticed that a rock characterized as “pine” stood physically far from its group and wondered why this is so (i.e., how could it be that it “grew” out of the “pine area”). To resolve this, two new elements were added to the interpretation: a bird which carried a pine seed from across the forest (Fig. 11).
Fig. 11. Conflict as a driver of interpretation in design.
The above extract can be understood as a discrepancy between two spaces, as shown in Figure 12. M1 represents the subject's view of external reality at the initial state, where certain rocks were grouped (those in the “pine tree area”) and others remained outside the group (e1 and e7). M2 and 
${\rm M}^{\prime}_2$ represent two possible metaphoric interpretation worlds, and M3 is the one which results from these. In M2 the grouped rocks, as well as one rock outside the group (e7), are all assigned with the interpretation of “pine trees”. This space represents how the designer sees the situation on the interpretive level without considering the implicit rule “all pine tree should stand together”. 
${\rm M}^{\prime}_2$, on the other hand, represents the understanding of the situation by considering the implicit rule. Accordingly, in this space, e7 also appears within the “pine tree” group, reflecting the designer's expectation of following the rule. The conflict stems from the fact that, while the designer expects all pine trees to stand together, he perceives a strong grouping which excludes one element (e7). M3 shows the designer's perspective after resolving the conflict – the fact that the implicit rule is broken is coped with creatively on the interpretive level, by forming the bird story, which enabled e7 to both be out of the group on the perceptual level and in the group on the conceptual level.
Fig. 12. Visualization of negotiation between different perspectives using our approach; drawing on Fauconnier (Reference Fauconnier1994) and Gero and Kannengiesser (Reference Gero and Kannengiesser2004).
Discussion and conclusion
Applicability to design activity
As a basis for the claims which follow, we open this section by applying our approach to a well-known example of architectural design – Michael Arad's plan for the World Trade Center 9/11 Memorial. Arad's design, which surpassed more than 5000 proposals, is widely acknowledged as a creative employment of metaphorical interpretation in designing. We have relied on an interview held with the architectFootnote 1 in which he described his design process and visualized two specific moments in time, as a means for getting insight into his interpretive practice. Our visualization (Fig. 13) and its explanation are given below.
Fig. 13. Application of our method to the design of the World Trade Center Memorial by Michael Arad (photo by Foundry Co Pixabay, with permission).
First, ta (consisting of M1 and M2) represents a moment in time where the architect began pondering an initial solution, which was still quite vague. He clearly mentioned that the first thing he envisioned was that a water element (w) would be incorporated into the design, while other elements were yet undecided (?). Additionally, at ta the tower ruins (tr) were not yet cleared from the site, and were thus perceivable by the architect. He further expressed his basic approach for the project as follows: “The way people in New York came together affected me greatly…I felt a desire to respond to that”. The concept of “getting together”, paraphrased here as unity (u) reflects his basic conceptual standpoint, which is further elaborated as imagined people (p) who come together and are thus grouped in the mental image. It is important to notice the disconnection between M1 and M2 at this moment, reflecting our lack of knowledge regarding how the external and interpretive elements relate (if at all).
tb (consisting of 
${\rm M}^{\prime}_1\; {\rm and} \;{\rm M}^{\prime}_2$) represents a later moment in time at which the solution was further developed. Here, a decision was made to form two main spaces as excavations (e1 and e2) in the ground, in place of the former towers. Each of these consist of various elements, two of which are mentioned here. The first of these is a void element (v), and the second a water surface element (ws). Regarding the former, the architect has explained, somewhat poetically, that the two voids denote “a sense of…continued absence” (a), thus commemorating the former Twin Towers by serving as their permanent marks (m). Regarding the latter, Arad's description is revealing of the interpretive depth of the project:
“…you can see each strand of water as it goes off the edge of the weir, appearing as separate…By the time it reaches the end, the clarity dissipates, and it becomes a woven tapestry of water. I think it speaks to individual and collective loss – bringing together many lives into one”.
Accordingly, each water surface (ws) contains water strands (st) which are associated with individual people (p) that together form a sense of unity (u).
Our visualization opens up a window to Arad's mental world, by enabling to ask specific questions regarding his interpretive practice, and the way it shaped the course of design. One key question concerns the process by which the final concept for the design was developed. Namely, noticing that in ta external reality (M1) and imagination (M2) are not yet connected, how did he succeed in bridging the two and reach the solution in tb? More specifically:
(1) how did he develop the visual expression for the sense of unity in
${\rm M}^{\prime}_2$ that first appeared in M2? His commentary regarding strands of water (which flow and unite) reflects success in tying his two initial motivations – using water, and embodying a sense of unity. However, it is not clear how this link was formed. At first water was conceived of as “water”, while later it was viewed as “strands of water” which go through a process of unification. Therefore, we can ask – what caused this elaboration in his metaphorical view of water?(2) as an extension of the above, we may also ask regarding his choice of a waterfall-like structure, rather than another water-based architectural element (e.g., restricting the design to a simple reflecting pool). Embodying the sense of unity by allowing strands of water to join seems to demand visible water flow. However, it is difficult to know whether the choice of this element acted causally in elaborating his interpretation of water, or was the case the other way round.
(3) furthermore, focusing on
${\rm M}^{\prime}_2$, why did the architect decide to enrich the initial concept of “unity” and add the notion of “absence”, which eventually seems to have come to dominate the project? What made him interpret the ruins as something which demands removal rather than reconstruction? Had he initially viewed the ruins as demanding reconstruction or restoration, the idea to concretize their absence in the form of voids would be excluded from his search space?
The above suggests that understanding the architect's design process depends on our ability to carefully trace interpretive events and inquire into their potential sources. Tracing the sources of such events requires the capacity to capture the architect's perspective at specific times, which can uncover critical changes in the way the project is understood and therefore navigated.
Implications
This work draws attention to the importance of momentary events in the study of design cognition, by focusing on capturing one's perspective at specific instances in time. Our method facilitates a “nano-scale” inquiry into the ways in which designers engage in interpretation, by focusing on the analysis of metaphoric interpretations themselves, and including them as explicit entities when visualizing design processes.
In comparison with our method, existing approaches for design visualization are suitable for mapping design processes at a lower resolution. Linkographs, for instance, commonly consist of hundreds of design moves and links (Goldschmidt, Reference Goldschmidt2014), which provide a global perspective on the relations between events in design. In our example (section "Applicability to design activity"), it may be possible to identify the relation between ta and tb (which we postulated) using a linkograph. However, in this case, we would still need to elaborate on the nature of this relation, which is rooted not only in the activity itself but also in the metaphorical mental content which drives it. Similarly, while creative segments (Sun et al., Reference Sun, Xiang, Chai, Wang and Huang2014) are useful for making closer observations relative to linkographs, their primary focus is on the description of the actions and would thus need to be complemented with an account of the momentary mental content which facilitates it in the background. Since design moves and paths are selected on the basis of one's understanding of the current situation, visualizing and inquiring into this content are essential for complementing existing methods, toward the elucidation of “on-line” cognition in design.
From the perspective of design cognition research, our method thus deepens our understanding of interpretation in design, by enabling to systematically inquire into the designer's perspective, when he/she is designing. Consider Goldschmidt's task of designing a residential unit based on the concept of a “cube” (Goldschmidt, Reference Goldschmidt1988); if a designer conceptually refers to a house as a “cube house”, we can now ask – what metaphorical meanings and mental images are projected onto the various architectural elements, and when? How do these drive his/her courses of action? Such questions, which are of great importance for studying design cognition, may be approached by modeling the relations between the (architectural) elements, their interpretations, and their groupings, using our approach.
Considering the above, one important human ability which can be studied using our method is that of “seeing as”, discussed by Goldschmidt (Reference Goldschmidt1991) and later by Schön and Wiggins (Reference Schön and Wiggins1992) as key for visual interpretation in design (see the “Metaphor and imagery in design interpretation” section). Since our method does not presuppose perceptually given objects, however, it is not restricted to this phenomenon (consider Michael Arad's water element which was imagined and selected as a component of the structure prior to producing any plan or layout for the project). Therefore, it may be used as a means for gaining a deeper understanding of the designer's view, both with respect to interpretations of external representations (as in “seeing as”) and with respect to generating them on the basis of mental images expressed in linguistic descriptions. Borrowing Searle's terminology regarding how our minds relate with the world, the former may be generally referred to as a process with a “world-to-mind direction” and the second as a “mind-to-world direction” (Searle, Reference Searle2004). Since in design exploration both the problem and solution are jointly conceived (Hay et al., Reference Hay, Duffy, McTeague, Pidgeon, Vuletic and Grealy2017), understanding cognitive processes in design requires to explain not only the former (i.e., how our thinking is shaped by what is in front of us) but also the latter (i.e., how the mind shapes reality in its image). To do so, it is essential to strive toward detailed accounts of the mental content which underlies designers’ thinking and action.
In a broader sense, this work deepens the relation between linguistics and cognition in design studies. In an era where design cognition research is in need of new tools and methods for moving forward (Hay et al., Reference Hay, Cash and McKilligan2020), and where new technology relies on insight from design cognition (Goel et al., Reference Goel, Vattam, Wiltgen and Helms2012), such interdisciplinary efforts to establish new ways for inquiring into the mental realm may prove of great value.
Limitations and future work
Key limitations and possible future directions are discussed. First, our description of metaphoric interpretation worlds serves as an account of the designer's perspective and enables us to derive information about the mental content held at a certain moment. However, much work will be required to elaborate on how this content is organized, beyond the basic linkages suggested in this study.
A possible approach would be to extend our model to explicitly include the notion of a “frame” (Dorst, Reference Dorst2015) that the designer holds at certain moments, which governs his/her interpretation process (albeit temporarily). Considering Beckman and Barry's two-phase process of framing and re-framing (capturing the process within an initial frame which is then improved upon; Beckman and Barry, Reference Beckman and Barry2009), it seems that a clear description of the constituent and boundaries of interpretive frames is essential for understanding creative processes in design. Doing so would require to move beyond metaphorical interpretations and include additional layers, which contribute to the activity as a whole. As an example, consider the social aspect of designing (e.g., the way designers see their personal role during the task) which surely shapes their perspective, and thus their thoughts and actions. This aspect could be hypothetically studied by harnessing additional tools from pragmatics (Searle, Reference Searle1969; Levinson, Reference Levinson1983) for understanding the context in which the discourse is embedded.
Second, our task was designed to reduce the complexity involved in real design situations, and enable a deep analysis of human interpretation of design representations. The choice of JRGs has enabled to achieve this at the cost of setting aside the important aspect of function. This aspect should be carefully integrated into further inquiries, while keeping in mind the need for providing sufficient opportunities to engage in interpretation. As a midway, we propose the possibility of designing miniature gardens which are associated with interpretative practices, but are not limited to viewing. For example, in Chinese traditional gardens, one can sit, walk around, observe, and even touch the decorative rocks (Zhang, Reference Zhang2018). Therefore, to facilitate interpretation, framing and re-framing, while attending to technical or functional aspects, we recommend that the task duration is significantly increased.
Conflict of interest
The authors declare none.
Yuval Kahlon, PhD is an Assistant Professor at the Tokyo Institute of Technology. He born in Kfar-Saba, Israel. He studied building and environment design at the Shenkar College of Engineering and Design, Israel, and engineering science and design at the Tokyo Institute of Technology, Japan. He was awarded by the Shenkar College of Engineering Design and the Tokyo Institute of Technology for Excellence. He conducts research for developing intelligent computational design systems, by integrating insights from linguistics, cognitive science, and artificial intelligence, with empirical data from design activity.
Haruyuki Fujii, PhD, Architect is a Professor of Architecture and Design Science at the Tokyo Institute of Technology. He born in Tokyo, Japan. He studied architectural design and engineering at Waseda University, Japan, and philosophy focusing on computational linguistics at Carnegie Mellon University, USA. He studied design computing and cognition at the University of Sydney, Australia. He has been constructing a methodology for design science that bridges the subjectivity in designing, such as subjective insight, illogical thinking, individual localization, and the objectivity in natural science, such as objective grounding, logical thinking, and universal explanation.
Open access
