Decorative, Evocative, and Uncanny: Reactions on Ambient-to-Disruptive Health Notifications via Plant-Mimicking Shape-Changing Interfaces

Many studies have looked into how notifications can be combined with personal health data to influence people towards healthy choices [7, 73], or which situations are best for notifying the user [25], particularly in the area of Just-in-Time Adaptive Interventions (JITAIs) [27, 44, 53, 59, 69, 72]. Similar to JITAIs but beyond the health domain, the question of when a disruptive notification should occur has been covered broadly, such as the appropriate contexts when interruption is suitable [28, 49]. Our research extends and complements these prior works by examining a novel dimension of notification, that is, the gradient in notification portrayed in a physical system such as a S-CI. While our work also does consider timing, we focus on the escalation of a repeated message and when disruption should be judiciously used rather than the tailoring of an initial message’s placement based on prior or current data.

Researchers have also looked into the categorization of notifications to determine when people should be disrupted by a message, e.g. by categorization rules set by users [18, 35], or via machine learning to replicate users’ prior actions [50]. Although we also investigate decisions people make about notification disruption, we emphasize discerning how individual messages may gain importance based on situation and timing, compared to the prior works that looked at a singular importance categorization per notification. Interruptions and disruptive notifications carry a negative connotation, and indeed, the negatives of interruption have been well researched, whether physiological and psychological effects on people [2, 32, 78], or its effects on their productivity [11, 33]. However, ambruptive technology does not only focus on disruption, as it employs multiple levels of noticeability, and thus we also reflect on work on the subtler side of notification. Pohl and colleagues charted the region of subtle interaction in HCI [61], specifically with the subsection of studies that aimed for non-intrusive notifications. Similarly, the large design program of Ambient Information Systems mainly targets non-disruptive notification, but in fact supports the concept of ambruptive technology within its definition.

Although AIS are generally designed to be non-distracting, AIS taxonomies offer a full range of notification levels as a key dimension—both Pousman & Stasko [62] and Matthews et al. [48] list levels of interrupt and demand attention. Thus, although AIS initially aim to reside in the periphery of users, they do not need to remain there. Little work has been done in designing systems that facilitate multiple notification levels. The Bus LED Display was an example application that flashed its LEDs to change a peripheral display into an interrupting notification, but acted only as a proof of concept and differs from our work in display type (light-based vs. S-CI). [47]. Angelucci et al. proposed an interface that “distracts users only if the severity requires it” [4], assessing mockups that mapped colors to severity levels, in either a horizontal scrolling or tab-based interface. Despite the similar overarching goal, our work differs from theirs in display type (screen vs. S-CI), context (telecommunication network fault vs. personal health), and study design (design and usability study vs. RtD approach).

The AIS space has been paired with a health-related context for multiple studies—a display-only example being Spark’s use of informative art to visualize physical activity [19]. Additional studies have investigated further in the ability for AIS to actively promote healthy activity, e.g., Fish’n’Steps using an ambient display to link a cartoon fish’s growth and demeanor with a users’ footstep count [45], and Rogers et al. investigating three ambient displays intended to promote stairs use versus the elevator [65]. In other health-related avenues, García-Vázquez et al. created three AIS to encourage elders’ medication compliance [22], and Wwall encouraged hydration through the use of a display wall [76]. While aimed at assisting similar health goals, these studies all focused on a single notification level and did not use notification escalation as our work does. MoveLamp was a study that touched on increasing notification level based on user activity, changing the color and brightness of a light display to promote physical activity [20]. However, it differs from this work both in the non S-CI aspect of the light display, its main focus on behavior modification, and in not focusing on when its participants wanted to be interrupted—instead using a predetermined formula for escalation.

AIS and S-CIs have been linked together since Dangling String [74], but do not need to coexist. Similarly, ambruptive technology is not limited to systems that use shape-change. However, exploring an abstract concept unfamiliar to the majority of users is a difficult endeavor both for researcher and participant. Therefore, we created a concrete example, in this instance an S-CI, to elicit reactions from users on which scenarios, if any, would warrant an ambient-to-disruptive escalation. Due to our qualitative focus in this study, this work contributes towards understanding the user experience of shape-changing interfaces, one of the grand challenges of S-CI [3].

Specifically looking at indicators for health-related systems, AIS that use shape-change have also been used throughout the years, from wall-like surfaces to portray biometric information [40, 77], or sculptural elements encouraging good posture [26] and work breaks [34, 68]. These share similarities to our work as they examine health-related scenarios, but none consider further behavior when a user fails to respond, and all remains solely on either the calmer side or disruptive side of notification level dimension. Of note is the participant of Breakaway [34] indicating that the system could be easily ignored if she was too busy—based on one’s situation, this could be a positive or negative trait. Singh et al. [68] created design probes to explore S-CI notifications for Repetitive Strain Injury (RSI) breaks, having separate disruptive and non-disruptive versions. However, their participants preferred non-disruptive notification, and all subsequent study phases aimed to reduce disruptive techniques. We expand this knowledge in health-related notifications by not only looking at a larger variety of health goals, but explicitly probing when varying levels of disruption would be accepted. This would require the creation of artifacts with the fluidity to move between the ambient-to-disruptive axis as needed.

Sienna wishes to be reminded to take her medication, which is scheduled every five hours. She loads a medication reminder app and schedules alarms at 9 AM, 2 PM, and 7 PM. At 9 AM, her device chimes with a notification, and she successfully takes her dose. However, at 2 PM, she is in the middle of an important video call and does not notice the notification. As she immediately pivots to an assignment after the call, she does not see the notification until time passes the recommended dosage period. To avoid being overmedicated at her next scheduled time, she skips her dose.

In an alternate timeline, Sienna uses an ambruptive technology system that uses shape-change for notification. At 2 PM, the S-CI subtly begins moving, but Sienna is again in an important video call and does not have the capacity to concentrate on anything else. At 2:05 PM, with no response from the user, the S-CI moves in a slightly more noticeable manner. Since Sienna is deep in thought on her assignment, she notices the alert peripherally, but completion of her task holds higher priority at the time, and she ignores the warning. A few minutes later, again with no user response, the S-CI now chooses a higher notification level and moves with a vibrant, distracting movement—forcing attention, as Sienna is now in danger of missing her medication time slot.

We began with conceptualizing different ways a plant-mimicking interface could be used to represent data, using both sketches and working mechanisms. This led to 25 ideas along multiple categories: shape: a change in the overall structure of the plant itself; orientation: moving the plant body along different angles or axes; plant state: portrayal of different aspects in a plant’s life cycle such as health state or fruiting phase; and external forces: change to the plant caused by a foreign source, such as weather.

Due to our interest in noticeability, we explored other themes researchers had used in their work in notification systems. Klauck et al. [37] experimented with varying the speed and size of movements and their effect on noticeability, and we planned to represent this in our own artifacts by varying speed and distance parameters. When sharing our initial prototypes with colleagues, we noticed that several people mentioned sad emotions from seeing the plant in particular positions (e.g., a drooping plant), and we investigated further into emotion portrayal in shape-change. Lee et al. [41] rendered a 3D shape with various levels of bending and convexity/concavity and measured how such poses interacted with user emotional reactions. Strohmeier et al. [71] investigated the correlation with emotion specifically to various shape changes. To gauge user reactions to our attempts in emoting sadness in our artifacts, we decided to use shapes that have correlated to sad emotions—mainly concave poses in a downward position. Finally, we explored cognitive theories on bottom-up saliency models mentioned in the psychology field [23, 38], but did not find many examples of what was considered salient—less so in the use of movement or shape change. However, experimenting with various motion patterns led to the accidental discovery of particular movements that led to uneasiness when watched. Intrigued with the effect it had, we decided to investigate naturalness–unnaturalness as another axis to experiment along as a possible use of salient notification.

To illustrate our ambruptive technology concept, characterized as systems with the capability to adapt their notification level throughout the ambient-to-disruptive axis, our artifacts required the maneuverability to portray different levels of disruption. With our three themes of noticeability, we reviewed our plant-mimicking conceptualizations and chose two mechanisms (Figure 4) that could represent different values within the related variables we identified. These would be used to explore how shape-change could be used for both ambient and disruptive notifications, and in turn, capture participants’ initial reactions towards ambruptive technology.

Artifact parts were designed in CAD and printed with a FDM 3D printer, with some post-print sculpting using heat. The artifacts were controlled by a Wemos-D1 clone microprocessor development board. Each artifact used custom code specific to its mechanism and was programmed to perform specific scripted actions on trigger. The artifacts can poll a remote web API and retrieve instructions, but for this study’s needs, movement was simply triggered over serial communication via USB.

The final version of the first artifact, shake-artifact (3a ), uses a pinion and rack mechanism to shake the leaves of the device, intended to portray wind naturally blowing through the plant (Figure 4 left). Control parameters for the shake-artifact were distance: the range of linear movement used by the artifact; frequency: how often a shake event occurred, and speed: how long the shaking movement’s back and forth action should take. With the shake-artifact, we chose a slower, subtle shake, and a much stronger heavy shake movement as our two possible actions. We note that the names used for the artifacts and movement types are only for clarity in this paper; the artifacts were not named during the study to prevent bias.

The final version of the second artifact, droop-artifact (3b ), manipulates attached monofilament to curl and straighten the plant stem (Figure 4 right). Control parameters for the droop-artifact were: distance: the amount that the stems were pulled down or released up; speed: how slowly or quickly to take from one position to another; direction: whether the movement rose up or down; and timing: when to perform a move. Selected movements for this artifact included basic up and down actions at various speeds. We also included four additional scripted movements, meant to delve into different ways movement could be used as effective notifications. Heartbeat used a duplicated abrupt start-stop movement to display a tick movement twice. Hiccup kept the plant in a down position but would pop up abruptly when trying to notify. Lift had the plant move its leaves downward so forcibly that it would lift itself slightly out of its own container. Panting used a smooth loop of undulating stems of the plant that was meant to look as if the plant were panting.

Participants were recruited over Facebook and Reddit, email mailing lists, and flyers across a college campus. A total of 169 respondents took a screening survey; 122 did not qualify due to answers or suspicious entries, and ultimately 10 participants were chosen, aiming in diversity of age groups and health goals. The criteria for participation included (with reasoning for each criterion):

Table 1 summarizes participants’ strategies and health goals along with their age range and location. A screening questionnaire, consent form, and sample interview were reviewed and approved by the University of Maryland’s Institutional Review Board.

For each participant, we held an hour-long video interview using a semi-structured interview process. We asked the participant to describe their health goals, and listened to the participant’s accounts of what strategies they use to keep track of their health-related activities throughout the day. We then introduced the concept of ambient notification by allowing the participants to experience the subtle shake portion of the shake-artifact, followed by the strong shake movement, an example of escalating notification. After hearing the participant’s reactions to these, we switched to the droop-artifact and played the four escalated notification movements: heartbeat, hiccup, lift, and panting (Figure 6), then held a discussion with the participant over which movement best represented subtle and escalated notification, and why. While some movements could relate to each other (e.g., the subtle shake amplifying into the heavy shake), the droop-artifact’s movements were very disparate, and so thumbnails of each movement were displayed during the presentation of those videos. Participants were invited to request the replaying of any video during our discussion.

We then explored the plant’s capacity of displaying state via shape-change rather than just a notification indicating the need for action. We showed various up and down movements and positions of the droop-artifact, and then asked the participants what they felt was best to display their state in an S-CI. Finally, we followed up with general questions around how the participants imagined these artifacts in their lives, such as choices in location, size, and aesthetics, and explored other features they wished to see.

All interviews were transcribed verbatim, with a light filtering of repeated words and vocalized pauses. We followed a reflexive thematic analysis approach [9, 10], starting with an inductive pass at coding on the first seven interviews. At this point, we began refining and clustering the codes that we had captured, generating broader themes from the data. Using new codes that represented these themes, we took a deductive coding pass against all interviews to continue building our outline of themes, refining as we saw fit.

Although interruptions are necessary in a notification system, designing interruption that is considerate to one’s mental state should be the default when considering a system meant for health-related activities. However, when considering ambruptive technology, this effort increases in complexity, as the designer must consider the balance between the ambient and disruptive states to maximize notification capabilities without annoying users. While interruptability (i.e., when it is appropriate to interrupt) and JITAIs have been investigated heavily, not much work has explored ambruptive notification, especially in the context of notification for health-related activities. Thus, to learn more about the propriety of disruption, in our study we first investigated the question: When do people accept escalated levels of disruption for health notifications?

The movements and postures of the artifacts do not impart a literal statement. They are abstract shapes, sans language, and must be actively interpreted by the user. In our interviews, we purposely did not explain what each shape-change was supposed to communicate, leaving their definitions open to the participant’s own reactions in order to investigate: How could varying levels of health notifications be portrayed in Shape-Changing Interfaces?

To avoid message fatigue, ambruptive systems should begin at an ambient state, and so we first needed to explore how suitable the artifacts were in conveying an ambient message. The artifacts did well in this area; participants considered shape change types with smaller movement ranges, smoother actions, and reduced movement altogether as suitable for ambient messaging, especially subtle shake. Conversely, for the most part participants felt movements with larger sized actions, such as hiccup with its broader up and down action, gave the best opportunity for a notification to be noticed as an escalated attempt. Participants also liked how the subtle shake movement led into the heavy shake when escalating—seen as an easily understandable progression from ambient to disruptive, as compared to some of the other varied movements that the droop-artifact was capable of. Designing the artifacts to study participants’ reactions around emotion and saliency ended up resulting in deeper discussion around the interpretation of abstract movements in a plant-like object.

As S-CIs, the physical aspect of the artifacts we created led to various discussions on how such devices could work in the lives of our participants. Although S-CIs have existed in literature for decades, they are still an uncommon sight in most homes. However, other devices have recently populated the home in the form of voice assistants, and users are now familiar with interacting with computing without looking at a screen and placing a standalone physical device in one’s environment. Our study provides a current look at participants’ reactions in the post voice-assistant world to answer: How do people envision using Shape-Changing Interfaces within their environments?

One interesting finding was the strong negative reactions to various movements and poses in our artifacts. This ranged from the sadness felt upon seeing the droop-artifact in the downward position, to the unsettled feelings when viewing movements that caused representational dissonance.

These negative reactions show how designers must consider more than mere interpretability in notification systems. Although the poses of the droop-artifact could successfully be interpreted by participants as representations of their state, the emotions and feelings that resulted from this interpretation caused them to reject the use of the system. This reflects how users avoided Fish’n’Steps if their activity led to a sad fish display [45]. While this study extends that work by reporting how some of these feelings stem from the caretaking relationship between humans and plants, but not in a non-living representation, future work could look into when an object’s aesthetic crosses the boundary from an independent object that does not rouse any empathy, into a “thing that is cared for." Learning more here could reduce cases where participants discontinue use of systems because of the adverse emotions that originate from their design.

Furthermore, although others have looked into creepiness and HCI [75], especially in the case of humanoid robots and the infamous uncanny valley, to our knowledge this is the first work that found inherent uncanny effects in a non-zoological representing entity, i.e, eeriness from the object’s own actions rather than from a perceived external force such as Living Room [5]. Future work could explore the precise factors of these artifacts that caused this discomfort, as this study was not focused on exploring the root causes of these emotions or measuring the effects of parameter changes. Nonetheless, using plant-like artifacts to explore uncanny reactions may be useful in future experiments where removing the zoomorphological factors could help isolate certain factors. It is unknown how the reactions we found relate to those stemming from the uncanny valley theory. As the uncanny valley explores the axis of realism, it would be interesting to see if adjusting the realism of the artifacts affected the uncanny reactions. We designed the artifacts with realism in mind, but would the presentation of a more realistic, less realistic, or even cartoonish plant change people’s reactions? Additionally, not all participants found the same revulsion towards the panting movement, and few even reacted favorably towards it. Another avenue of exploration would be into which prior experiences cause people to react positively to artifacts that normally invoke feelings of the uncanny to others. Could increased exposure of something change the response of users who initially feel some revulsion to a movement? Extending on this idea, would it be more worthwhile to learn how to nullify the uncanny valley reaction rather than traverse the steep levels of achieving true realism in robotics?

In addition, the growing interest towards investigating the interplay between humans and plants cannot be ignored, whether plant-mimicking systems as ours or those that manipulate living flora directly. With both Human-Plant Interaction [14] and studies that deal with artificial plants, further research to understand the reactions of users towards systems that use plant motifs will be key in ensuring successful outcomes.

In investigating three avenues of notification—speed and distance parameters, emotion portrayal, and salience, we found that all three had at least one movement type that was described as effective in prompting the participant to act. However, as the previous section discusses, some prompts were coupled with discomfort. As designers, if we solely look at the analytic results, we might miss these negative emotions that arise from the product due to the user still reaching the end result we aimed for. But is this healthy for the user? Are guilt trips, uncanny unease, and morose reactions worth it if the user still acts towards their health goal, or could this be more damaging to them mentally, nullifying any gains from their activity? Especially in the health-oriented sector, we feel this question of what additional mental effects could emerge must continually be reviewed when investigating new technologies and techniques, such as Owens & Cribb’s investigation into personal autonomy with wearable technologies [58].

Long term negative effects of technological aids must also continue to be studied. For instance, literal crutches aid in the prevention of further damage to a user’s leg, but over time may cause shoulder injuries with prolonged use [56]. What are the long-term effects of using technological crutches to aid users in remembering their health activities? Is reliance on technology a gain or loss in the overall health of a user? Again, future work should look into this area to ensure we are net positive in our users’ health.

In the context of RSI breaks, Singh et al. (2021) examined mainly non-disruptive notification after they learned participants’ preferences in the earlier phase of their study. As non-disruptive is only one end of the spectrum our study has explored, their findings cover a subset of our findings, having some related themes around the calmer side of notification but with differing and additional aspects. Table 2 summarizes the overlap describing both similarities and differences. As our study investigated a broader spectrum, our findings include many unique aspects we reported in the result section, i.e., all aspects under the Interruption Etiquette theme, most aspects under the Shape as a Notification theme, and the Reinforcing Locations aspect under the Existing in the Physical World theme.

Reasons for the differences could be as simple as the variance of individuals in each participant pool. However, their study both used co-design and encouraged the selection of items already familiar to their participants—these two personalization-related factors could plausibly have also affected the relationship participants felt towards the S-CIs. It would be worthwhile exploring further whether personalization affects the avataric designation of notification systems—that is, how people cognitively categorize their systems in relation to a representation of self. Even broader, the exploration of how study motif may affect results could help situate future findings and add awareness of result bias.

Our study was an initial look with a limited set of participants who were already familiar with using systems to notify them regarding their health-related goals. Future research could broaden the participant pool and investigate if the scenarios identified as being worthy of an interruption also apply to other health goals or people with no experience in health-goal notification. It can be assumed that many in the world’s population do not use systems for their health-goals notifications or may even have health goals at all. Viewing how users who are new to health-goal notification or tracking and investigating their reactions to ambruptive technology will help the community understand how prior experience in such systems affects one’s reaction.

Additionally, our study participants did not have disabilities, severe illnesses, or age-related disorders, but interviewing those populations or possibly their caretakers would be worthwhile to learn how ambruptive technology could work in their situations. Would users with more serious health concerns have a different position on when notification escalation is warranted? For caretakers, when health-related notifications are not for you, but for someone in your care, how would that change how notification escalation is received? Ambruptive technology could reduce the mental overload of monitoring a patient by the initial use of ambient notification, while providing a safety measure in the notification escalation ability if an ongoing situation reaches a critical state, so these situations are well worth the effort.

Practicality forced some aspects in the design of this study to be fixed, such as the specific shape change types used by the artifacts. However, ambruptive technology is a much broader area than what these artifacts cover. Further work could also look into decoupling or remixing the various design elements and technological areas in these artifacts, and investigating any resultant changes in user reaction. For example, if the S-CI aspect of an artifact was removed, how would users react to a screen-based ambruptive system with plant-like design? Or, how much would abstract sculptural design rather than plants change user reactions?

Also limited were the methods of noticeability that we chose to examine in this study. Because our focus was to capture initial reactions to the concept of ambruptive technology, we chose a concrete subset of disruption strategies that could represent the concept. Future work could look into different means of disruption, whether choosing new movement types, or even implementing modes aimed at other senses, such as light or sound. This work should also examine if these new disruption types would change reactions to ambruptive system.

Our study occurring during a pandemic forced us to hold our interviews via videoconference, which meant participants were reacting to 2D video representations rather than the actual live artifacts. This representation may have altered the reactions from the participants but we do not know of prior work comparing the emotional arousal from 2D video vs. in-person portrayal. However, our opinion is that participants reacting toward this restricted interaction strengthens the findings, and live, in-person interaction would likely show stronger reactions to the artifacts. When safe, future studies in ambruptive technology should include in-person elements to further validate our findings.

Finally, we chose to perform an exploratory study in order to gauge the receptiveness of ambruptive technology while removing the risk of introducing health related issues. Although we were able to explore how people initially react to ambruptive technology, these findings cannot immediately be extrapolated to predict extended usage, and so further research is needed in-the-wild to continue assessment in this space.

Our study uncovered an important factor that would have interfered with an in-situ study: morose and uncanny reactions to the artifacts’ movements prompting participants to end usage. Future studies looking at shape change must include an onboarding process to allow participants to personally confirm that the system’s movements are considered at least neutral, preventing emotional discomfort from use and possible discontinuation.

Next steps should also involve finalizing some technical portions and design decisions to ready the artifacts for use as a research product [54], allowing the concept of ambruptive technology to be tested in-the-wild. This includes adding backend infrastructure to data from sources such as health trackers or a user’s medication schedule, which would allow these systems to run independently of researcher steering. Given participants’ desire for circumstantial notification, this work could be combined with current research on interruption suitability and JITAIs to help with the placement of notifications. Once these factors are in place, the in situ study would be able to continue work in this area, observing how participants act when using ambruptive technology in their daily life. A prolonged study could give insight into how users adapt to the technology over time, whether this alters their health goal success, and the overall suitability of ambruptive technology towards activities that must be done, perhaps not immediately, but eventually.