Olfactory Wearables for Mobile Targeted Memory Reactivation

Scent-related information is the only sensory data not relayed by the thalamus into the cerebral cortex. This might be why the sense of smell is so successful at triggering emotions and memories: it is the only sense that is closely connected to the brain’s primary emotion and memory areas of the brain (as shown in Figure 3), [12, 28]. Moreover, this particularity of not being routed through the thalamus is one of the reasons why olfactory stimuli during sleep are unlikely to cause awakenings (unlike other modalities).

Previous work has demonstrated that odor-evoked memories also produce stronger emotional arousal than events triggered by other sensory modalities [29, 30]. Humans can even smell happiness and feelings of fear and disgust through sweat and then experience the same emotions [18, 19]. Hence, a growing body of research in Neuroscience and Psychology has looked into the behavioral and physiological effects of essential oils and fragrances for memory consolidation, anxiety and stress reduction, depression, pain, and improved sleep quality [37, 39, 40, 56]. For example, researchers administered Heliotropin (a vanilla fragrance) to patients undergoing cancer treatment and showed a decrease in anxiety by 63 % compared to a placebo [56]. Others have examined the effects of aromatherapy on users’ mental states: EEG activity, alertness, and mood were assessed before, during, and after math computations. The results of this study suggest that users subjected to lavender scent performed math computations faster and more accurately and reported feeling more relaxed [20].

Scientists have also shown how the sense of smell influences behavior and improves well-being [1, 31, 62]. For example, researchers have demonstrated that scent can influence respiration [5, 25] and improve relaxation and sleep quality [26]. Moreover, scent can also reduce stress and anxiety [41, 48], increase performance, and reduce pain [69].

New memorization and learning techniques have been researched for a very long time in various age groups. For example, multi-sensory interactions have been studied for a small group of seniors with dementia and have provided findings indicative of positive recollection of memories [24]. Multi-sensory technologies have been shown to support embodied and enactive pedagogical approaches [65]. For example, Norooz et al. explored tangible wearable-based learning with children through BodyViz [50] while introducing a new approach to body learning. Other researchers have explored the use of multi-sensory technology for impaired individuals [46], such as MapSense, that uses multi-sensory maps for visually impaired children [10]. While audio-visual based techniques are most common, Covaci et al. discuss how the introduction of olfaction in a multi-sensory educational game improves users’ learning performance, engagement, and experience [16].

Unlike audio and visual, olfactory feedback is relatively challenging owing to the lack of spatio-temporal control over the scent using delivery devices. Despite these challenges, researchers continue to develop scent delivery devices for improved olfactory user experience. Scent has been increasingly used as a less disruptive notification interface to minimize reduction of attention from the primary task. One of the early articles that introduced notifications through olfaction was AROMA [8]. AROMA used stationary scent diffusers with directional funnels to release scent based notifications with two different scents. Recently, it has been used for in-car notification, using a scent delivery device attached behind the steering wheel to release a variety of scents, without compromising focus on driving while improving mood and well-being [22, 23]. Yongsoon Choi et al. also designed “Sound Perfume" to direct scent through glasses to improve communication in face-to-face interpersonal interactions by activating memory using a combination of sound and scent [14, 15]. Other researchers also experimented with scent wearables for improved social interactions [45].

Scent delivery has proven to alter cognitive and emotion-related responses to audio-visual stimuli and odor. For example, a viewing experience is rated more arousing with odors than without [57]. In the area of multi-sensory research, Metatla et al. [47] found evidence of cross-modal and emotional associations between tangible shapes and odor-based sensory inputs. In Augmented and Virtual Reality technology, scent devices can easily be attached to head-mounted displays to provide the users with a more holistic experience. Nakamoto et al. designed a head-mounted display-attached olfactory device made of multiple micro scent dispensers and surface acoustic waves for quick delivery and odor dispersion to study odor’s effects in virtual environments [55]. Yanagida et al. explored the space of head-mounted displays (HMDs) for delivering scents through a projection-based system [68]. Brooks et al. engineered a wearable device attached to a HMD that releases thermal scents directly to the users’ nose. Some head-mounted scent-delivery devices are also now commercially available (Feelreal, Inc.). However, although these HMDs based devices may provide a high-quality sensory experience, XR headsets might still be too heavy and challenging to use for longer durations or in a mobile manner while performing other tasks in a physical space.

Yuxuan et al [42] have extensively explored the scent delivery mechanisms and present O&O, a DIY toolkit for designing olfactory interfaces using scent generators kit and module construction kit. They demonstrate five prototypes using the O&O kit including: Aromatic mask, Scent-notification watch, Scent-enhanced coffee cup-holder, Olfactory augmented VR headset and Desktop olfactory displays. Brooks et al [9] proposed a novel olfactory device that renders readings of external odor sensors as trigeminal sensations using electrical stimulation of the user’s nasal septum. Seah et al [59] presented a chrono-sensory mid-air display system that generates scented bubbles to deliver information to the user. SensaBubble uses different frequencies, sizes and scents to deliver information. A final category of scent delivery devices are jewelry-like or accessory-based [3, 66] and are compact, simple, lightweight, and easy to use. They can subtly trigger bursts of scent in the periphery while performing other cognitively or physically demanding tasks. Therefore, they might be suitable for learning tasks requiring mobility and can be used for longer duration of time and for applications that require use in different spaces (e.g. at home or the office).

Targeted Memory Reactivation (TMR) is a method used for improving memory consolidation which consists of cueing an odor or sound while a person is learning and reactivating the same stimulus during sleep. TMR is essentially classical/Pavlovian conditioning during sleep. This relatively new technique was pioneered in 2007 [54], and has been gaining traction over the years [33, 51, 58].

Previous work has used this technique effectively in laboratory settings using both sounds and scents. For example, researchers used audio stimuli to unlearn implicit bias during sleep [32] and memories have been strengthened by releasing scent paired with the content of 2D card object locations and reactivating the same odor during sleep [38, 54]. Other studies have shown how a foul odor coupled with the smell of a cigarette (presented only for one night of sleep) reduces smoking behavior the following days [4].

Researchers have shown how memories consolidate during sleep and can be experimentally manipulated to be strengthened or weakened using odor or sound cues. If these stimuli are associated with prior learning and presented during certain sleep stages, memory performance will improve by “reactivating" the content learned through association. TMR has been effectively employed in scenarios where the learning tasks do not require the user to move when learning. However, whether Mobile TMR (mTMR for short), benefits of memory consolidation remains unknown. We define Mobile TMR as a Targeted Memory technique that might require the user to move in space such as during a spatial-navigation, cognitive–sensorimotor task. Thus, the learning task is spatial and requires the user to walk/move in a space, which is likely to occur in a naturalistic setting. Due to the challenges of conducting studies in a mobile and naturalistic setting, the physiological effects of olfactory TMR at home (as opposed to a lab-based sleep setting) have not been reported. In summary, state-of-the-art technologies that were tested to provide olfactory feedback for TMR did not allow for mobility. Therefore, most of these studies are conducted with devices that are not mobile or wearable, and the tasks are performed in supervised settings such as sleep laboratories. However, a slowly growing community of researchers is interested in conducting these studies in an unsupervised manner or “in the wild". Recent studies in the fields of Psychology and Neuroscience were successful at demonstrating this concept using audio [27] and odor cues [49] in a stationary, but unsupervised setting. In their study, odor cues were applied via conventional commercially available incense sticks; thus, scent release was not automatized, and instead the tasks were conducted while the user was sitting. However, these researchers successfully demonstrated that TMR is an effective tool that can also be used in unsupervised scenarios while the user sleeps in their home. This supports the primary motivation and contribution of this paper for using TMR in an unsupervised manner to further explore its effects on mobile memory tasks where the user is required to move in a physical space while learning.

Thirty-two healthy subjects with a mean age of 27 ± 6.7 years old (18-43), 14 female, 16 male, one non-binary, and one transgender female were recruited by e-mail and other forms of advertisement. Participants were non-smokers and did not suffer from any respiratory problems, odor allergies, or anosmia. Additionally, they reported not knowing any Spanish. Seven participants reported being native English speakers. Other participants spoke Hindi, Chinese, French, Japanese, Russian, Greek, Korean, Arabic, Portuguese, Ukrainian, Indonesian, German, Finnish, Marathi, Turkish, Estonian, Sinhalese, Dutch and Bangla. Six participants were native Hindi speakers, four Chinese, two French, and one native speaker for every language mentioned above. All participants also spoke English.

The study was conducted at the research laboratory as well as at participants’ homes and was approved by the institutional review board (IRB) at the home institution. All subjects signed a consent form before participating in the experiment and were compensated with a 30$ voucher at the end of the study.

A within-subjects experiment was designed with a balanced order across subjects, with one week in between the two experimental nights. The total duration of the study for each participant was 22 days. Participants met with the experimenters at the laboratory to learn about the study and sign the consent form. They were randomly assigned to one of the following orders: Order 1/A: control on days 1 and 2, odor on days 8 and 9; or Order 2/B: odor on days 1 and 2, control on days 8 and 9 (see Figure 4). Unlike other sleep experiment studies, participants slept at home and there was no adaptation night. They were given instructions to connect to the Empatica E4 and Fitbit devices and set up the olfactory delivery through a smartphone that they kept during the study. At the time of sleep, participants simply went home and slept with the device and physiological sensors the whole night, nothing else required.

The study consisted of two different memory tasks. The first was a 3D memorization task where the user had to walk around a real-world, large lab-based space with objects placed around various locations. The second, a 2D memorization task with cards placed in a two-dimensional grid, was inspired by a previously conducted 2-D card memory task [54]. The memorization tasks had time constraints (5 and 10 minutes), but the tests did not. As depicted in Figure 4, participants would start with a 10-minute walk-around memorization task (for each object location and its Spanish translation) and proceed with the test. Then, they continued with a 5-minute memorization task in-situ for a 2D grid card in Spanish and another test.

Olfactory stimulation was delivered using a mobile olfactometer worn as a necklace. The device was worn during the tasks and the tests at a distance of ∼ 23 cm from the nose. During sleep, the device was placed over the participant’s nose at a distance of ∼ 40 cm following the guidelines of previous work [2]. The essential oil used was commercially sourced [61] and was chosen due to its high pleasantness ratings and minimal trigeminal activation [6, 43]. The ingredients contained lavender, clary sage, and copaiba and were diluted using odorless Isopar H (20% oil, 80% Isopar). We conducted a hedonic test were participants ranked the pleasantness and strength of smell on a 7-point Likert Scale (3.5 is neutral, and 7 is very much). They found the smell pleasant (M = 5.66 ± SD = 0.22) and not strong (M= 3.25 ± SD = 0.19).

The duration of the olfactory stimulus used in the memorization tasks and tests was customized based on the user’s preferences and their olfactory thresholds to ≥ 10ms and ≤ 90ms every ∼ 15 seconds. Thus, each participant had a unique setting during the day. On average, they chose 53ms bursts during the memory tasks, and 55ms burst during the test. At night, the interstimulus interval was 60 seconds for durations of 10ms that were repeated overnight (approximately 8 hours). For the control condition, no olfactory device nor odor was used.

For our study we focused on mainly two types of data as our core contribution: 1) Language & Object Memorization and 2) Subjective Experience. In addition, we also collected physiological information as our exploratory outcome to gain preliminary insights and relationship with memory and sleep.

We collected some of the data through Google forms designed to test memory and request the subject experience with and without the device. While the physiological data points were mainly collected using two commercially available wristbands: Fitbit Inspire HR and Empatica E4. Fitbit was used to collect the data associated with sleep stages and Empatica E4 was used to monitor Electrodermal activity and heart rate activity throughout the study. Additionally, the olfactory wearable was used to track heart rate and respiration in real time from the built-in accelerometer during the memorization tasks and tests.

The results suggest that unsupervised, Mobile Olfactory Targeted Memory Reactivation benefits memory consolidation in a spatial-navigation, cognitive, sensorimotor task and that the effect persists for at least one week.

Most users (21 out of 32) rated the odor as “very pleasant" and additionally used “relaxing,” “calming,” “pleasant,” “nice,” “feeling of being awake” to describe the effect of the scent. Two participants marked the odor pleasantness very low (2 and 3), and were very explicit about disliking the fragrance. They also disagreed on improved focus, relaxation, and sleep quality and were the only ones out of 32 who reported not wanting to use the device again. This highlights the importance of customizing the type of fragrance based on the user’s preferences and evaluating the odor’s strength and pleasantness prior to the tasks.

These findings coincide with the general comments given by participants around the importance of letting users customize their scent frequency; for example, one participant noted: “During the memorization tasks I am not sure if it helped me or not, I was focused on learning and did not notice it too much. Helped a lot during the recall tests, scent was very calming and helped me focus. When I first fixed the frequency and burst duration it was too often and too strong, was a little overwhelming. However I was able to adjust it to my own preference and then it was an enjoyable experience.".

Other participants might have needed more time to customize their preferences, for example one user mentioned that “I believe it was good during the memorization task, when I was just mildly aware of it, but a bit annoying during the recall test. Maybe because of the width of my neck which puts the scent device very close to my nose, or maybe because of the frequency I chose".

Participants felt significantly less stressed while using odor in the memorization 3D tasks than in control (see Figure 9). In the case of the 2D task, there was no significant difference between the two conditions (scent and control). The average stress level in the 2D task was higher than the 3D recall, suggesting that the 2D task was harder (see Figure 9).

Several users found the scent to be helpful during memorization, whereas others claimed that it helped them in the recall tasks. These reports are also reflected in Figure 9, where participants ranked feeling significantly more relaxed with scent than in the control condition P < 0.05, and reported being more focused. Several participants perceived learning with the device to be easier, which also correlates with the findings shown in Figure 10 and the NASA Task Load Index results, depicting a significantly lower perceived temporal demand and workload.

“I found it easier to memorize the words while wearing the device and I felt more relaxed." and “It felt easier to remember the words when I was wearing the scent prototype", and “Scent stimulation with scent device I felt memorizing a bit easier."

Other subjects noted that the device was subtle and unobtrusive, some were not even aware of it: “the smell was not distracting during the tasks, and helped with the relaxation during the tasks" and “I did not notice the scent while doing the test. I am unfamiliar with the scent. It would be more interesting if I am allowed to choose my favorite scent.", some even compared it to body odor or perfume: “I didn’t feel like I was wearing the device at all cause I was too focused on the task, it felt like the smell was my own body odour (perfume)"

Some participants were more aware of the fragrance, but still found it helpful: “It wasn’t very intrusive and I could clearly smell the scent being released.", “The scent kept refreshing my focus to task & test, and gave me better feeling of being awake."

Some users reported on the difference between learning with odor for translation versus object-locations “without scent, took longer to recall. felt like I mixed up letters in spanish word. the english words in both cases were equally easy to remember but the translations without scent were a bit harder to focus on.".

This experiment was conducted with the hypothesis that using a wearable olfactory device during a spatial navigation task and reactivating the same odor throughout an entire night at home could boost memory consolidation. This research sheds light on the potential of wearable olfactometers for experiments that require mobility. Portable olfactometers can bring new opportunities to conduct mobile research at scale and from the comfort of participants’ homes.

In this study, olfactory stimulation using a mix of lavender, clary sage, and copaiba essential oils during a spatial navigation task and overnight sleep induced memory reactivations and boosted the consolidation of hippocampus-dependent declarative memories. Furthermore, participants significantly improved their memory performance during the experimental condition compared to control, and the effect lasted more than one week. The results suggest a significant effect of odor compared to control during the encoding of spatial memory tasks. Moreover, during spatial memory tasks involving the recollection of object locations, Targeted Memory Reactivation using odor significantly improved memory performance compared to the control condition. However, this study does not provide evidence of TMR in translation tasks; instead, it demonstrates that sleep significantly impacted memory performance for this particular task type in the odor condition. It is unclear why; even though odor benefits the encoding during daytime learning, perhaps the effect was only significant after a delay; thus, the results shown are only significant after sleep. Another hypothesis is that the improved sleep quality we observed using odor might have influenced this type of memory more than object location.

The effects of the olfactory device on memory performance for other types of odors or tasks are yet unknown. An earlier study showed that using this device with a water odor vehicle did not cause significant effects on the perceived quality of sleep [2]. Therefore, the study described in this paper aimed to understand if, by using a mobile olfactometer with odor, memory consolidation could be boosted in comparison to not using the device. The memory performance results might differ if the device used for olfactory stimulation were different and varied in form factor, weight, size, comfort, etc. For example, a heavier device worn around the neck might cause discomfort; similarly, a non-controllable wearable might cause habituation. Memory performance might vary if using olfactometers that require nasal masks or nasal cannulas, which are more likely to cause an uncomfortable night of sleep and are less socially acceptable.

Finally, the current physiological and sleep results are limited and need to be further evaluated with a larger number of participants and demographics and standardized sleep staging methods.

To design tools for catalyzing memorization that allows for mobility using olfactory technologies, several independent variables must be taken into account such as 1) the form factor of the olfactory device, 2) individual’s preferences towards odor 3) odor settings (frequency and amount), 4) speed, friction, and interaction between the olfactory device and the environment, 5) cognitive-sensory techniques.

In this section, we describe a set of envisioned scenarios that aimed to provide a foundation for HCI researchers to further explore memorization applications with mobile olfactory technologies for spatial memories. We divide these into three areas: 1) workplace & school 2) tourism, entertainment & outdoors 3) well-being, health & exercise.