Keywords

1 Introduction

Reading is a vital skill to function in today’s society. But why do we read? What are the reasons? In a research, Clark and Rumbold [5] explain the link between reading enjoyment and reading motivation and stated that readers who are intrinsically motivated are more likely to benefit from enjoyment and achieve a deep learning.

Nowadays, we witness to a shift in the way we read and in the materiality of the book brought about by digital technology. E-books are a digital replication of a paper book that in some cases is augmented with functionalities such as dictionary or games. Typically, these augmentations happen in the same medium where the text is. But how can we foster enjoyment and meet the reader’s motivations better without compromising the reading performance? In this paper, we describe a concept that aims to allow the interaction with the reading space just by reading. In this space, digital media infrastructures react to the reading performance and the narrative, promoting enjoyment and immersion.

A significant number of studies have investigated how to digitally augment the book. The MagicBook project [4] explored the potential of augmented reality to enrich the reading-experience. Since then, several academic and commercial books employed augmented reality. Recently, researchers developed the SequenceBook [10], an interactive book with blank pages that serves as medium for a dynamic projection. Another interesting concept explored the experience of a textbook with a fold-out screen-margin extension [6, 7]. The margin space allows complementary content.

A small number of studies have been done in the area of automated environments that reacts to the reading activity. Back et al. (1999) presented The SIT book [2], a prototype, able to use the reader hands’ speed as control parameter for a narrative soundscape. Later in 2005, Bahna and Jacob presented a system [3] in which extra information is peripherally conveyed through a video projection. In 2013, Alam et al. proposed the augmentation of an e-book reading-experience by controlling haptic and audio-visual interfaces existing in a living room [1]. Finally, Schafer et al. developed a system that uses a multimedia room to transform an interactive read-aloud experience, providing feedback that supports children’s enjoyment and meaning-making [8].

2 Prototype

Story reading enrichment (SRE) means to synchronize the reading with one or more digital media experiences. A SRE system must allow (1) the definition of how digital media infrastructures will respond to the reading performance and narrative; (2) the control and perception of a SRE. In order to explore the SRE, we developed a prototype composed by a Tablet and a Smart bulb. The Tablet displays the story, identify the reading position and control the bulb. In order to obtain, the reading position, the system employs speech recognition. Then, depending on the specification of how the light must behave, in a specific reading position, the Tablet controls light’s properties.

3 Scenario

Emma uses a Tablet to read “The Hobbit” [9] to her son Matt. Matt can see on his Tablet a map of “Middle-earth”. He easily realizes where they stopped and by touching the spot, the text appears on both Tablets. The room turns dark, and a cavern soundscape can be heard. When Emma mentions a character, she also associates it to a light (by pointing the lamp). The Light reflects the character’s emotions. While reading, the riddles’ “battle”, the lights of Bilbo and Gollum start to oscillate between white (fear), yellow (joy) and red (angry). At the same time, a melody intensifies the drama. Later, Matt uses the map to jump to a beloved section, where dwarves sing in choir. Matt can then see a lyric and starts to read it. Matt follows a word highlighting feature, which provides the correct pace. When Matt starts to read, a melody starts playing. The longer Matt respects the pace, the more instruments and voices densify the music. By reading correctly he unlocks different music channels, coming from different positions. This game progressively immerses him in a joyful environment.

4 Story Reading Enrichment: Features

Based on the scenario, we identified the four main features of a SRE:

Required attention: Reading is a high attentional activity, demanding constant foreground visual attention. The SRE must be able to display information both to the peripheral attention and to the foreground attention. Peripheral interaction aims to subtly augment the reality with the narrative’s context, e.g. white light to reflect the fear felt by Bilbo. On the other hand, the interaction can obviously encompass a more concrete stimulation e.g. an image of a frightened Bilbo, providing an interactive context that supports the storytelling or meaning-making.

Virtuality: The SRE is characterized by its virtuality and can vary in a continuum that ranges between (a) real environment (b) virtual environment, wherein the intermedium is characterized by a mixed reality experience.

Locus of agency: A SRE can be proactive and/or reactive. In the simplest usage, a SRE shall be proactive, because whenever the reader is focused on reading, an implicit enrichment must happen e.g. melody intensifying the drama. On the other hand, enrichment can be reactive when a reader explicitly requires enrichment e.g. storyteller that uses SRE to complement his performance - make a gestures to set all lights off.

Interactivity: SREs can be controlled and perceived by readers through a multimodal interaction. Regarding the control of a SRE, the reading position and performance can be detected through different input modalities (touch, speech recognition, eye gaze direction, gesture recognition or even emotion recognition). The perception of a SRE can be promoted by different modalities such as visual, auditory, haptic, etc.

5 Story Reading Enrichment: Roles

Based on the SRE’s features, we envision roles with a focus on multiple aspects of the intrinsic motivation [5] such as curiosity, involvement or social interaction.

Meaning-making: What if the reader does not know a specific object or idea? In order to foster the curiosity and to avoid losing involvement, the SRE can function as a facilitator of meaning-making e.g. when several characters are introduced in a short amount of text, reader may benefit from additional digital media. This kind of SRE can be explicitly initiated by the reader or implicitly through concentration sensing.

Intensification/clarification: Another way to promote involvement, is by intensifying the reading. A scene can be intensified through music targeting an emotional state or a fictional genre. The SRE can also serve as a clarification to interpret text. A good example is, when a figure of speech like irony or metaphor is used.

Support: In another perspective the SRE can support the reading performance. This functionality is characterized by a playful manner of using a SRE, targeted to extend the reader’s expressivity e.g. a storyteller controlling the SRE through gestures or speech. This supportive role shall promote social activities such as storytelling.

Training: The SRE as a playable environment can be used in a gamified experience e.g. rewarding by mastering reading and interpretation skills. Another example is to encourage a focused reading, by providing a SRE based on neurofeedback technic.

Topography: The SRE can also be used to simplify the text navigation. We believe that, if the reader can use a simple metaphor to map and mark the written content, we can promote a feeling of being in control of the reading. An example is the creation of a virtual map where the reading-experiences or digital marginalia are mapped.

Inspiration: The SRE can also spark the imagination. When an author deliberately describes a scene, in an ambiguous way, he is challenging reader’s imagination. SRE can have the mission of supporting the imagination challenge. For example, by allowing readers to decide in which kind of virtual environment they want to read (forest, city, etc.) or by synchronizing the SRE with the narrative and reading performance.

Cumulative reading: The SRE must be able to learn from each reading session and enhance the reader’s capacity to recall/relate thoughts and emotions from previous readings and adapt SRE’s behaviours based on reader’s preferences.

6 Conclusion

In this paper, we explore a concept that employs digital media infrastructures, that react to the reading performance and the narrative, promoting a more joyful and immersive reading-experience. Based on the user experience envisioned through a key scenario, we analyse the principal features of a SRE and discuss how SRE can assume a variety of roles, promoting experiences that aim to match readers’ intrinsic motivations. This paper highlights the potential and open questions of this concept. It also serves as a conceptual framework for developing SREs. Further work will involve building a functional SRE system that allows the proof-of-concepts development. Furthermore, evaluations will be performed to understand how to effectively enrich the story reading and to identify story genres’ constraints and side effects.