Physical mixed reality environments are, relative to other emerging technologies, not a modern concept. Advancements in technology and the interests of society, have progressed mixed reality environments into the high-tech, immersive and engaging experiences we have today. This post explores examples, dating back to the 90s, right up until today, discussing what makes them so compelling in their narrative and experience design.
KidsRoom (Pinhanez et al., 2000), developed by MIT Media Lab in 1996 is an early example that used ‘unencumbering, rich sensor technology’ to ‘facilitate user immersion in the experience’ as the story progressed (p.438, ibid). The experience is a fantasy story based on interactions with computer generated monsters that facilitates the use of the physical environment for children to interact with. This includes hiding behind the bed, talking to furniture and even using the bed as a boat, where sensors are used to determine the direction in which the children are rowing. The experience ends by following some simple dance steps that the monsters mimic. It was highlighted that ‘there has never been a situation where the children did not understand that they are characters in a story and that they have to act out their parts to make the story flow’ (p.442, ibid). This experience lends itself naturally to children, who would often ordinarily use objects within their own bedrooms for the exact same activities as they were instructed to within KidsRoom. The difference being that they no longer need to imagine the imagery and the narrative, as it is brought to them. This demonstrates the importance of having a well-constructed story that the whole experience is centred around, and specifically in this example, exploits the interests of the audience;
‘The existence of a story seems to make people, and especially children, more likely to cooperate with the room than resist it and test its limits. The well-crafted story also seems to make participants more likely to suspend disbelief and be more curious and less apprehensive about what will happen next. In fact, the users of KidsRoom have absolutely no control of the overall story development and do not seem concerned at all about that.’ (ibid)
Another example of a environment that uses a narrative to facilitate an experience is the ALICE installation (Hu, et al., 2008), which is based on the ‘Alice in Wonderland’ books, whereby the user ‘takes the role of the character Alice and experiences the sequence of emotional and behavioural states as Alice did in the narrative’ (Nakevska et al., 2017). This experience takes a different approach to the previous example, as it has six consecutive experiences, which take the user through an exploration and discovery. This is suggested to be an effective method of storytelling within a physical mixed reality environment as ‘[a] particular sequence of events has a significant positive influence on the arousal and curiosity and on triggering and guiding intended user behaviour’ (p.99, Aart et. al cited in ibid). Similarly to the previous example of KidsRoom, the ALICE installation used props as a storytelling method within the environment, specifically in the ‘Eat me, drink me’ stage, whereby a bottle and a cookie box where placed within the CAVE environment. The box was fixed with an IR sensor that detected movement and the bottle was fitted with a tilt sensor (p.99, Nakevska et al., 2017). Each object triggers the CAVE system to scale up or down, simulating the growing and shrinking of the user. The specific use of the CAVE system here is very interesting, as it allows the use of physical props and environments, yet still facilitates a fully immersive virtual environment, much like VR does. This is a method of storytelling that is proven to be effective as Nakevska et al. (2017) concluded that after testing the CAVE as an ‘interactive environment (IE)’ (as described already), a ‘non-interactive, but dynamic setting (NIE)’ (the room still grows and shrinks, but not based on interaction) and a ‘non interactive setting with minimum stimuli’, it was concluded that the ‘immersiveness’ of the environment was not impacted by the modality of the stimuli, but from the ‘time density’ spent within the environment itself (p.103, ibid). From an individual evaluation, having as few mediums of stimuli as possible is likely to be a successful method of storytelling, as users are already aware of the story the experience is based upon – having read Alice in Wonderland – and the spectacle of the environment is often enough of a stimuli to immerse the user.
In contrast to a pre-defined narrative approach to a mixed reality experience (as explored in the above examples), teamLab, an international art collective, creates a variety of experiences that primarily utilise exploration and admiration. Examples such as LIFE (teamLab, 2020), exhibited in Seoul, South Korea, do have underlying themes and narratives of nature and living; ‘nature and civilisation are always connected […] we want to affirm that we are alive regardless of the situation’ (ibid), yet this message is visually mediated to the audience through immersive and interactive projected content. Users aren’t encouraged, but naturally lead through the experience and discover interaction through exploration. Flowers grow around the audience, yet are destroyed upon standing on them. Water particles follow the users path and artwork is generated dynamically. What is significant about the experiences that teamLab creates, is the role they provide for the user within the experience, giving them agency and importance. By itself, it appears as a mesmerising spectacle yet every interaction controls the artwork. TeamLab demonstrates how an engaging experience can be generated using the physical presence of the user within that space, requiring very minimal thought from the user, thus providing them with the opportunity to think about the underlying messages of connection with life.
Nakevska, M., van der Sanden, A., Funk, M., Hu, J. and Rauterberg, M., 2017. Interactive storytelling in a mixed reality environment: The effects of interactivity on user experiences. Entertainment Computing, [online] 21, pp.97-104. Available at: <http://dx.doi.org/10.1016/j.entcom.2017.01.001> [Accessed 25 November 2022].
Pinhanez, C., Davis, J., Intille, S., Johnson, M., Wilson, A., Bobick, A. and Blumberg, B., 2000. Physically interactive story environments. IBM Systems Journal, [online] 39(3.4), pp.438-455. Available at: <https://doi.org/10.1147/sj.393.0438> [Accessed 25 November 2021].
Hu, J., Bartneck, C., Salem, B. and Rauterberg, M., 2008. ALICE’s adventures in cultural computing. International Journal of Arts and Technology, 1(1), p.102.
teamLab. 2020. teamLab: LIFE. [online] Available at: <https://www.teamlab.art/e/ddp/> [Accessed 29 November 2021].