Research was conducted into what storytelling medium could facilitate the use of a multi-sensory narrative, possibly based on the concepts around Synaesthesia. I found that, despite Virtual Reality (VR) providing a platform that elicits emotion and empathy, and also facilitates a way of controlling the use of senses, this medium may not be suited towards a narrative that creates a full sensory environment. Based on my prior research into Synaesthesia, the vestibular and proprioceptive senses, that often require full user perception within the environment they are placed, require the other five senses as well. Therefore, the conclusion to explore Mixed Reality (MR) environments, remained my focus for the duration of my research.
During this exploration, I found that MR could successfully allow the user to ‘co-exist’ within their existing environment (Jones et al., 2014), and have control over their ‘perceptual apparatus’, which indicates that, because a correlation can be made between virtual items and a physical objects, notions of synaesthesia can be representated through the relationship between a real-life object and the synaesthetic responses that are produced from it. This reflects Dow’s (2008) Embodied Narrative Engagement (ENE) framework, specifically the presence ‘experiential pleasure’, which is said to be a narrative feature that can be manipulated by creators. I will explore how I can utilise this in further research.
The importance of story within MR was also explored. Whilst story is important for any narrative medium, I specifically learned that a story encourages curiosity and reduces apprehension within an environment (Pinhanez et al., 2000), and that, in the case of the ALICE installation, can evoke the sense of immersion without any other stimuli (Nakevska et al.,2017). Further research could include what types of stories and narrative perform the best in MR and how I could implement them.
Finally, my interview with Mark Congilio was an insightful look into a working practitioner who has experience is every stage of the MR experience development process. The importance of co-presence was discussed, and how a MR environment can truly allow for a collaborative, shared experience, which Coniglio expressed as being an important factor. Considerations on physical environment, and the items within that environment, were represented through very interesting examples, such as Seven Tales of Misery, and that this alone is enough for most experiences. My main conclusion from this interview was the importance of a narrative release, or what Coniglio referred to as ‘the payoff’, whereby the experience is concluded using narrative devices, in this case a theatrical performance, where the discovery made by the audience is contextualised.
This research has primarily concluded why I should use MR as the medium to facilitate a multi-sensory, synaesthetic environment. Future research will explore how I can use storytelling and narrative devices, experience design, technology and other physical props to construct an immersive and interactive environment. Specifically, I would like to explore:
- Should my experience be purely experiential and discovery based, or should I have an underlying, pre-determined narrative that acts as an underpin for the experience
- If narrative based, what theoretical narrative structures will I explore?
- Environment is important – which physical space should I explore for my experience?
- What technology and systems are needed to be implemented in my environment?
- Will this be a shared or individual experience, and if shared, how will users react together?
Jones, B., Sodhi, R., Murdock, M., Mehra, R. and Benko, H., 2014. RoomAlive: magical experiences enabled by scalable adaptive projector-camera units. UIST ’14: Proceedings of the 27th annual ACM symposium on User interface software and technology, [online] p.637. Available at: <https://doi.org/10.1145/2642918.2647383> [Accessed 19 November 2021].
Dow, S., 2008. UNDERSTANDING USER ENGAGEMENT IN IMMERSIVE AND INTERACTIVE STORIES. Ph.D. School of Interactive Computing, Georgia Institute of Technology.
Nakevska, M., van der Sanden, A., Funk, M., Hu, J. and Rauterberg, M., 2017. Interactive storytelling in a mixed reality environment: The effects of interactivity on user experiences. Entertainment Computing, [online] 21, pp.97-104. Available at: <http://dx.doi.org/10.1016/j.entcom.2017.01.001> [Accessed 25 November 2022].
Pinhanez, C., Davis, J., Intille, S., Johnson, M., Wilson, A., Bobick, A. and Blumberg, B., 2000. Physically interactive story environments. IBM Systems Journal, [online] 39(3.4), pp.438-455. Available at: <https://doi.org/10.1147/sj.393.0438> [Accessed 25 November 2021].
Coniglio, M., 2021. Interview by James Hooton. Via Zoom (UK and Berlin). 29th November 2021.