Immersive Storytelling

Discussing Immersive Mixed Media With Isadora’s Mark Coniglio


As part of my research into immersive environments and installation pieces, and my current exploration into Troikatronix’s Isadora, I interviewed Mark Coniglio, the creator of Isadora and the founder of Troikatronix. From this interview, I gained further insights into what makes a successful immersive narrative within a physical space, how technology should (and often shouldn’t) be used to assist the narrative, and Coniglio’s aspirations for the use of Isadora. Coniglio also provided thoughts on the progressive use of VR in mixed reality experiences, and where this medium fits in with installation and physical mixed media experiences.

Coniglio’s favourite immersive experience

Having personally had limited experience in visiting immersive media installations, it is difficult to get a first-hand understanding of the types of narratives and stories that perform well in this environment, especially from an emotional and cognitive understanding. I asked Coniglio what is the most engaging installation and mixed media experience he has been to/been involved with. He concluded that a show named ‘Seven Tales of Misery’ by Signa, exhibited in an old building in Copenhagen in 2006 was his most prominent. Coniglio described the space;

“I don’t know who the set designer was, but whoever this person was, was an absolute genius because it was a huge space – 3 floors of a building – and every little detail [was there]. There was all this stuff on the walls [and] I could’ve, without any performances or anything, just been left to wander in that space and felt like I was getting told a story. So, I think that the environment which you are in, if there is a lot of richness in that and it has its own story to tell, obviously as long as its in line with the theatre piece you are showing. I think that’s very important”.

Seven Tales of Misery Experience (Signa, 2006). Photo by Erich Goldmann

Coniglio explained to me that despite the environment being so immersive, the 200 audience members who were in the building started to leave after 4 hours of wandering around, as they thought they had been through the entire experience, and only 30-40 people remained. However, the characters in the performance began to gather the audience into the prince’s room and “then they gave you 25 minutes of absolute scripted, super tight, very intense, incredibly well-acted theatre [and] all of the discovery that you had made came together, and you actually understood what was going on. It was really an intense thing”.

One of the immersive experiences I have visited is the Van Gough Alive exhibition. Whilst visually mesmerising, the discoverability aspect was personally quite disappointing. It may be because I was not really interested in Van Gough and spent a lot of the time looking at the make and model of the projectors, but I felt like there was nothing that triggered inquisitive thoughts, and after a while it became very predictable, repetitive and honestly … boring.

 

Audio tape in Seven Tales of Misery (Signa) that, as Coniglio described, ran throughout the entire building, playing the soundtrack for the experience. Photo by Erich Goldmann

After Coniglio left the Copenhagen experience, he described what made it really outstanding for him. There were two of the performers stood on the street, watching the audience members leave the building and talking about the experience. Coniglio concluded that “because of this, I feel like it never ended […] so they created a place where you could discover and use your own curiosity, then if you have the patience to really stick with the piece, they gave you this incredible payoff, where 170 people did not know this had happened. There was also a feeling of ‘wow, I stuck with this long piece and I get something really big because of it'”.

It’s this payoff that was really lacking in the Van Gough experience that made it very disappointing. The videos ran on loop, and at times I was asking “haven’t we seen this bit before?”. Much like Coniglio’s experience, I noticed people who arrived after me leaving before me. They must’ve been bored too. And what did I get as a reward for accidentally watching it one and half times through? Nothing. No payoff. Well, apart from being offered an over-priced Van Gough keyring.

Van Gough Alive Experience, MediaCityUK

Technology and its place in immersive experiences

The Covid-19 pandemic has accelerated the use of technology being used to create shared, collaborative and interactive experiences online. Coniglio highlighted some of these advancements in the online theatre piece ‘Russian Troll Farm’, which was live-produced in Isadora. Coniglio suggested that by using Isadora as a platform to create this experience, with remote actors being aware that they are live, “changes the energy with which they perform […] that sense of live-ness that I think is one of the reasons we go to theatre, was present in this piece”.

However, what is disappointing for Coniglio is the lack of co-presence –  how do we make the audience aware that they are sharing the experience with other people, and make the performers also aware that they are being spectated? He explained that “there’s a certain feeling when the lights go down in the theatre and you know you’re with 200 other people, and you’re there to focus on this work, that’s special and it’s very hard to do online”. This was an area on which he felt the previous example of ‘Russian Troll Farm’ fell down, as there was no spectator to performer feedback. Coniglio explained that it’s difficult to try and replicate this online, but a successful experience must make you feel like you have made contact with someone. He spoke about a particular online experience he had where he did begin to feel this connection:

Coniglio expressed that now we can have avatars, profile pictures, characters, emojis, and eventually easily accessible motion capture cameras that will allow you to be “anyone you want to be, dressed as anyone you want to be”. This physical connection through a virtual space is something that Coniglio highlighted as being a vital part in the development of these platforms:

“There’s almost nothing that is a certainty in this world, but the certainty I feel like I can go with is that, human beings want to connect with other human beings. That’s almost universally true and if you can find a method that provides honest, authentic, direct, intimate and personal communication [and] if you can create that environment, then I think you are really onto something and that’s when these things will start working”.

This is really interesting to me, and perhaps explains why companies involved in extended reality development, such as Meta, are becoming fixated on this desire for humans to communicate through their mediums. VR has always been a personal experience, and it becomes very appealing when an offer is made to make virtual reality more real. Furthermore, Coniglio spoke about the advantages of remote performance and interactions; it’s safer, especially in the virus-ridden world we are living in. Of course, he also highlighted that whilst these platforms aren’t susceptible to pandemics, they are breeding-grounds for malicious behaviour. An area that Coniglio felt virtual reality performs at its best is when the “power of these of virtual environments [are used to] create a world that could not have existed in the real world”.

Coniglio concluded that VR could and should be used as a way of “still feeling a sense of community and coming together” whilst not being physically together, but it is up to artists to create this sense of co-presence of everyone involved in the experience.

The future of Isadora

Finally, I spoke with Coniglio about his aspirations for how Isadora can be used further in immersive pieces. He explained that he wants to use Isadora to bring audiences together. He further explained that there is a project he is currently working on, that despite currently being ‘under-wraps’ and undisclosed, aims to tackle this area;

“Is there a way to know how audiences are reacting to what they are seeing, and can that information be shared with the performers, and can  that can be something that influences the way [they perform?] We need to try and close that loop between the audience and the performer. We’ll be using Isadora to attempt some things in this realm”.

Coniglio further emphasised the importance of co-presence when creating work that is experienced by multiple people, and provided an example of a simple Isadora patch that simply changed the position of the normal rectangular Zoom grid, into a sphere, that rotated the viewer into view. This spherical positioning of the participants entirely changed the way how Coniglio communicated with them:

Final words of advice and conclusions

I briefly discussed with Coniglio the best practices of generating thoughts, stories and narratives when creating an immersive media piece with Isadora. He concluded that his recent developments are constrained by time and money, and the development methods of these pieces hugely reflect that. However, we did discuss the process of using Isadora as a storyboarding device as well as a development device, which I have personally found to be my favourite method of using the platform. Coniglio’s final advice around this topic was to “let yourself respond in the moment to what you can see – if you can build a piece this way, it’s luxurious”. However, when bringing media, graphics, technology and assets in, “it needs to be there for reason, that takes us to a place you can’t achieve any other way. Otherwise, it seems like an add-on”.

My conversation with Mark Coniglio was extremely insightful. Firstly, it was a pleasure to talk to someone with such expertise and insight in this field, who just happens to create a piece of software that I keep returning to time and time again, due to it’s seemingly endless options. Three words that really stick with me from this conversation are ‘discoverability’, ‘co-presence’ and ‘payoff’. When we speak about creating an ‘experience’, I personally have little thought in what it actually means. The discoverability that Coniglio spoke about in ‘Seven Tales of Misery’ is a really interesting notion that is evidently enough of an experience for most audiences, that they left before the show had finished. However, the pay-off that the remaining audience received was something special, and when creating my next immersive piece, I really want to explore how I can create a narrative that allows both of these things. In addition, the ideas around ‘co-presence’ are really interesting to me. Coniglio explained that, as humans we thrive when interacting with other humans. Therefore, I will use this next time I develop an experience as a storytelling device, or in some cases, it could be used as an isolation method, where you are made to feel alone. I can see this working successfully in the medium of Virtual Reality.

References

Coniglio, M., 2021. Interview by James Hooton. Via Zoom (UK and Berlin). 29th November 2021.