How immersive technologies have sparked a theatrical revolution
Embracing new technologies and integrating them into performance not only foster a new form of theatre but can enhance traditionally staged work. Giverny Masso explores the kit pushing the boundaries of creativity
Whether projecting a 20ft computer-generated Ariel into The Tempest, conjuring an opera diva from the dead, or allowing the audience to step into a three-dimensional illustrated world, theatre is exploring how a range of pioneering technologies can take the form in new directions.
The industry is experimenting with so-called immersive technologies including: virtual reality, where participants put on a headset to enter a computer-generated world; motion capture, which enables an actor to control a digital avatar through their own movement in real time; and projection mapping, where scenery is projected on to a physical environment and can be changed in the blink of an eye. The biggest theatres in the UK, from the National to the Royal Shakespeare Company, are exploring how best they can use them.
Andrew Chitty, director of government programme Audience of the Future, which seeks to make the UK a world leader in immersive storytelling, says: “Theatre has been using all sorts of technologies for hundreds of years to make things more immersive. Whether it’s lighting effects, developments in sound or stage effects, anything that immerses you in a story is an immersive technology.” Some digital experts believe these new developments in technology have the potential to revolutionise what we see on stage.
The RSC made groundbreaking use of technology in 2016 for its production of The Tempest, starring Simon Russell Beale. It teamed up with technology company Intel and digital group the Imaginarium Studios – which pioneered the use of motion capture in films – to use motion capture to create a CGI Ariel, controlled by actor Mark Quartley.
Sarah Ellis, the RSC’s director of digital development, calls motion capture “21st-century puppetry”. Quartley, she says, wore a suit with 17 sensors to log his movement and facial expressions, which were transferred to special software via Wi-Fi, and created the image of Prospero’s sprite to project on stage. “Audiences that came in through the Shakespeare walked away with the technology, and audiences that came in through the technology walked away with the Shakespeare,” Ellis says.
Not only can technology conjure a non-human character, but it can also be used to resurrect real-life performers from the dead, with holography and projection techniques used to ‘bring back’ artists including rapper Tupac Shakur and Michael Jackson.
Chitty says: “The great interest in holography is with dead artists because it’s the only way you can get them in the theatre, so I think you’re going to see that a lot in the next few years. It’s pre-recorded but often performed with live elements. You might see a famous opera singer performing as if they were in concert mode with a live orchestra, but the opera singer is from a recording from when they were alive and their character is holographically projected.”
That has happened with Callas in Concert – The Hologram Tour, reviving opera diva Maria Callas, which plays the London Coliseum next month. “You’re combining motion simulation and archive recording and holographic projection and sound to make it more compelling,” Chitty says. “Once you create a computer model of the person, then you can do anything.”
‘I felt wonder’
London’s Southbank Centre recently hosted a virtual reality experience with the Philharmonia Orchestra, writes Giverny Masso. I stepped into a seven-metre-square box with five other participants. We sat on rotating stools and put on the rather clunky headsets. Suddenly I was thrust into the middle of the orchestra, which was playing a segment of Beethoven’s Fifth Symphony. As I turned my chair, I could watch the facial expressions of different musicians and focus on the different instruments – due to the 360 degree video and sound.
The experience was created to commemorate the 40th anniversary of the launch of the Voyager 1 spacecraft in 1977, which included within it a recording of the first movement of Beethoven’s Fifth Symphony. As the experience continues, the visuals begin to flicker to fragments of outer space, until you become entirely immersed in space with the music still playing around you. This moment was surreal, and made me feel wonder about the potential for this technology and where it could take us.
After removing the headset and standing up, it took me a minute or so to regain my balance and become reoriented with the real world. It’s performing arts, Jim, but not as we know it.
Other immersive technologies the RSC is exploring include virtual reality, augmented reality, and mixed reality. As part of a newly announced partnership with tech company Magic Leap, fellowships will be given to allow artists to experiment in these fields. Virtual reality involves placing a participant in a world that is entirely computer generated and viewed through a headset. Augmented reality involves layering something artificially created on to the real world, often viewed through a smartphone or tablet screen, with the most successful example outside of theatre being the game Pokemon Go. Mixed reality brings together both a virtual and real environment, merging the boundaries between real and imaginary, and is often viewed through glasses or a headset.
“VR is put on a headset, AR is hold up a device and see a different view of the world, and MR is put on some glasses and the magic happens,” Chitty says. He adds that the “most exciting” theatrical development in VR is taking place in the National Theatre’s Immersive Storytelling Studio, which was set up to explore how such technologies can widen the scope of theatre and enable audiences to stand in other people’s shoes.
Toby Coffey, head of digital development at the NT, describes the studio’s current in-development project, Draw Me Close, in which participants put on a VR headset, take off their shoes, and step into the childhood home of writer and director Jordan Tannahill, which is illustrated around them.
“If you hold your hand in front of you, you see an illustrated hand,” Coffey says. “You hold out your hand, see an illustrated door handle, and as you reach out there’s a physical one there. You open the door to the front of the house and walk in.”
This technique, where the virtual-reality world is mapped to a physical environment, is called room-scale VR. “You can feel the carpet under your feet, then what happens is mum comes home from work and walks through the living-room door, and the reason mum exists in this illustrated world and in your headset is because there’s an actress there with a motion-capture suit on,” Coffey continues.
“Mum does what any mum would do: she comes over and gives you a hug. You feel it, and it’s what I call the ‘holy shit moment’. You see it in people’s body language. They are like: ‘Woah, my God, this is a whole new kind of experience.’”
• Augmented reality: Layering a computer-generated image on to the real world, viewed through a headset or glasses.
• Haptics: Simulation of touch sensation using a computer application, transmitted on to glass – such as on a smartphone – joysticks or modified clothing.
• Holography: Displaying a three-dimensional image of a digital recording, using lighting patterns, which can be viewed without special glasses.
• Immersive technologies: Any form of technology used to draw audiences into a story or to blur the boundaries between the physical and digital world.
• Mixed reality: Brings together both a virtual and real environment, merging the boundaries between real and imaginary, and is often viewed through glasses or a headset.
• Motion capture: The process of digitally tracking a person’s movement via a special suit with sensors. The data is then used to create and animate a digital character.
• Projection mapping: Uses objects of any shape as a display surface for video projection.
• Virtual reality: An entirely computer-generated simulation of a world that can be viewed through a headset.
• Volumetric video: The ability to record something from every angle.
Chitty says the biggest challenge is making VR “anything more than a solo experience” – companies such as Curious Directive have experimented with it – while Coffey admits that this kind of work is not commercially viable, but is about pushing the boundaries of theatre.
“It feels like this is day one of a significantly long journey in terms of seeing how we embrace these technologies and meaningfully integrate them into performance,” Coffey says. “I think it’s huge. At the moment there are two directions of travel in terms of performance. One is that immersive technologies can foster a new form of theatre, a new genre, and the other is that traditionally staged work will be enhanced by the inclusion of them.”
The National Theatre’s Immersive Storytelling Studio also previously experimented with virtual reality in an immersive music video inspired by its 2015 musical Wonder.land. This was a collaboration with 59 Productions, which specialises in another important immersive technology – and which has already been integrated into theatre – projection mapping.
Projection mapping involves projecting an image on to any object or physical environment, turning common objects of any shape into interactive displays. Shows in which 59 Productions has used this technique include City of Glass at the Lyric Hammersmith and Life of Galileo at the Young Vic.
Mark Grimmer, director of 59 Productions, says: “Projection mapping is interesting for writers and directors. Writers will be able to write storylines they might not have conceived as possible before. It can make theatre more cinematic because of how you can jump from location to location.” He added that the technology, however, should not become a distraction from the storytelling.
Other immersive technologies the wider cultural sector is currently experimenting with include haptics – the simulation of touch already used in some mobile phones – 360-degree sound, volumetric video, the ability to record a performance from every angle, and developments in lighting. The key thing, Chitty says, will be how all of these technologies will be combined within theatrical experiences.
He adds that people with stagecraft will be those at the forefront of developing immersive technologies, because “theatre tells a story in space, whereas film tells a story in time” and the virtual or immersive world is space.
Coffey agrees. “VR, for example, is closer to theatre than it is for film in the practice of how you make it. We’re in a space where your audience can completely decide where they look and what the focus is,” he says.
“I hate the word ‘immersive’, it’s overused now and it doesn’t really mean anything,” Grimmer laments. “People are looking for a new vocabulary to describe the way we are all making work. I think [this] is good because it suggests we are doing things that defy description or definition, which I think is really exciting.”