After suffering a paralyzing viral infection, XR Director Naima Karim started to notice the natural world around her in a more visceral way, She “experienced the monsoon even more intensely than usual: the initial gusts of wind, the rapidly darkening sky, the oppressive humidity and finally the downpour of rain, which washes everything clean and leaves behind a fresh scent.” A painter by trade, she took up virtual reality painting with Open Brush, and created a romantic and poetic sim tribute to Monsoons with a multi-sensory piece called “The Anticipation of Rain.”

I had a chance to briefly catch up with Karim at IDFA DocLab 2022 to hear more about her journey into VR & immersive storytelling. I also wanted to learn a bit more about what’s happening with VR in Saudi Arabia, as she was one of five selected projects amongst a tout of 231 in a competition there.

Missing 10 Hours VR is an interactive immersive narrative designed to counter the bystander effect by recreating different social dynamics and peer pressure as your virtual friend attempts to spike a woman’s drink with a date rape drug GHB and then isolate her by the end of the night. Here’s a description of the piece:

In this interactive VR piece, the viewer is led on a night out by Greg, a big-headed guy with bad intentions. He spikes 22-year-old Mara’s drink with the drug GHB, and she gradually loses control of her actions. A walk through the night ends in a backstreet, where Greg maneuvers Mara into a dark corner. The choices the viewer makes during the evening—go along with Greg, watch passively or intervene—determine the outcome: a safe ending, or arrest and questioning at the police station.

-- Missing 10 Hours VR Synopsis

Sarah Wolozin is the director of the MIT Open Documentary Lab, and I had a chance to catch up with her at the IDFA DocLab 2022 in order to record her journey from being a storytelling to working at MIT’s Open DocLab. Here’s the mission statement of the Open DocLab: “Drawing on MIT’s legacy of media innovation and its deep commitment to open and accessible information, the MIT Open Documentary Lab brings storytellers, technologists, and scholars together to explore new documentary forms with a particular focus on collaborative and interactive storytelling.”

I was able to capture some of the history of MIT’s Open DocLab in my three previous conversations with it’s founder William Uricchio (#855, #1042, & #1160), and Wolozin mentioned the inaugural New Arts of Documentary, one-day summit on March 20, 2012 as being a key turning point for her after seeing the energy and excitement from the community of creators exploring new documentary forms.

Ikhet (Sound Pyramid) is a 14-channel spatial audio installation that included ButtKicker Haptics, LED tubes with diffractive glasses giving a ghostly, analog Holographic effect, experimental field recordings, and a harsh electronic music track that reflected the artist Ali Santana’s strained emotions throughout the pandemic. I had a chance to chat with Santana to unpack his journey and process in creating this piece. Here’s his description of his immersive installation, which details more of the technology and emotional intention for the piece:

This installation inspired by the ‘Glorious Light’ integrates a multi-channel sound collage with audio-reactive LEDs to create an experience that engages viewers via light, spatial sound, and haptic feedback (vibration). The dynamic lighting which changes color and pattern in sync with the experimental sound piece, immerses the audience in a multi-sensory experience that they will see, hear and feel.

Field recordings made between 2019 and 2022 along with experimentations in analog audio synthesis and hardware samplers were combined to create textures, noise, beats, and tones that reflect the artist’s personal feelings of resilience, anxiety, grief, and frustration, during a period of unprecedented change.

Ali Santana’s description of Ikhet (Sound Pyramid)

Bag of Worms is a live XR sketch comedy performance developed by Matt Romein in collaboration with Peter Mills Weiss and Julia Mounsey that showed as a part of the ONX + DocLab MoCap Stage at IDFA DocLab 2022. It’s difficult to fully describe the surreal quality of Bag of Worms, but their synopsis does a pretty good job of summarizing the key points, “Wearing motion capture suits, Bag of Worms uses technology to create a new kind of hybrid performance, with the performers constructing and manipulating a digital playground viewable to the audience in real-time. The lines between the physical and the digital become blurred, challenging the audience’s perception of where the actual performance is taking place.” Because the audience can see the physical bodies of both performers, Bag of Worms plays with “creative violence” in the context of the virtual avatar representations of their motion captured embodiments that are being projected onto the back of the stage. I had a chance to catch up with Romein at the IDFA DocLab to get more of the backstory of how this project came about, how they’re translating some of the language of theatre into rules for this project, and the quality of attention within a theatrical proscenium stage with adding video screens and hybrid performance.

The ONX + DocLab MoCap Stage at IDFA DocLab 2022 featured a number of experimental XR performances using a high-end OptiTrack – Motion Capture System within a theatrical stage with a screen that was projecting the virtual representations. Each of these projects are exploring how to blend aspects of the virtual and physical in the context of a live performance. I had a chance to get the backstory and more details on all of the featured projects from Matthew Niederhauser and John Fitzgerald, who are both from Sensorium & ONX Studio.

I was invited to give a keynote at the Capturing Reality in Motion session as a part of the ONX + DocLab MoCap Stage. I gave a 20-minute talk reflecting on how some of the different projects on the film festival circuit have been using motion capture technology either in the display, production, and live performance of their piece. [Video of this talk is coming soon]. There was also a 5-minute introduction to this session by Sandra Rodriguez (Creative Director & MIT Open DocLab Fellow), but unfortunately it was not recorded.

The second half of this is a panel discussion moderated by Rodriguez and featuring Avinash Changa (We Make VR), Matthew Niederhauser (Sensorium & ONX Studio), and myself.

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

In Pursuit of Repetitive Beats is a landmark immersive documentary that seamlessly integrates so many different immersive storytelling techniques and XR modalities earning it the IDFA DocLab Award for Immersive Non-Fiction. You’re transported back into the 1989 Underground Acid Rave Scene in Coventry and West Midlands where you go on an epic journey retracing the convoluted steps to avoid detection by the police in order to make it to a warehouse of dancing bodies. Director Darren Emerson from East City Films gives us a masterclass of techniques including impeccable set design of filled with media artifacts of that era, hybrid drone shots, projection mapping, 6DoF interactions, Haptic suit design, volumetric capture, wind effects, a well-considered 4m x 6m location-based entertainment design space with streamlined onboarding and offboarding, multi-modal media integration from diegetic archival video footage, fliers, and photos, a climatic psychedelic and vaporwave dance scene, and innovative integration of interviews that projected as silkscreen projections on fliers. Not only were they able to cover everything you’d want and expect within a 2D documentary covering the topic, but this piece managed to recreate the key components of the visceral an embodied experience of going on a musical adventure. All while using the medium of VR to provide additional historical context every step of the way.

The book Collective Wisdom: Co-creating Media for Equity and Justice by the MIT Co-Creation Studio founder Kat Cizek & MIT Open DocLab founder William Uricchio was the named as the recommended book of 2022 by IDFA. They held a launch party during DocLab as well as a couple of different sessions, and I had a chance to catch up with Cizek and Uricchio in Amsterdam on the day of the launch party to get an overview of what they’re covering in their book.

This definition of Co-Creation that they provide in the book gives a great overview of some of the themes they’re covering:

Co-creation offers alternatives to a single-author vision, and involves a constellation of media production methods, frameworks, and feedback systems. In co-creation, projects emerge from a process, and evolve from within communities and with people, rather than for or about them. Cocreation also spans across and beyond disciplines and organizations, and can also involve non-human or beyond human systems. The concept of co-creation reframes the ethics of who creates, how, and why. Our research shows that cocreation interprets the world, and seeks to change

Cizek, K., & Uricchio, W. (2022). Collective wisdom: Co-creating Media for Equity and Justice. page 19. The MIT Press.

Many emerging technologies like virtual and augmented reality naturally lead towards co-creative strategies because it’s such a novel medium that requires a lot of interdisciplinary collaboration. At the beginning of their chapter on co-creation within communities, they point out how the non-fiction and documentary genres are typically early adopters of technology, and how this spirit of innovation often requires a degree of co-creative and collaborative strategies.

With These Hands is a VR installation that “shares challenging, complex and under discussed stories of sexual assault and recovery. It explores the role of listening in the recovery of both survivors and those responsible for sexual harm.” It was produced by Tessa Ratuszynska in collaboration with Survivors Leading Essential Education & Change (SLEEC), which is a “survivor-run organization that changes systems, supports survivors and dismantles the roots of male violence.” The primary mode the narrative is delivered is through voice-over narration where you listen to survivors of sexual violence or other members of SLEEC as they share their own experiences. The piece uses hand tracking technologies so that you can embody the hands of speaker as they share their story, whilst also being in a mostly void VR space aside from a small table-top installation. If the audience member puts their hand on the table, then the narrative will pause, which allows the audience to have more agency in how the VR experience is delivered to them. The first of three segments of the VR installation was based upon this article by SLEEC member Naroa Hammerson, which was published on SLEEC’s Disobedient Survivor blog.

I had a chance to catch up Tessa at the IDFA DocLab in Amsterdam where we talked about this VR installation, as well as some of the unique approaches to consent that she’s using that is modeled after the idea that consent should be ongoing and enthusiastic. This type of co-creative strategy was also featured in the recent book MIT Co-Creation Studio book, Collective Wisdom: Co-Creating Media with Equity and Justice, which I will be covering in the next episode of this series (Voices of VR episode #1160).