The intersection of XR and Artificial Intelligence is a hot topic, and this is the start of a 17-part Voices of VR podcast series where I've interviewed different immersive artists and XR developers over the past four months about how they're integrating AI into their workflow in creating virtual and augmented reality projects. 



I'm starting the series off with Evo Heyning, how has done a deep dive into dozens of generative AI programs, and she wrote an entire book called Promptcraft: Guidebook for Generative Media in Creative Work that explores our relationship to generative media. I had a chance to talk to her about her book at AWE where we talk about her journey into generative AI, her world in virtual worlds and the Open Metaverse Interoperability Group, and her various projects that live at the intersection of immersive media and AI.



Here is a list of the 17 episodes in this series on the intersection of XR and AI:




#1253: XR & AI Series Kickoff with Evo Heyning on a Promptcraft Guide to Generative Media



#1254: Using AI to Upskill Creative Sovereignty with XR Arist Violeta Ayala



#1255: Using GPT Chatbots to Bootstrap Social VR Spaces with “Quantum Bar” Demo



#1256: Using GPT for Conversational Interface for Escape Room VR Game "The Unclaimed Masterpiece"



#1257: Talk on Preliminary Thoughts on AI: History, Ethics, and Conceptual Frames



#1258: Using XR & AI to Reclaim and Preserve Indigenous Languages with Michael Running Wolf



#1259: AWE Panel on the Intersection of AI and the Metaverse



#1260: Using ChatGPT for XR Education and Persistent Virtual Assistant via AR Headsets



#1261: Using ChatGPT for Rapid Prototyping of Tilt Five AR Applications with CEO Jeri Ellsworth



#1262: Using in AI in AR Filters & ChatGPT for Business Planning with Educator Don Allen Stevenson III



#1263: MeetWol AI Agent with Niantic, Overbeast AR App, & Speculative Architecture Essays with Keiichi Matsuda



#1264: Inworld.ai for Dynamic NPC Characters with Knowledge, Memory, & Robust Narrative Controls



#1265: Integrating Generative AI into Live Theatre Performance in WebXR with OnBoardXR



#1266: Converting Dance into Multi-Channel Generative AI Performance at 30FPS with "Kinectic Diffusion"



#1267: Frontiers of XR & AI Integrations with ONX Studio Technical Directors & Sensorium Co-Founders



#1268: Survey of Open Metaverse Technologies & AI Workflows by Adrian Biedrzycki



#1269: Three XR & AI Projects: "Sex, Desire, and Data Show," "Chomsky vs Chomsky," and "Future Rites"












This is a listener-supported podcast through the Voices of VR Patreon.



Music: Fatality

#1253: XR & AI Series Kickoff with Evo Heyning on a Promptcraft Guide to Generative Media

Designing for Virtual Reality. Oral history podcast featuring the pioneering artists, storytellers, and technologists driving the resurgence of virtual & augmented reality. Learn about the patterns of immersive storytelling, experiential design, ethical frameworks, & the ultimate potential of XR.

Society & Culture

Personal Journals

History

Philosophy

Places & Travel

Arts

Literature

Performing Arts

Design

Food

Fashion & Beauty

Visual Arts

Technology

Tech News

Podcasting

Gadgets

Software How-To

Raindance Immersive 2024 opened last weekend in VRChat, and will run for the next four weekends featuring the latest innovations of virtual culture with 77% of experiences happening on social VR platforms and 69% that feature VRChat. I spoke at length with co-curators Mária Rakušanová and Joe Hunting about each of the 36 experiences that span nine different categories with four experiences each including Best Art World, Dance Experience, Game, Live Show, Music Experience, Narrative, Out of Competition experiences as well as the Music Video of VR and Short Film of VR. 




https://www.youtube.com/watch?v=PZ-VmsZ5oAU




There are a number of mixed reality games that will be showing on the Apple Vision Pro and Quest 3, as well as the four narrative experiences that will be showing at a physical exhibition at Raindance, but otherwise the other 30 experiences all have some connection to a social VR platform.



The best live show and best dance experience categories are new this year, and Hunting is continuing to cultivate the two film categories featuring short films and music videos, all of which were shot within VRChat. I talk with both Rakušanová and Hunting about the shift from June to November, and then dive into each of the nine categories and 36 experiences highlighting the latest trends and innovations by indie XR artists who are primarily working on social VR platforms including VRChat, Resonite, and EngageXR.

#1393: Overview of Raindance Immersive 2024 Selection Featuring Social VR Indie Artists & Virtual Culture

Curator Ana Brzezińska announced the Tribeca Immersive 2024 line-up on May 23, 2024 mentioning that "this year we are not presenting any VR/AR experiences," but rather that they're pivoting this year to featuring eight immersive art pieces by six different artists being shown at Mērcer Labs: Museum of Art and Technology, which first opened to the New York City public on January 4th, 2024. Tribeca Immersive opens on June 6th and runs for 12 days until June 17th, and you can get tickets to see it here. 



Some immersive creators discovered this surprising news by receiving a refund to their submission, which said "As the immersive landscape evolves, our category is taking new shape to explore formats outside of VR/AR this year and our decision is not a reflection of the quality of your work… Because our change in format was a recent shift and changed our consideration of the works, we will be refunding you your submission fee." 



Tribeca Immersive will be taking over five of the rooms at Mērcer Labs that includes "a two-story high 5-wall LED immersive hall; a 4DSOUND installation space; and two infinity rooms, including one equipped with volumetric technology that makes you feel like you’re watching a space opera." 



This space opera location is called the Dragon Room, which is described by Mērcer Labs on their Instagram page in the following way, "Volumetric lighting allows visitors to see voxels (volumetric pixels) of light in space. The light source is modeled as a transparent object in a container of volume. The resulting effect is of passing through a hologram. The use of mirrors and advanced technology, unlock the power of three-dimensional visualization to create a cascade of luminescent particles that appear suspended in infinite space. The individual light particles in an architectural format offer virtually endless opportunities for formal reinvention. The volumetric installation powered by @ledpulse, Dragon02 technology, features a unique configuration exclusive to Mercer Labs. This remarkable installation, the largest of its kind, and incorporates more than half a million LED neurons-microchips, meticulously synchronized to create an unparalleled visual experience."



I had a chance to sit down with Brzezińska last week to get a bit more context on this pivot as well as on the eight immersive art pieces in the selection that will be spread across three different daily selections repeating four times over the 12-day run. Also be sure to check out her Substack post titled "In The Stillness of Synthetic Light*. Note on Tribeca Immersive 2024" to get even more context for her intentions and motivations for this year's selection where she says, "I believe that before we start seeking hope — which I expect to become a major theme in many upcoming art events — we need space for reflection and respite that allows for a radical suspension of judgment."



I unfortunately won't be able to travel to Tribeca to check it out myself due to a family medical situation that's limiting my travel this year, but I'm looking forward to hearing back from folks in the community to hear how it all comes together. There will not be a formal Storyscapes competition this year, and so I will also be interested to see how the Tribeca Immersive selection continues to evolve over time and whether this represents a permanent shift away from immersive stories towards immersive art, or if we'll see an the inclusion of both forms in the future.











This is a listener-supported podcast through the Voices of VR Patreon.



Music: Fatality

#1392: The Story Behind Tribeca Immersive’s Pivot Towards Immersive Art with Curator Ana Brzezińska

ILM Immersive & Marvel Studios is releasing What If...?: An Immersive Story exclusively on Apple Vision Pro tomorrow, and I had a chance to take an early look and speak with producer and executive producer Shereif Fattouh as well as technical art director Indira Guerrieri about the process of creating this experience. It's mostly an immersive story with light interactions, but you can't actually die and so it's more about immersing yourself into these worlds with first-person embodiment and exploring around 10 different gesture-based interactions in nearly 50 different interactive moments across the 40 to 45-minute experience. There's also some disembodied moments where you're watching 3D movie cinematics on shards of glass floating in the environment, and so they're also really leaning into how good 3D movies look on the Apple Vision Pro within this experience.



Overall, it's blending lots of elements together that feels like something new and a new genre of immersive storytelling that's building upon existing IP, and giving new opportunities to engage with these character's stories as well as their worlds.



This is also the first Unreal Engine experience releasing on the Apple Vision Pro store, and so Fattouh told me that ILM Immersive has been making private modifications to get things working. There are features like mixed reality passthrough that are not available yet in Unreal Engine 5.4 yet. Agile Lens' Alex Coulombe told me, "The Epic Games XR team is small so I would I err on the side of assuming no major Vision Pro updates coming soon from Epic. To have the current features working it still requires a source build of Unreal Engine 5.4 (the Epic Games Launcher doesn’t work) and current features include fully immersive mode, hand tracking / gesture recognition, forward and deferred rendering." So hopefully we'll get more on this soon from Epic Games, but at least we know it's possible and we'll start to see more Unreal Engine experiences on the Apple Vision Pro soon.



Be sure to check out the transcript below for more information, and also don't miss my episode featuring my talk on an Elemental Theory of Presence, Experiential Design, and Immersive Storytelling, as well as my previous 480+ episodes on the topic of immersive storytelling over the past decade including previous coverage on Cycles, Myth: A Frozen Tale, Star Wars: Tales from the Galaxy’s Edge, and a conversation with John Gaeta about ILMxLab's The Holo-Cinema from Sundance New Frontier 2016.











This is a listener-supported podcast through the Voices of VR Patreon.



Music: Fatality

#1391: ILM Immersive’s “What If…?” Marvel Experience Forges a New Genre of Immersive Storytelling blending 3D Movie Cinematics with First-Person Embodiment

The 77th edition of the Cannes Film Festival is featuring an inaugural Immersive Competition with eight pieces in competition along with six pieces outside of competition. I had a chance to speak with Elie Lavasseur, the Head of Immersive Competition Festival de Cannes, to get a bit more context for how this came about, their curatorial intention, and an overview of the program. I've had a chance to see 5 out of the 8 pieces in competition, and all 6 of the non-competition pieces curated by Atlas V in an exhibition titled "exhibition 'exhibition 'Cinema Beyond the Frame: An Introduction to Virtual Reality'. I'll link my Voices of VR podcast coverage down below.




https://www.youtube.com/watch?v=uh7XTZNtzGs




Immersive Competition for Festival de Cannes:




Evolver -- #1104: Tribeca XR: “Evolver” is an Awe-Inspiring Fluid Dynamics Simulation of Human Blood Flow Giving an Embodied Experience of Interbeing



Colored --  #1251: From Book to Play to AR Installation, “Colored” Explores the Forgotten Segregation History of Claudette Colvin



Maya: The Birth Of A Superhero -- #1244: “Maya: The Birth” Animation Uses Mythic Symbols & Magical Realism to Explore Menstrual Taboos as well as #1379: “Maya: The Birth of a Superhero” Evolves Storytelling Grammar with Magical Realism, Dream Logic, & Interactive Embodiment



Traversing The Mist -- #1325 End of Tung-Yen Chou’s Gay Sauna Trilogy with “Traversing the Mist”



Human Violins: Prelude (Multi-user Version)



En Amour



Telos I



The Roaming




Immersive Selection of Non-competitive works:




Notes on Blindness (2016) -- #1068: “Notes on Blindness” VR Experience Receives a Legacy Peabody Award for Interactive Storytelling



Battlescar (2018) -- Unpublished interviews coming soon 



Spheres (2018) -- #617: Journey into a Black Hole with “SPHERES: Songs of Spacetime”



Gloomy Eyes (2019) -- Unpublished interview coming soon 



Missing Pictures: Naomi Kawase (2022) -- #1103: Tribeca XR: Translating Unmade Films into Spatialized VR Story Treatments with “Missing Pictures”



Emperor (2023) -- #1276: Beautiful “Emperor” Explores Aphasia Communication Gaps with Compelling Interactions

#1390: First Edition of Cannes Film Festival Immersive Competition with Elie Lavasseur

One of the most common ways to describe the experience of VR is through the lens of presence, and in this episode I'm going to do a deep dive into the existing academic literature of presence in order to more fully contextualize my approach to it. This is my Storycon 2022 keynote where I elaborate on my elemental approach to presence as well as my thoughts on experiential design and immersive storytelling. I use the four archetypal elements to break down presence into four primary components where fire represents active presence, air represents mental and social presence, water represents emotional presence, and earth represents embodied and environmental presence. And I use the lens of quality, context, character, and story to explore the fundamental components of experiential design.




https://www.youtube.com/watch?v=n_2_qvIymt4




I started to develop my approach to presence through a number of Voices of VR podcast interviews in the fall and winter of 2016, and I was then introduced to the work of Dustin Chertoff, whose presence work from 2008-2010 draws upon the field of experiential marketing. He similarly boils presence down into the components of Active Presence, Cognitive Presence, Relational Presence, Affective Presence, and Sensory Presence. When I interviewed Chertoff in February 2017, we agreed that our approaches to presence were functionally identical. I then went on to present my preliminary ideas on presence at a Silicon Valley Virtual Reality conference keynote in March 2017, and I've continued to develop these ideas over the past 5-7 years.



I gave this Storycon Keynote on May 5, 2022, which happened to be the 8th-year anniversary of the Voices of VR podcast. Now two years later, I'm celebrating my 10-year Voices of VR podcast anniversary in part by airing this talk as well as a few others. I consistently refer folks to it as one of my more fully-formed and rigorous elaborations that I've given of my elemental theory of presence. I not only contextualize it with Chertoff's work, but also with the broader body of academic presence research and it's history. 



I would often find there were would be different presence theories talking about some of the same concepts, but using different terminology. In this talk, I take an archetypal approach that synthesizes these different frameworks through an archetypal lens. At times it can be a bit laborious reciting an archetypal complex of keywords, and there are definitely sections of this talk that are probably better off read than spoken. So I'd highly recommend also checking out the video version as well as the PDF of the slides, and/or the fully-annotated transcript in the shownotes that includes the embedded slides and linked citations. 



That all being said, be warned that this is still probably way too dense for an hour-long talk, as it's more like an entire semester's worth  of material. It's also probably closer to a Ph.D. defense than a synthesized book or practical handbook that's ready for prime time. As with many other aspects of XR, many theoretical aspects are still developing, emerging, boundaries being pushed, and rules being broken. So it's in that spirit that this is my latest fully-formed iteration of these ideas.



This talk also leans more into the theoretical parts, and some of the more practical applications often come within the context of individual interviews where the context is a lot more bounded to a specific situation, experience, or story. Be sure to check out my 30+ hours of coverage from Venice 2023 as an example for how I've put these ideas into practice. 







It's been through many invited lectures and keynotes around the world where I've been able to develop these ideas by engaging with audiences and interviewing thousands of creators over the past decade. At some point, I'm still hoping to go through a more formal write-up of these ideas to give them even more rigor, and perhaps potentially even go through a more for...

#1253: XR & AI Series Kickoff with Evo Heyning on a Promptcraft Guide to Generative Media

Download our free app to listen on your phone