Brandon Powers is a creative director and choreographer who is creating experiences across physical and virtual space at the intersection of performance and technology. He was showing a dance performance at ONX Studios during Tribeca Immersive that was titled Kinetic Diffusion. It was created in collaboration with Aaron Santiago, and featured three screens that were being filled with delayed generative AI footage in near real-time and 30 frames per second, which required eleven 4090 GPUs in the cloud to achieve. 



Powers was recording his dance with a mirrorless camera, and then was applying a depth map AI model to extrapolate his embodied movements so that it could be input as a real-time feed into Stable Diffusion with a set of prompts that were precisely timed out. The AI generated images ended up having a 2-8 second delay, which gave the effect of Powers dancing in a duet with himself, but modulated through a series of style transfer prompts. Overall, it was a hypnotically impressive display of generative AI at the intersection of XR and dance. I had a chance to catch up with Powers after his performance to get more context for how it came about, and the long evolution from his previous explorations at the intersections of AI and dance with Frankenstein AI that premiered at Sundance 2019 (see our previous conversation about it in episode #728). You can see a brief explainer video of Kinetic Diffusion within from Powers' TikTok channel.















This is a listener-supported podcast through the Voices of VR Patreon.



Music: Fatality

#1266: Converting Dance into Multi-Channel Generative AI Performance at 30FPS with “Kinetic Diffusion”

Designing for Virtual Reality. Oral history podcast featuring the pioneering artists, storytellers, and technologists driving the resurgence of virtual & augmented reality. Learn about the patterns of immersive storytelling, experiential design, ethical frameworks, & the ultimate potential of XR.

Society & Culture

Personal Journals

History

Philosophy

Places & Travel

Arts

Literature

Performing Arts

Design

Food

Fashion & Beauty

Visual Arts

Technology

Tech News

Podcasting

Gadgets

Software How-To

Curator Ana Brzezińska announced the Tribeca Immersive 2024 line-up on May 23, 2024 mentioning that "this year we are not presenting any VR/AR experiences," but rather that they're pivoting this year to featuring eight immersive art pieces by six different artists being shown at Mērcer Labs: Museum of Art and Technology, which first opened to the New York City public on January 4th, 2024. Tribeca Immersive opens on June 6th and runs for 12 days until June 17th, and you can get tickets to see it here. 



Some immersive creators discovered this surprising news by receiving a refund to their submission, which said "As the immersive landscape evolves, our category is taking new shape to explore formats outside of VR/AR this year and our decision is not a reflection of the quality of your work… Because our change in format was a recent shift and changed our consideration of the works, we will be refunding you your submission fee." 



Tribeca Immersive will be taking over five of the rooms at Mērcer Labs that includes "a two-story high 5-wall LED immersive hall; a 4DSOUND installation space; and two infinity rooms, including one equipped with volumetric technology that makes you feel like you’re watching a space opera." 



This space opera location is called the Dragon Room, which is described by Mērcer Labs on their Instagram page in the following way, "Volumetric lighting allows visitors to see voxels (volumetric pixels) of light in space. The light source is modeled as a transparent object in a container of volume. The resulting effect is of passing through a hologram. The use of mirrors and advanced technology, unlock the power of three-dimensional visualization to create a cascade of luminescent particles that appear suspended in infinite space. The individual light particles in an architectural format offer virtually endless opportunities for formal reinvention. The volumetric installation powered by @ledpulse, Dragon02 technology, features a unique configuration exclusive to Mercer Labs. This remarkable installation, the largest of its kind, and incorporates more than half a million LED neurons-microchips, meticulously synchronized to create an unparalleled visual experience."



I had a chance to sit down with Brzezińska last week to get a bit more context on this pivot as well as on the eight immersive art pieces in the selection that will be spread across three different daily selections repeating four times over the 12-day run. Also be sure to check out her Substack post titled "In The Stillness of Synthetic Light*. Note on Tribeca Immersive 2024" to get even more context for her intentions and motivations for this year's selection where she says, "I believe that before we start seeking hope — which I expect to become a major theme in many upcoming art events — we need space for reflection and respite that allows for a radical suspension of judgment."



I unfortunately won't be able to travel to Tribeca to check it out myself due to a family medical situation that's limiting my travel this year, but I'm looking forward to hearing back from folks in the community to hear how it all comes together. There will not be a formal Storyscapes competition this year, and so I will also be interested to see how the Tribeca Immersive selection continues to evolve over time and whether this represents a permanent shift away from immersive stories towards immersive art, or if we'll see an the inclusion of both forms in the future.











This is a listener-supported podcast through the Voices of VR Patreon.



Music: Fatality

#1392: The Story Behind Tribeca Immersive’s Pivot Towards Immersive Art with Curator Ana Brzezińska

ILM Immersive & Marvel Studios is releasing What If...?: An Immersive Story exclusively on Apple Vision Pro tomorrow, and I had a chance to take an early look and speak with producer and executive producer Shereif Fattouh as well as technical art director Indira Guerrieri about the process of creating this experience. It's mostly an immersive story with light interactions, but you can't actually die and so it's more about immersing yourself into these worlds with first-person embodiment and exploring around 10 different gesture-based interactions in nearly 50 different interactive moments across the 40 to 45-minute experience. There's also some disembodied moments where you're watching 3D movie cinematics on shards of glass floating in the environment, and so they're also really leaning into how good 3D movies look on the Apple Vision Pro within this experience.



Overall, it's blending lots of elements together that feels like something new and a new genre of immersive storytelling that's building upon existing IP, and giving new opportunities to engage with these character's stories as well as their worlds.



This is also the first Unreal Engine experience releasing on the Apple Vision Pro store, and so Fattouh told me that ILM Immersive has been making private modifications to get things working. There are features like mixed reality passthrough that are not available yet in Unreal Engine 5.4 yet. Agile Lens' Alex Coulombe told me, "The Epic Games XR team is small so I would I err on the side of assuming no major Vision Pro updates coming soon from Epic. To have the current features working it still requires a source build of Unreal Engine 5.4 (the Epic Games Launcher doesn’t work) and current features include fully immersive mode, hand tracking / gesture recognition, forward and deferred rendering." So hopefully we'll get more on this soon from Epic Games, but at least we know it's possible and we'll start to see more Unreal Engine experiences on the Apple Vision Pro soon.



Be sure to check out the transcript below for more information, and also don't miss my episode featuring my talk on an Elemental Theory of Presence, Experiential Design, and Immersive Storytelling, as well as my previous 480+ episodes on the topic of immersive storytelling over the past decade including previous coverage on Cycles, Myth: A Frozen Tale, Star Wars: Tales from the Galaxy’s Edge, and a conversation with John Gaeta about ILMxLab's The Holo-Cinema from Sundance New Frontier 2016.











This is a listener-supported podcast through the Voices of VR Patreon.



Music: Fatality

#1391: ILM Immersive’s “What If…?” Marvel Experience Forges a New Genre of Immersive Storytelling blending 3D Movie Cinematics with First-Person Embodiment

The 77th edition of the Cannes Film Festival is featuring an inaugural Immersive Competition with eight pieces in competition along with six pieces outside of competition. I had a chance to speak with Elie Lavasseur, the Head of Immersive Competition Festival de Cannes, to get a bit more context for how this came about, their curatorial intention, and an overview of the program. I've had a chance to see 5 out of the 8 pieces in competition, and all 6 of the non-competition pieces curated by Atlas V in an exhibition titled "exhibition 'exhibition 'Cinema Beyond the Frame: An Introduction to Virtual Reality'. I'll link my Voices of VR podcast coverage down below.




https://www.youtube.com/watch?v=uh7XTZNtzGs




Immersive Competition for Festival de Cannes:




Evolver -- #1104: Tribeca XR: “Evolver” is an Awe-Inspiring Fluid Dynamics Simulation of Human Blood Flow Giving an Embodied Experience of Interbeing



Colored --  #1251: From Book to Play to AR Installation, “Colored” Explores the Forgotten Segregation History of Claudette Colvin



Maya: The Birth Of A Superhero -- #1244: “Maya: The Birth” Animation Uses Mythic Symbols & Magical Realism to Explore Menstrual Taboos as well as #1379: “Maya: The Birth of a Superhero” Evolves Storytelling Grammar with Magical Realism, Dream Logic, & Interactive Embodiment



Traversing The Mist -- #1325 End of Tung-Yen Chou’s Gay Sauna Trilogy with “Traversing the Mist”



Human Violins: Prelude (Multi-user Version)



En Amour



Telos I



The Roaming




Immersive Selection of Non-competitive works:




Notes on Blindness (2016) -- #1068: “Notes on Blindness” VR Experience Receives a Legacy Peabody Award for Interactive Storytelling



Battlescar (2018) -- Unpublished interviews coming soon 



Spheres (2018) -- #617: Journey into a Black Hole with “SPHERES: Songs of Spacetime”



Gloomy Eyes (2019) -- Unpublished interview coming soon 



Missing Pictures: Naomi Kawase (2022) -- #1103: Tribeca XR: Translating Unmade Films into Spatialized VR Story Treatments with “Missing Pictures”



Emperor (2023) -- #1276: Beautiful “Emperor” Explores Aphasia Communication Gaps with Compelling Interactions

#1390: First Edition of Cannes Film Festival Immersive Competition with Elie Lavasseur

One of the most common ways to describe the experience of VR is through the lens of presence, and in this episode I'm going to do a deep dive into the existing academic literature of presence in order to more fully contextualize my approach to it. This is my Storycon 2022 keynote where I elaborate on my elemental approach to presence as well as my thoughts on experiential design and immersive storytelling. I use the four archetypal elements to break down presence into four primary components where fire represents active presence, air represents mental and social presence, water represents emotional presence, and earth represents embodied and environmental presence. And I use the lens of quality, context, character, and story to explore the fundamental components of experiential design.




https://www.youtube.com/watch?v=n_2_qvIymt4




I started to develop my approach to presence through a number of Voices of VR podcast interviews in the fall and winter of 2016, and I was then introduced to the work of Dustin Chertoff, whose presence work from 2008-2010 draws upon the field of experiential marketing. He similarly boils presence down into the components of Active Presence, Cognitive Presence, Relational Presence, Affective Presence, and Sensory Presence. When I interviewed Chertoff in February 2017, we agreed that our approaches to presence were functionally identical. I then went on to present my preliminary ideas on presence at a Silicon Valley Virtual Reality conference keynote in March 2017, and I've continued to develop these ideas over the past 5-7 years.



I gave this Storycon Keynote on May 5, 2022, which happened to be the 8th-year anniversary of the Voices of VR podcast. Now two years later, I'm celebrating my 10-year Voices of VR podcast anniversary in part by airing this talk as well as a few others. I consistently refer folks to it as one of my more fully-formed and rigorous elaborations that I've given of my elemental theory of presence. I not only contextualize it with Chertoff's work, but also with the broader body of academic presence research and it's history. 



I would often find there were would be different presence theories talking about some of the same concepts, but using different terminology. In this talk, I take an archetypal approach that synthesizes these different frameworks through an archetypal lens. At times it can be a bit laborious reciting an archetypal complex of keywords, and there are definitely sections of this talk that are probably better off read than spoken. So I'd highly recommend also checking out the video version as well as the PDF of the slides, and/or the fully-annotated transcript in the shownotes that includes the embedded slides and linked citations. 



That all being said, be warned that this is still probably way too dense for an hour-long talk, as it's more like an entire semester's worth  of material. It's also probably closer to a Ph.D. defense than a synthesized book or practical handbook that's ready for prime time. As with many other aspects of XR, many theoretical aspects are still developing, emerging, boundaries being pushed, and rules being broken. So it's in that spirit that this is my latest fully-formed iteration of these ideas.



This talk also leans more into the theoretical parts, and some of the more practical applications often come within the context of individual interviews where the context is a lot more bounded to a specific situation, experience, or story. Be sure to check out my 30+ hours of coverage from Venice 2023 as an example for how I've put these ideas into practice. 







It's been through many invited lectures and keynotes around the world where I've been able to develop these ideas by engaging with audiences and interviewing thou...

#1389: Elemental Theory of Presence + Primer on Experiential Design& Immersive Storytelling

May of 2024 marks the ten-year anniversary of the Voices of VR podcast, and I wanted to air a featured session that I presented at SXSW 2023 called "The Ultimate Potential of VR: Promises and Perils." This is a 45-minute talk that distills down the biggest insights from my 3-hour episode #1000 of the Voices of VR podcast, which aggregated 120 of the best answers to the question of "What is the ultimate potential of VR, and what it might be able to enable?" This talk synthesizes all of the most exciting potentials as well as scariest perils, and provides an annotated roadmap of my coverage over the past decade on the Voices of VR podcast.



Back in 2016, I synthesized the first 400 answers to the question of the ultimate potential of VR in order to map out the contextual domains of virtual reality within a Silicon Valley Virtual Reality conference talk. I also use the same contextual mapping within my XR Ethics Manifesto talk in 2019, which a summary can be seen in my Landscape of XR Ethics talk that I featured in the previous episode. I provide a lot more background to this archetypal mapping of contexts in a forthcoming paper titled "Privacy Pitfalls of Contextually-Aware AI: Sensemaking Frameworks for Context and XR Data Qualities" that will hopefully be published sometime in 2024.



This Ultimate Potential of VR talk combines both the promises and perils exploring both the benefits and risks of XR through these different contextual domains. I have also added a categories section on my VoicesofVR.com website that uses this same contextual framework as a taxonomy to classify over 1000 episodes. I'm still in the process of tagging my entire backlog, but this talk should provide a good overview to this contextual taxonomy if you want to dive deeper into more episodes across any of the different areas. Again, you can explore these different sections by navigating to the categories section of my website.



I also just uploaded a new video version of this talk with some updated slides and citations. Because I do rely heavily on about 140 slides throughout this talk, then I do highly recommend either watching the video version, listening along with the PDF of slides, or reading through the show notes afterwards where you'll find a full transcript with all 140 slides embedded throughout the transcript as well as links to over 170 of the footnotes and citations with hyperlinks.  The podcast version and write-up will also include the Q&A; session that happened after the talk at SXSW.  



I also wanted to elaborate on one other point about my motivation in putting together this talk. Over the past decade, there's been a persistent cycle where each year there's some journalist or tech pundit who pre-maturely declares the death of virtual reality. Now there have not been very many clear objective metrics to help quantify what's been happening in the broader XR industry to the outside world, which has made it difficult to establish a broad consensus as where the XR industry is at and how it might be evolving. But it's also also fostered an environment where skepticism about XR generally has been able to persist. 



After conducting over 2000 oral history interviews with XR creators over the past decade, it's given me many insights about the underlying affordances of the XR medium across many different contextual domains, which can be difficult to summarize or even communicate. That's what I hope to do in this talk. I wanted to lay out the evidence for why I believe virtual reality and spatial computing more broadly is a compelling enough medium to persist throughout these cycles of skepticism, and eventually find real utility throughout the full spectrum of the human experience. I actually have no idea when virtual reality may become a mass medium that's completely ubiquitous,

#1266: Converting Dance into Multi-Channel Generative AI Performance at 30FPS with “Kinetic Diffusion”

Download our free app to listen on your phone