SPAITIAL

Podcast Recorded on February 2nd, 2024 (Groundhog Day)

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From Violet:

Apple’s Vision Pro is a technological marvel and marks a significant moment for our industry, bringing mixed and augmented reality into many more people’s hands and enabling developers to build high-fidelity apps that turn our world into an interactive computing surface.

It’s like living in the future – experiencing today what we’ll achieve with less cumbersome devices in the near future.

That’s what makes it important and amazing.

https://nianticlabs.com/news/living-in-the-future

From Mirek:

If you were wondering how Apple plans to mass market its upcoming Vision Pro mixed reality headset, leading Apple supply chain analyst Ming-Chi Kuo reports that the company is actually looking to make the headset a pretty scarce item from the get-go.

“Apple will produce 60,000 to 80,000 units of Vision Pro for the February 2 release,” Kuo  writes in a recent X post. “Since the shipment is not large, I believe that Vision Pro will sell out soon after the release.”

Provided Kuo’s analysis is true, this would make for one of Apple’s most ‘artisanal’ products at launch. Granted, at that price it’s likely only hardcore Apple acolytes and would-be software developers will jump on board, the latter of which no doubt will be looking to create apps for what hopes to be a new ongoing hardware platform for the company, and not just a ‘flash in the pan’ experiment.

https://www.roadtovr.com/report-apple-vision-pro-below-100k

From AB:

Sure, the Vision Pro hardware is impressive – take a look at just how far ahead of the pack Apple is compared to the complete history of VR/AR Headsets here on our new Timeline – but Fabian shows how it’s actually Apple’s entire ecosystem that is there superpower.

If every product in Apple’s lineup has the potential to merge into a user’s spatial experience, then what does this bode for the future of work, the future of entertainment? Watching Alicia Keys as if you were up on stage with her is only the beginning…

https://fabiankreuzer.de/en/visionos

From William:

NOT Vision Pro, specifically, but noting an adjacent major announcement about Google’s ImageGen 2 model, that’s been packaged into a product called “ImageFX” plus the SynthID watermarking tech.

https://blog.google/technology/ai/google-labs-imagefx-textfx-generative-ai/

To absent friends.

The post Episode 005: Apple Vision Pro – love it or hate it – it’s here appeared first on SPAITIAL.

Recorded on January 26th, 2024 (Australia Day!)

From William:

Legendary Disney Imagineer Lanny Smoot shows off his latest invention: a tiled floor called the Holotile, that used small rotating pucks to keep a human centered within the floor, rotating the tiles to align the resulting movement vector. The demo shows Lanny makin gfree movements, and the Holotile counters his steps to return him to a central location – then it shows two people using the Holotile at the same time – breathtaking thinking.

We are looking forward to when the Holotile can be miniaturised sonewhat, so that the user has more support-per-inch, if that’s even a unit of measure. In this way, the experience would be even more natural and fluid.

https://www.youtube.com/watch?v=68YMEmaF0rs&ab_channel=DisneyParks

From Violet:

“As we document our lives more than ever before, smartphones may paradoxically diminish our ability to recall these moments. Modem collaborated with industrial designer Betuel Benitez and design researcher Kotomi Tanaka from UC Berkeley to explore how spatial technologies could revolutionise our approach to memory retention and recall. Memory Machines envisions a speculative device designed to enhance our memory through the recreation of immersive 3D scenes.”

https://modemworks.com/research/memory-machines/

From AB:

Well dang – only a few days after our previous Episode on Spatial LLMs and semantic understanding of place & space, and Google/MIT/Stanford release a model on exactly that – the ability to understand a 2D image or video, and break it up into segmented, annotated features, based on depth and volumetrics.

From this amazing data pipeline, the last step seems… trivial? Being able to ask meaningful questions about the scene, including deep detailed questions about estimates of distance, height, and volumes.

Bonus points for being the “first Internet-scale 3D spatial reasoning dataset in metric space“. Freedom units need not apply. Research paper and project page, below – but no code released. Watch this space for fast-followers in the open-source world, no doubt.

https://spatial-vlm.github.io/

From Helena:

Richard van der Blom has been busy running experiments on LinkedIn’s main feed algorithm, and has turned up a series of ‘quiet metrics’ that seem to amplify whether of not your post gets put high in the endless scrolling of work-related news and job anniversaries. Richard’s contention is that dwelling over a post, clicking the ‘see more’, and opening the comments panel all contribute significantly to the overall post ‘juice’ – moreso if those eyeballs are on the fringe of your connections than your superfans.

https://www.linkedin.com/feed/update/urn:li:activity:7155077483376091138/

From Mirek:

Google has been busy this week, releasing an ‘Olympiad-level’ AI model that understands maths problems involving geometry. Unlike the above SpatialVLM model, this one has project page AND fully working code release, so AlphaGeometry is more firmly aligned to acamedic research than commercial gain.

That said, the ability to interpret geometric math problems, and construct valid solutions/working proofs, makes this model (and those that will follow it), the obvious next step in the progression from Wolfram Alpha’s query by text, to a new world of visual-inputs and complete sementic interpretation therein.

https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

To absent friends.

The post Episode 004: Spatial UX appeared first on SPAITIAL.

Podcast Recorded on January 19th, 2024 (AEST)

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From William:

From the Rabbit Research Team:

“We have developed a system that can infer and model human actions on computer applications, perform the actions reliably and quickly, and is well-suited for deployment in various AI assistants and operating systems. Our system is called the Large Action Model (LAM). Enabled by recent advances in neuro-symbolic programming, the LAM allows for the direct modeling of the structure of various applications and user actions performed on them without a transitory representation, such as text. The LAM system achieves results competitive with state-of-the-art approaches in terms of accuracy, interpretability, and speed. Engineering the LAM architecture involves overcoming both research challenges and engineering complexities, from real-time communication to virtual network computing technologies. We hope that our efforts could help shape the next generation of natural-language-driven consumer experiences.”

https://www.rabbit.tech/

From Helena:

https://www.opus.pro/

From Mirek:

The generative AI revolution embodied in tools like ChatGPT, Midjourney, and many others is at its core based on a simple formula: Take a very large neural network, train it on a huge dataset scraped from the Web, and then use it to fulfill a broad range of user requests. Large language models (LLMs) can answer questions, write code, and spout poetry, while image-generating systems can create convincing cave paintings or contemporary art.

https://spectrum.ieee.org/global-robotic-brain

From AB:

My Fast Five for this week is https://mingukkang.github.io/GigaGAN/ – but not the fact that this is a GAN, making any generative artwork you can prompt it with – no – I’m more interesting the insane super-resolution that its doing to the initial output: going from 128×128 pixel GAN output, all the way up to a 4000×4000 pixel image. About 1000 times the input.

So while it’s very obviously ‘making stuff up’, the question for us all is: is it making stuff up, semantically ie: with a level of understanding of objects/depth/relationships – or is it just superbly trained to go from small–>huge images, until the meat sacks in the loop (us!), think that its simply magic…?

https://mingukkang.github.io/GigaGAN/

To absent friends.

The post Episode 003: Spatial LLMs – the semantics of place & space appeared first on SPAITIAL.

Podcast Recorded on January 12th, 2024 (Weather-dependent)

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

We’re down by one William this week – we trust he is in bed, resting – and dreaming of electric sheep.

From Violet:

Is it a projector on wheels? Is it a battery-powered tennis ball? What can it interact with – and can dogs see projected images, anyway? Only time will tell…
https://www.designboom.com/technology/samsung-ballie-ai-robot-ces-2024-01-09-2024/

From Helena:

Direct from CES, the Consumer Electronics Show in Las Vegas, Volkswagon is announcing a new partnership with OpenAI to put ChatGPT into their new car interfaces – starting as soon as Q2 2024.

What are the potential use cases? Tour guides en route / at destination – or (yikes!) – replacing podcasts with location-specific historic stories along your way?

https://www.volkswagen-newsroom.com/en/press-releases/world-premiere-at-ces-volkswagen-integrates-chatgpt-into-its-vehicles-18048

From Mirek:

“The wondrous truth of wearable XR is that it’s NOT about visuals. In all my years of XR work, the best examples had only one thing in common: absolutely minimal pixels.

Its not about screens everywhere, that is what’s known as a transition marketing term used to get folks to see the immediate understandable use. Its low hanging fruit in the most meanings possible.

XR tech is actually about human native inputs. It’s being able to sample headpose and eye vectors and hands DIRECTLY, in a low latency manner while the user is in flow. Not through an abstract interaction or a drop down interface.

This means you can lower the amount of visual feedback necessary in the past because of the terrible mismatch to humanity.

There’s a reason Apple called their device Vision Pro and not Visuals Pro.

You will know folks have finally found their XR sea legs when you use an application and you realize it couldn’t possibly be done with a mouse and keyboard. Or a pile of shortcuts. Or a controller.

XR isn’t dead, it’s barely fetal. It’s worming its way into the brains of the creators out there. True change takes time, most especially in the actual experts that need a while to work their way around all the decades of experience that solidified the glory of screen optimized experiences.

Its coming. Open your head, eyes, and hands…and try to see new ways using old tools.”

https://www.linkedin.com/feed/update/urn:li:activity:7149741318913630209/

From AB:

Luma’s Genie – for high-fidelity/high-polygon count 3D objects from your text input:

https://techcrunch.com/2024/01/09/luma-raises-43m-to-build-ai-that-crafts-3d-models/

Text2immersion – using Gaussian Splats (GaSPs) to make a… 2.9D? not-quite-3D scene that the user’s camera sits firmly in the middle of. Movement is limited to the early NeRF/GaSP small movements around a centra lpoint, so there’s no ‘exploration’ to speka of just yet. This would be brilliant for a new genre of text-based games, though: Go North. Open Chest. Use Axe. etc etc.

https://ken-ouyang.github.io/text2immersion/

And finally, En3D: a tool for making rigged human 3D models for future metaverse avatars – from either zero-shot single images of existing people – or to more slowly build up your avatar from text input.

Definitely missing out on a lot of depth detail in many places, moreso when using the zero-shot mode – but a great leap towards simply descripding what you want.

https://menyifang.github.io/projects/En3D/index.html

All three tools suffer from ‘not-quite-there-yet’ syndrome, but similarly to using the new breed of LLM tools, these should all be considered as input into a further steps of refinement. The Luma Genie tool has the most potential, given it can output to fully enclosed and completely arbitrary meshes – and at the other end of the scale, Text2immersion’s gaussian splatting output seems to not have a pathway to that task of further refinement. Yet.

To absent friends.

The post Episode 002: What is Spatial AI? appeared first on SPAITIAL.

Podcast Recorded on January 6th, 2024 (YMMV)

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From Violet:

How could the Humane device alert you with time sensitive information about your Uber, and provide updates on your ride while you’re in transit?

https://michaelmofina.com/blog/humane-concept-04-riding-an-uber

From Helena:

You need answers, and you need them now. But sometimes, you need more than a quick search can offer. Enter Perplexity Copilot, your new digital assistant for in-depth answers.

Powered by the most powerful AI models like GPT-4 and Claude 2, Copilot is in a league of its own. Unlike basic search engines that shoot back quick answers, Copilot chats with you. It asks. It listens. It refines its search based on what you really want. The result? Spot-on answers, virtually every time.

https://www.perplexity.ai/

From Mirek:

Imitation learning from human demonstrations has shown impressive performance in robotics. However, most results focus on table-top manipulation, lacking the mobility and dexterity necessary for generally useful tasks. In this work, we develop a system for imitating mobile manipulation tasks that are bimanual and require whole-body control. We first present Mobile ALOHA, a low-cost and whole-body teleoperation system for data collection.

https://mobile-aloha.github.io/

From AB:

Two years ago, Matt Parker of Standup Maths fame, released a video where he hacked together a way to map all 500 lights on his Christmas tree into a 3D mesh – then invited people to submit their own patterns/lighting designs to try out on his own Christmas tree. The results ranged from meh to simply stunning patterns.

BUT in 2023, a company called Twinkly (https://twinkly.com/en-au) has brought out an off-the-shelf product that replicates all that Matt Parker and the YouTube community made, and wrapped it up into a brilliant iOS app. It’s pretty amazing – it not only lets you map your lights in 2D AND in 3D, but it also lets you gang up multiple sets together, and map then all as one large set.

This means that you can just roll out your lights however you want to / however you’re able, and let the app figure out the spread of lights, and make some truly amazing light patterns.

My wife, Beck, and I bought one set of 200 lights… and had so much fun we kinda had to go back for two more sets. So in the show notes I’ll put links to some quick videos of the final product.

Needless to say, Christmas 2024 will be a VERY Twinkly Christmas at the Ballard house.

Our lights in action: https://www.youtube.com/watch?v=4i-Zbs1HCNc&ab_channel=AndrewBallard

From William:

Is there a world where Digit can leverage a large language model (LLM) to expand its capabilities and better adapt to our world? We had the same question. Our innovation team developed this interactive demo to show how LLMs could make our robots more versatile and faster to deploy. The demo enables people to talk to Digit in natural language and ask it to do tasks, giving a glimpse at the future.

To absent friends.

The post Episode 001: Introductions / 2023 in Review appeared first on SPAITIAL.