SPAITIAL

This episode we cover news from the NVIDIA Conference: AI models like Groot for humanoid robots, AI infrastructure platforms, advances in weather simulation and graphics chips enabling exponential growth in capabilities. We take a peek at early Apple Vision AR apps – revealing the challenges of designing truly immersive experiences. And finally, we cover the rush to build general AI versus finding valuable near-term use cases for current tools.

From William & AB:

Nvidia made several significant announcements at their recent ‘event ‘Woodstock for AI’ event – this includes the Groot foundation model for training dexterous humanoid robots using simulation, new hardware platforms like Jetson-Thor to power edge devices, and Blackwell chips drastically improving computational power for applications like weather forecasting.

https://nvidianews.nvidia.com/news/foundation-model-isaac-robotics-platform

https://www.youtube.com/watch?v=9vEaImsSCrw

We’ll return to the NVIDIA week of announcements in coming episodes – including looking at their potential move to being cloud-only/timeshare-only for their new Blackwell super-high-end GPUs.

From Mirek:

TechCrunch is trying to be upbeat, but in this list, it’s plain to see that early Apple Vision augmented reality apps are mostly disappointing ports of iPad games and tools not taking full advantage of spatial computing. We call out a few notable exceptions, but we’re also seriously waiting for more compelling use cases – likely involve complementing existing human ways of working.

https://techcrunch.com/2024/03/22/apple-vision-pro-apps

From Violet:

As voice interfaces become more common, conversation design will be critical for natural interactions. Tools like PlayHT and Voiceflow show promise for integrating both structure and AI to balance flexibility with deterministic flows for business needs.

https://www.voiceflow.com/

https://play.ht/studio/files/b75eb4c3-6662-45b3-99ab-a062b561a78c

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

To absent friends.

The post Episode 015 – Spatial AI News of the Week appeared first on SPAITIAL.

In this episode we’re going to look at IoT sensors and how that relates to spatial AI. We know we have computer vision, NeRFs and GaSPs and photogrammetry and (insert any acronym here) – but there’s so many more ways that we can make our local worlds much more intelligent through spatial sensing.

Fortunately, we have some professors in the room. We have Mirek and Violet and William here with us. Violet, this is absolutely your domain. We’d love you to give us a bit of a better summary than mine of today’s topic and how it relates to Spatial AI – cheers!

Violet 01:21
Yeah, so I want to start out with a question to you all and then I’ll give you the question and I’m going to talk a little bit more before I let you answer, and then I’ll ask you the question again. I’m telling you that now so that I can get good answers from you all.

So the question is, how does an environment become intelligent with spatial sensing? So what I mean by that is – how the environment itself becomes intelligent, not the agent, and not the individual robot.

So we have a bunch of robots now in our home environment. Just think you have security camera or a smart nanny cam. Maybe you’ve got things like motion sensors and object recognition in that. Maybe have smart door locks. Maybe, you you smart lights. Maybe smart heaters and coolers if you a nest. Maybe your have Roomba. Maybe in the future we’re gonna have humanoid robots.

Okay, so they all kind of live in their own world and they have their own reasoning and sensing.

So something, you know, William has thought a lot about this as well, is: How do we make our environment intelligent? So not these individually acting agents, but how to we get them rather than being individually programmed?

Violet 02:47
Like right now I have a sensor or a light and I say, hey, go on every time it’s this time of day. But you never have any sort of coordinated actions across different devices. So in my classes, we play a lot with If this, then that (IFTTT). We use a little bit of, you know, some off the shelf open AI stuff. See what we can do to get different things talking to each other to do more coordinated action. But that’s all very still recipe driven – it’s orchestrated. It is not reasoning in any way. It’s not like you have some situation where, I don’t even know if this is a, this isn’t a good design prompt, but let’s say you had an AI that can control all those things in the home.

Now an intruder comes into the house and maybe it’s just the camera that starts, but the lights start flickering to like divert them from the fact that you’re actually in the other room or, or that, yeah – Home Alone!

Violet 03:52
You know, it’s just like, there’s more, and I’m sure that’s like a terrible use case, but there is more capability of orchestrating programs that are multi-dimensional like that but require, that have multiple sensors and actuators from reasoning.

Okay, so one more like little story I am going to share and then I’ll ask the question one more time:

So I was recently talking to someone who worked as a data analytics something at Ford Motor Company and what was really interesting when talking them was it what really was impressed upon me is in every single one of these cars there are just so many sensors and actuators just in an individual car and same thing for our phones…

…and it wasn’t until that moment that I really realized how much our world is just a fleet of so many sensors and agents and actuators out in the world — but they are all not aware of one another — they’re not orchestrated or coordinated as like a swarm.

And the same in our homes even when I you know if you ever give presentation in a big room. If you just think about how many phones, individual devices are in that room with sensors and all their capabilities, each one of those is acting individually. It’s not coordinated, but maybe there are opportunities to create these kind of coordinated environment kind of behaviors.

Violet 05:27
So I just thought that was interesting. You know, that technology right now is kind of lacking that kind of coordination – so my question to you is more of a technology prompt or like you know having come from a product management background I’m trying to make my billion dollars now and I am hoping that you will tell me how to do it.

How do you actually build a system that makes an environment intelligent? What is the product architecture, maybe is there certain types of technologies that need to exist? Like, do we need a local network?

Is it all Wi-Fi? I don’t know. Is there a certain type of reasoning that needs to be in place to enable an intelligent environment? So maybe, yeah, I, I dunno if you have any initial thoughts about that, but…

If you have any ideas, I’m going to just take some notes and then, you know, just go make my billions.

AB
Yeah, sure. Mirek, I can see your cogs working over time…? Penny for your thoughts!

I think this comes down to like the years ago, started to be this hype about IoT, and you know everything will be connected. Everything will be generating data, broadcasting, but then it sort of went away. And I believe that’s because quite often there’s simply no need for everything to be interconnected with everything else, as long as your device that you bought performs the task that you want from it.

Mirek 07:11
That’s fine. And it doesn’t need to talk to everything else unless you need this agent that connects things together and that might be your personal AI of talking to APIs that individual devices have if you want that but very quickly you might have not just Roomba but robotic lawnmowers and I don’t know, you might think of a burglar being attacked by in this fictitious scenario by your water sprinklers right – but most of the time water sprinkers don’t need to be really smart! So would you pay $2 ,000 more for having them connected to your home computer or would you not want a $10 for a smart on-off sprinkler?

There’s two different modes for no reason but that’s maybe the the reason why these things aren’t integrated because people use appliances or tools or machines for individual reasons and I don’t necessarily need my dishwasher to be connected to my Roomba you know.

AB
I’m a bit more ‘pro’ the topic… Violet, I think you’re definitely ahead of the game without fail – but that’s always a problem being out there because you don’ t exactly get to buy your own tropical island and – case in point – us four here are NOT living on our tropical islands, so we haven’t yet ticked that box yet.

I’ve always been completely wired, definitely had the fastest internet connection in my whole town since day dot. I think most of the neighbors know my Wi-Fi password and use ours if they desperately need to.

My son bought me a Google Home which I thought “oh that’s nice” – didn’t really use it much – and then I realized oh my god I can go crazy now : ) so I had to buy a second router just to handle all the low bandwidth, Wi-Fi lights, taps, everything.

Would you believe my coffee machine requires both a WiFi password and a Bluetooth? If I put in a new capsule kind that it hasn’t seen before, it goes and checks what settings to do the coffee on and the Bluetooth is if there’s an immediate problem it gives me a tingle to my phone and watch.

AB 09:31
But they are all, I think you’re perfectly right, the if-this-then-that approach or Zapier or any of the cloud mini low-code integration tools are exactly as the integration world was 20 years ago. Even today it’s still not amorphous and AI driven, but they’re still, you can test rules…? and it makes enterprises very happy when you sign off on rules!

But you soon get to a point where there aren’t enough rules in these things to cater for every type of intruder event coming from the south lawn, and I need to fire the sprinklers at them. But that’s the logical progression, isn’t it?

One thing I do smart (I’ll say it nicely before my wife walks out the door for work) is that I have a geofence specifically set up for her that when she returns within a 10k radius it gives me a tingle to my watch and I go and put the kettle on to prepare a cup of tea for her.

So yes I win the internet just for the five minutes for doing that but it’s easy. It’s painless. If I boiled the kettles and she doesn’t want a cup a tea, no peril. But if she does walk in the door chances are the kettle’s just boiled.

AB 10:40
Now those things all combined. Yeah I would love a local smarter way of doing it than have me to do more if this than that to connect to Zapier to connect to Google Home, to connect the Apple HomeKit to connect, it does feel like the integration spaghetti code of 20 something years ago.

I lived through that. I repressed it. It’s better today, but it still a element of spaghetti. It’s a different kind of pasta, but it is still code-based, logic-based, you-can-read-it-out-and-sign-off-on-it. That’s not AI. At some point in time, you realize I could write another million lines of code or I could just get this thing to learn what I do and that would be even better.

William, are you signing up for it and are you quickly doing a geofence to put the kettle on when Violet comes back from work?

William 11:34
Yeah, most likely! I think it’s curious to me that there are some things that we don’t do in the homes that I think are kind of natural extensions of what we do already but aren’t quite smart enough? I think that the Nest thermostat was the big promise and the big bet that Google made and that was a kind of sort of like automated home management direction.

So you could you can set up routines and for when you were at the office to save energy and then you know anticipate your return and then start the air conditioner or start the heater so that the temperature is comfortable when you come home.

It’s curious to me that thermostats in modern homes don’t actually have thermometers deployed throughout the home, so why not have an actuator that controls the vent in someone’s room in a thermometer there so you actually effectively have thermostats for all of your house members.

William 12:40
That would seem to be something that would be not really AI in a sense, but more accommodating. A lot of these home systems have a compatibility problem. They have the problem where companies like Philips and TP -Link and whatnot want you to buy into their own ecosystem, and the compatibility between them is a blocker to adoption.

And then the Google Home and Amazon echo are the examples of the sort of integrator, aggregator pattern. Yeah, yeah, totally. It’s still not AI. Although there’s a famous MoMA entry here at the, in the New York MoMA of where the anatomy of an AI system is, is mapped out, which is the Amazon, which the Amazon Echo is the centerpiece of that work.

But it’s a rather strange work because I wouldn’t have considered the Amazon Echo an AI system. It’s more like a voice control assistant. But I think we’re starting to see glimpses of these sort of intelligent environments.

It is hard to talk about an environment without talking about the agents or occupants of those environments, and really the impact we are talking is how it might affect either the the agents themselves or whether an environment has an impact on a larger environment, that it sits inside.

So for example, what we’re talking about our homes as an environmental and then the agents as humans, the occupants, or maybe your Roomba as well, your favorite robot assistant. Amazon has, I think, an interesting take on an intelligent environment with the Amazon Go stores where you walk into the store and you swipe your card or you swipe you phone and then you can pick whatever you want off the shelves and as you walk out it charges you.

And that’s implemented with like a hundred cameras on the ceiling that then as your going through the turnstile it sort of locks onto you and your identity using computer vision and literally follows you around the store and then maps what you’re reaching for to the sensor that’s on the shelf.

William 15:06
And so they can confirm that, yes, you pulled the protein bar off of the shelf and the Coca -Cola off that shelf, and so that I think is a good example of how an environment gets more intelligent in the AI sense, at least in a machine learning sense.

And I there are other similar examples as well, like the whole promise of self -driving cars is that, where you put enough sensors on a car, it can sort of navigate. But the real benefit to save lives comes from the collective intelligence of all the cars coordinating with each other, either on kind of a neighborly basis or with some centralized system that can predict things like traffic jams and start routing cars around them.

So I think we’re starting to see a glimpse of that. But the Amazon Go store is the one that’s really the kind of commercial implementation. And they started licensing that technology as well. So it’s not an Amazon go store you have to go to anymore.

Now there are these sort of airport magazine shops that implement that same pattern but with the amazon go framework and technologies and I finally have a few photos of some of them. And there are like a hundred cameras on the ceiling that are watching you as you’re walking through the store.

It’s kind of creepy, but maybe something like that we could become accustomed to because our expectations are that it’s just going to charge us. It is not actually going send our information to some other central system, maybe in the future we will have robots in those stores that recommend items to us as we are walking around.

Violet 16:50
It made me think as you were talking about this – maybe there’s a possibility that so many of the robots and sensors and actuators we have today are so single purpose like the Roomba or the smart lock. Yeah, exactly. And so much of that has to do with the limitations of the tech today, but if you have a more coordinated system, perhaps you have more general purpose actuators, sensors, it can be orchestrated together.

II was thinking about, you know, is this is very future oriented. So like deal with my Futurism here.

But let’s say that you have… a cafe in the future. Maybe the cafe itself makes decisions to update dynamically it updates the position maybe it like puts out different types of chairs or changes the music volume because it’s Tuesday or actually tends to be busy right now or actually this crowd over here is being too loud so we’re gonna like mitigate it this way actually we were gonna close an hour early there was a spill over here, you know, like, is there a, um, yeah, maybe in the future, those types of things are controlled by humanoid agents or is it some combination of…

AB 18:22
The environment itself reconfiguring reconfigure or making adaptations?

Violet
There’s something to not having to have single-purpose sensors and actuators and duplicate them everywhere in whole products or devices. But because we have these multimodal things that can reason, let’s just slap like some sensors over here that sense light and sound. there are some cameras and because you have that you can get kind of higher order reasoning that like a person might have.

Mirek 19:06
I still play the devil’s advocate because that’s not how we engineer things and one example to that might be the until recently Roombas didn’t do SLAM – they didn´t have any internal map of your of you home!

They would just go randomly in a direction and turn around if they hit something and that’s how they just randomly covered all the floor space in your house and then was done for cost-effectivity and that what made Roomba so popular because you could actually get a robot that did that and there was not much intelligence to what it was actually doing.

So if it were over-engineered to talk to everything you couldn’t be able to afford it and it’s definitely possible to make system like that but you probably need to have a strong motivation for that – like if you’re building a cafe that responds to moods and you know you can reconfigure it and there’s a certain level of autonomy I think that’s your design objective right? and then you engineer everything around it.

But most likely you won’t be ever able to just hijack whatever sensors are in that area because people just don’t want you to use their phones to stream audio and video just because you decided you would do like to do that.

Mirek 20:30
So there’s all sorts of reasons and I think that the intelligence of the environment is sort of emerging with you, know more and more individual elements sort of living with us in that environment. But if we want something sort of overarching sort, of tying everything together you must have a strong reason for that.

And then, you know, I don’t expect that to be to be a standard because, not anytime soon, because I imagine not that many people enjoy, that sort of surveillance that are inflicting on themselves for a little bit of comfort.

So there’s lots of different elements to it. My biggest sort of point is devices and utilities are engineered for for purposes to be as good as and cost efficient as they can. The integration is.

Violet
And. And I would say that people for these companies sell these specific one use devices because capitalism and we’re going to sell like if you think about the way that our sensor actuator world works right now, you’re gonna buy a nest and you are going to buy a nanny cam and you’re going to buy it.

Mirek
So I would even find examples of things going the other way. I recently bought a new Philips toothbrush and I’ve had plenty, you know, like fifth one in the row. I’d been using them for years, but this time around, I couldn’t find a version with Bluetooth, which I never would buy, but they just removed it because nobody needs that.

Who needs Bluetooth in your tooth brush?

You know that sort of thing. So, yeah, I think you’re right.

William 22:25
But I do think that there are cases outside of consumer environments where this could be valuable. There are there’s some case studies around cities using coordinated sensor networks around like water management and electricity city management.

There was some talk around some big financial institutions in New York City that wanted to coordinate energy consumption based off of predictive models around the traffic of their employees. So they knew that if rain or a storm was predicted over the of days that the traders weren’t going to come into work, that they were going to work from home, things like that.

I think it works if you’ve already in, like it might work in the first places when there are already environments that are already outfitted with actuators and sensors. And so if your tower already has like ID badge scanners and automated locks, then it becomes a lot easier I, think because you already have that centralized system in place to do something like electricity optimization.

And if you can drop your electricity bill by 20% or 30% and you’re just using the IT department that you already have, maybe there’s something there. Maybe there is some low -hanging fruit in some of those cases.

But again, those are really isolated cases, it’s not that the billion or 2 billion folks that are connected to the internet are going to be super hyper-automating their homes, but maybe a skyscraper that’s 80 stories tall and it accommodates 10 ,000 people during the day.

William 24:14
Maybe that’s better? The New York Times Building in New York is another example where there’s automated lighting louvers and things that can adapt the quality of space based off of how much sunlight is hitting the block and they can optimize the usage of artificial lights and balance that with sunlight.

I don’t know how well that’s working, although it’s one of my favorite buildings in New York. I dunno practically how it actually works. And then there are also examples of people contributing their data, like in Waze, for example, I think that where you’re crowdsourcing traffic.

AB
“My traffic is this”, therefore 10 minutes back in the traffic, a person behind has that knowledge transferred to them in real time.

William 25:02
Yeah, brilliant collaboration. Yeah. It’s like the, I think that there’s some emotional point there where it’s like if you see a, if see, a police officer like hiding, like with a like setting a speed trap that you kind of want to like, at least in the US, the attitude is just to kind of stick it to authority a bit.

And so I people feel some satisfaction with warning others that there’s a speed trap coming up.

AB
In this country, we talk that we should hire 10 people to drive around full-time and just flash their lights. That would lower the speed around the whole nation instantly. And I’ll let you in on a semi-secret: when we’re on motorbikes – and we just did a lovely 1,000 km/600 miles on weekend in high heat – there is a universal signal for police hiding in bushes up ahead. You tap your helmet with your free hand on the motorbike – if you see a motor bike rider there are signals between riders because you’ve got a brief second to wave or say g’day or whatever but yes there are signals for slow down now… that is a thing!

That’s one example of collaborative, slightly self aware environment. So we’ve got ‘home’, which is probably the one where it is obviously most personal. ‘Office’ is where its most practical, but you’d have to say post COVID helped there.

Mirek 26:26
I would just say William’s example with the power optimization – that already exists in new constructions. I don’t want to sound like I’m dismissing – but that’s definitely the case.

But in this case, the savings justify the investment. And you know, the engineering you need to do on top of that. And then you come up with a smart skyscraper, which is positive, you know in terms of energy use, and it’s optimizing as much as possible.

That’s definitely a good use case. And we I think we see a lot of these but it is not every day you see new skyscrapers being built or open. So it was taking some time.

AB
So… rolling all the way back to your opening question, do you think some of the drivers of smart environments or I guess, ‘outside-in’ smarts as opposed to ‘agent-out’ – what would be some of the main drivers?

Obviously smart is good, and there’s a few percentage of the world who are just going to chase after it. “My second router be damned: 10 more devices is okay!” But do you think the drivers might have to be a really between cost/value or safety, a higher order thing to really be the killer reason to get more traction?

Violet 27:31
You know, the interesting thing for me is: Mirek – your original answer was more… actually didn’t answer my question because you focused on the question that you’re asking right now, Andrew, which is ‘the value’.

“Why would we do this?” Like, “people aren’t going to pay for it”!?! I did ask, how will we accomplish should have technically. The reason why I say that is because I’m less interested in it for this kind of capitalist reason.

I joked about the billion dollars, but really my interest isn’t like, how can we get people to buy this? I am all about like making this happen, which you know I should care about that because we live in this capitalist world and that’s the only way anything is going to happen.

But the reason why the value I see in this – which is not necessarily monetary – is that we live in a world that is so driven by personal computing, by individual devices that sell one thing. And I believe strongly in collaboration.

Violet 28:52
And I believe in social environments. And if we all drive our own little car, we get caught in the traffic jam. If we sit on our laptops in a cafe, maybe we do our little thing, but we stop talking to each other.

So I think the more we start to think about coordinated devices, that there’s more potential for us to have more like socially orchestrated things. But I also think that there’s you know that then there is also benefits at like a city and social scale to like the ways that might change.

Now I do really need to address Mirek’s question which is like how like, how do we actually sell this? Because there else is never gonna happen. Sorry that didn’t answer your question at all. I don’t know.

I hope you all have answers for that because I actually don t.

AB 29:51
It’s a joy to be able to ponder the world of theory of ‘what if the world was a happier place and everything was free’. So, yeah, I know, bring on the world of Star Trek with, what was it?

What was the thing that got them over the world? Their “tea Earl Grey hot”? Replicators! The replicator was the thing which made everybody just print gold ingots on day one which suddenly in the world financial crisis made the world economy stop dead.

Mirek
Replicators will make things so much easier.

AB
That’s it. It just turned the word from being money-focused to ‘money is now not the object’. Its research and science – but I do love that and thank you Mr Roddenberry for bringing that to us. And Violet for carrying on the tradition!

So I think the answer is yes we want this. I would be happy to put more cameras, Nests, things in my house if it meant that spatial intelligence was happening faster.

We’ve got a couple of our kids at home, some are already adults – we still have their rooms ready for them when they come back.

But we do a thing called closing their doors which keeps the heater from having to heat that room as much and keeps the animals out. That’s a three-second zero-cost, perfect solution. But yeah, I would also be definitely the first to sign up for a solution that was, you know, “close Jason’s room now”. Or Jason leaves for the weekend and suddenly his room closes itself.

It is interesting. I think, you know, some of these earlier points about just how frustrating it is today to coordinate some these things, we’re moving more towards a world that’s driven by intent. So actually, to Mirek’s credit, one thing you were recommending is it might be something like an AI assistant that coordinates something like this, and we were starting to see these like kind of wearables.

Violet 31:58
So, you know, if my wearable that I have or my personal AI assistant doesn’t even have to be a wearable starts to know the types of things I like, I don’t have this program, hey, turn the lights off at this time, blah, blah blah. They just kind know when I come and go and I can give my higher, like they kind get my a higher level intent. And then I think that is maybe, you now, maybe that’s what a system could look like. It’s like in it. It’s much more about permissions on an individual level and it’s like the home system.

AB
Love it. Let’s keep ‘the future’ in our topic list on high rotation. This is something we can revisit literally every two or three months because the world -as we’re seeing in our Fast Five weekly episodes, is changing ever so quickly.

Robots coming to a house near you / AI responding, coding for us. Yeah all the things which we started this podcast for are definitely been ramping up even in the three months we’ve been doing it.

Might leave it there – that’s Episode 14 – we’ll call that a wrap!

Catch you next week for News of the Week and some more deep dive topics as well as more guest interviews being sprinkled throughout.

Thanks for listening everyone and we will catch you next time on SPAITIAL.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

To absent friends.

The post Episode 014 – Spatially-aware Environments appeared first on SPAITIAL.

From Mirek:

From Figure, in collaboration with OpenAI, we get a vision of the future called Figure 01… if that future has both some minor pauses due to latency AND an overly-humanistic set of filler/padding ‘ums’ and ‘well…’s…

All minor mocking aside, this is a radical fusion of sensors and capabilities, mixed in with natural speech inputs and outputs – and with a side-order of explainability: the robot can simultaneously explain what it’s doing while (or before!) it performs those actions.

Sign me up.

https://www.youtube.com/watch?v=Sq1QZB5baNw

From William:

Devin’s Capabilities: With our advances in long-term reasoning and planning, Devin can plan and execute complex engineering tasks requiring thousands of decisions. Devin can recall relevant context at every step, learn over time, and fix mistakes.
We’ve also equipped Devin with common developer tools including the shell, code editor, and browser within a sandboxed compute environment—everything a human would need to do their work.
Finally, we’ve given Devin the ability to actively collaborate with the user. Devin reports on its progress in real time, accepts feedback, and works together with you through design choices as needed.

https://www.cognition-labs.com/ YouTube Announcement

From Violet:

Bringing back the feel of a hand tool with the precision and repeatability of a CAD/CAM/3D-printer – Shaper’s Origin router has it all. Watch the video to see how it allows for predetermined shapes and patterns, but across any size object. We can’t wait to a) get our hands on one, and b) see what other hand tools can turn into smart assistants!

https://www.shapertools.com/en-us/origin

From AB:

While patents aren’t worth the paper they’re not written on anymore, they do give a glimpse into what a company is thinking about – in this case, Apple, and in this case: if your new MacBook Pro had up to 7 cameras arranged around the corners, what could you do with that setup?

Jeremy Dalton’s post on LinkedIn started a great discussion about the potential implications and future uses. Simple ones like using the cameras to capture small objects via photogrammetry, sure – but how about tracking eye movements and hands/finger movements, to bring the VisionOS gesture commands to a ‘regular’ laptop?

Other commenters have also suggested that it may bring a pseudo-stereoscopic 3D avatar into Facetime calls, for crossover potential with Vision Pro users, and invite more people to collaborate in the spatial world – although non-immersive.

Jeremy Dalton’s Post on LinkedIn

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

To absent friends.

The post Episode 013 – Spatial AI News of the Week appeared first on SPAITIAL.

Well, g’day, and welcome to SPAITIAL. Here we are on a topic edition of the SPAITIAL podcast. The previous episode was a Fast Five where we discussed News of the Week, but this time we’re gonna do a deep dive on a topic all our own.

This week’s topic is Spatial AI for All, and really the subtitle here is How Can Anyone Access Spatial AI? Last week we covered How to Get Started in Spatial AI, and if you haven’t caught it, link in the show notes, there’ll be a link to the 101 Resources to Get Started, podcasts, people to follow, companies to look at, tools to learn, that kind of thing, books, as well as, you know, non -technical advice.

But given that you are on the path to learning Spatial AI, as we all are, the next question is, do you have to start at zero? ‘Blinking cursor’,you know, ‘insert your code language here’. Do you have to get my first Python book and do it hardcore? Or can we get ahead quickly? I guess one of the things is gatekeeping Spatial AI. Can we find a way to fast-track our journey? Or sadly, is it back to first year university for all of us to learn the basics before we’re allowed to do the hard work?

Joining me today, I’ve got Mirek in sunny, sunny? Yes, I can see it’s sunny out of Vancouver, Canada. Freezing but sunny, okay. Well, it looks sunny. There wasn’t a temperature gauge on your screen. Okay, good.

And in sunny Philadelphia, where it’s always sunny in Philadelphia, oh, I’m getting shakes and pouts. So you mean the show title’s not correct? I’m not happy now.

Violet
It might even be sarcastic.

AB
Oh sarcasm, oh right didn’t get that. Violet and William coming to us from not sunny Philadelphia. I dare say probably not hot either and I must say here I’m in sunny Melbourne where we are gearing up for a long weekend of 36 or 90 odd degrees across the weekend. Probably bushfires and my wife and I will be on our motorbikes enjoying it and dodging it so in six months we’ll swap roles. All good.

Violet love to start with you. Coincidentally or perfectly timed or I don’t know how you managed it but really awesome – you have a very long post on Medium under your account of your and William’s projects, open questions, actually calls for help as well as you know anyone who wants to come and join. With the massive heading exclamation point “we need a natural language programming environment”.

Now I take it it’s a general sense but we’re going to put that into the context of spatial AI. Can you give us a rundown of what you’re thinking and basically outline the points that we’re going to talk about the next 20 minutes of how do you get started fast.

https://medium.com/spatial-pixel/we-need-a-natural-language-programming-environment-c4dfd3efcc7f

Violet 03:13
Yeah. Well, it’s interesting because also this, you know, we just heard the CEO of NVIDIA say, your kids don’t need to learn how to code in the future. We’re all going to be programming with plain language.

And I think it’s interesting that right now, when we’re programming, people that are programming, you’re still interacting with the chat interface. So, you know, for me, I do a lot of working with GPT4 just in the chat GPT interface and a lot of copying and pasting code back and forth.

I’m a terrible software engineer, but I’m very good at hacking things together and looking things up. So this is how I do, this is how I program work anyways.

AB
I think we, and I’ll say we call ourselves script kitties, is that still the term? We borrow code and sort of mash it together. So script kitties is the, “I didn’t write the script, but I can turn it on and interface it and send it on its way”.

Violet 04:17
Yeah, yeah, I like the term. So yeah, there’s something to that, but the whole point of the article is that programming is a lot more about thinking much more holistically about the system, understanding concepts like recursion, how things interconnect, the operational float.

So I would love that if in the future, program literacy is increasing, and let’s say anyone can program with natural language, that that’s not a hand it off to chat GPT, or AI, like AI go do this thing for me, that it’s actually upskilling people just like literacy was when we learned to read, to learn to read and write.

So in order to do that, I think we need more interfaces like programming interfaces that allow people to really see the whole picture, not just in language, probably visual elements. So that’s really what it is.

And I think it relates a lot to spatial AI, not just language, because programs, in my opinion, naturally, are spatial, any type of program is, even when it’s in the computer, it’s like oftentimes intended to impact things that are in the physical world, a calendar is orchestrating where you go throughout the day.

Violet 05:57
It’s controlling elements, oftentimes like where we go and things related to a map. So I’m interested in a way that in the future, when there’s more literacy for people, anyone can write a program, how might we start to build interfaces that allow us to interact more naturally with the physical world, with things like language and gesture and, yeah, that kind of thing.

So yeah, and I’m actually curious to hear from everyone else here about like, is there potential to allow more people to design and create things with spatial AI without having to understand the latest machine learning models?

In a lot of my physical computing classes, we try and take off the shelf sensors and hardware to build any sort of tangible computing interaction and we’ll slap AI on top of that. But we might not be the person writing AI, but there’s still so much to be designed at like a few layers up.

AB 07:22
But that same off -the -shelf for hardware, you’re asking, can I have an off -the -shelf? Assume that the black box will do whatever I need it to be, and I’ve got to say, trusted. But yeah, how far can you climb up the ladder without having to resort to a blinking cursor?

And I will fix it myself manually. Mirek, you’re definitely deep in the weeds. You said you’ve just spent the last nine months, 12 months playing with middleware, and I dare say you dream of code. But what’s your sort of experience in leaning towards tools or being 100 cents self -sufficient on this, you know, when you are far deep in the weeds?

Mirek 07:59
Yeah, I don’t really know how to approach this conversation because I’ve seen memes about coding no longer being relevant and then somebody comes up with the definition of coding and how it’s actually a human description to the machine of what the machine should be doing.

And we just happen to call it called and write it in Python, right? And you might say Python is pretty English -like already, one of its design principles. And it’s not so hard to learn Python, like they’re much more difficult languages to learn and I could go on about that forever.

But there’s also this other element, which is that if you write code, you use all sorts of aids and assistance all the time and the IDE has, you know, you might start with code compeletion or some sort of, you know, insights into your code and refactoring and all that.

Mirek 08:53
So we’ve been using pretty sophisticated tools to write code already. And most of programmers, including myself, would be lost without an ID that does have the job for you. And if you just try to open a note button, try to write some C sharp, you won’t get very far.

I don’t really know where this is going. I think we’ll see more and more tools like Copilot, which is brilliant. And I wish we would see something that you can just throw a bunch of code at GitHub repo and it would just understand it.

And if we would train it on the spot, maybe, you know, it will be able to tell you different versions and how different API is very over different. You know, this is the hard part to find matching pieces of code that you need to compile together, make them talk together.

And this is insanely tedious and something like, you know, Copilot that can actually sort of help you figure out what the proper version of something or fork or even, you know, just different packages doing the same thing.

So if you have if you had something that you can feed your problem and, you know, code and stack overflow related posts and to just be able to have a conversation with you about what the best approach and then write a code on this isolated topic or, you know, the covering this this particular issue you have.

I think that’s going to be really, really enormously useful. And we see that coming already. I don’t think you like what engineer does essentially isn’t just writing this complicated code that’s complicated for its own sake.

It’s it’s, you know, sort of connecting various different things together, making them talk. And that itself requires knowledge of those individual pieces, right? If there’s an AI helping you with that, amazing.

But the person in the middle will be probably most likely more of an architect. And you know, with help of different subsystems, you’ll be a better architect, better software engineers. And I’m all for that.

Mirek 11:14
I just feel like it’s not going to be like new language coming along. Maybe you’ll get far with, you know, just just having GPTs generate Python code for you. That’s that’s fine. Where there’s lots of examples such as, you know, the whole that industry of development, you know, subset of computer engineering, that’s really well, there’s lots of data that you can train on, which means that these these compilers and produce really good results and are really helpful.

I’m just not sure it’s going to be a new thing that comes along and, you know, will replace all the tools that we’ve been using. I think it will be gradually more and more helpful in the process of, you know, meshing things together and even figuring out what library or what kind of solution to a particular problem you might you might be using.

I think this is pretty, pretty broad and, you know, general sort of vision for how we write code, how we how we talk to computers.

AB 12:16
No, it makes perfect sense. I could probably be controversial, which is pretty rare for me, and agree with our NVIDIA CEO. But I definitely would not want my kids to be coders these days. Not saying that’s stop everything, but I’m looking forward to a future where, like right now in a software team, for a team of 10, there might be an architect or two, a project manager. The majority are developers, and then there’s a few testers. I think that’s going to flip on its head big time.

I can see a future where an architect – enterprise architect kind of thing, not a building architect – is the one to sort of know what they want and know how, that we have to find a way to get Connect A to B and do requirement X, Y, and Z. If they can enunciate that in the future, I think the big move in the next 10 years is going to be testers. I think the ratio of architects to testers is going to be the moral majority of technical people, potentially, in five or 10 years’ time.

AB 13:13
The reason why is that you’ll probably be able to get a lot of code out pretty quickly from someone doing the generative assisting world. Granted, asterisk, the person doing that still needs to know what’s possible and is it vaguely doable versus asking for the world.

But then once that’s out, I think then, we’re going to need squads of testers to put it through its paces. To be controversial and controversial for a different topic, for a different day, I love the fact that, if I was putting it, it would be really nasty.

I could say, you can never fully test AI ever, and that normally scares a lot of people who I talk to. But it’s kind of true in one way and kind of not in the other. You can’t say exhaustively, we’ve tested everything, it’s good, we can finally go with it.

At some point in time, you’ve got to say, let’s just, we’ve tested enough, we have to start using it. So I think testers will be the ones to assess whether the code that you’re about to deploy is or put into practice is, it’s within band of safe, we’ve tested it for X days.

AB 14:14
We can go further if you want, but hey, it looks good to us for now. That kind of approach might be the ratio.

Violet
Mm.

AB
and rolling all the way back if that means that there might be, yes, more capabilities in people to do more things fast, but the inbuilt knowledge of people to know what they can ask for. Again, Mr. Donald Grumsfeld, unknown unknowns, you’ve got to be able to know what you can ask for.

So rolling all the way back even to the topic again, William, question for you is in a future natural language sort of way to interact with the world slash spatial data or projects or just the world around us.

AB 14:53
How do you discover, how do you sort of learn and play with an interface that doesn’t have anything more than even just a microphone? We used to a blink and cursor, the moment we start typing a function, hopefully the coding environment finishes it for us. How do you play and how do you learn when there’s not even that? How do you sort of explore and vamp on threads?

William
Well, I don’t think that that’s ever going to be the case, really. There’s always feedback in programming. And I think part of the skill of programming is learning how to craft the feedback that you’d need and want at any given time.

And so and that and it’s. And when we when we talk about coding anything like a video game or a complex system or even just a kind of automation routine for our desktops, it’s it’s either it’s quite extraordinary how deep all of that implementation is.

It seems very easy to us when we write Python, for example, for say calculating a budget or a financial use case. But like once you once you get into something and you remotely complex, it’s there’s so many layers of abstraction to go through.

I think that’s what was some folks on the outside of coding don’t quite don’t quite realize the kind of problem that Mirek was talking about. Mirek, what you were talking about was, you know, like, can you even get these two things to compile with each other?

Mirek
Right. That’s not Python. That’s, you know, that’s later you figure out you can’t and this is because of these little things that one guy somewhere is maintaining and went on vacation. But I think there’s space to assist with these really mundane tasks.

You have to have all these things in your head, usually it’s one person because it’s really hard to share. I think there’s a space to actually assist and aid people with all this.

William 17:00
Yeah, totally. When I’m using GPT -4 right now to aid in my coding, it’s usually an interface to a library that I know exists, but I don’t want to spend the time to read all the documentation for it.

So I’ll say, hey, GPT -4, can you give me an implementation in Node .js to open a WebSocket and hold it open for these types of requests, and it’ll give me a stub of code. I still have to know what a WebSocket is.

I still have to know the semantics of WebSockets and so on and so forth. I’m very much looking forward to that future you’re pointing to, Miracle, where I can say, like, here are the libraries that I’m interested in.

Here’s even a higher level than that. Here’s the functionality that I need. And it just sort of goes and grabs the libraries and pulls them together with a package manager and gives me the environment, and I’m ready to go.

I think beyond that, what we’re talking about in terms of natural language coding is really about giving the thought processes that coders use every day things like creating your own abstractions and formal logic with the things that you’re working with, making that easily accessible to people.

William 18:23
But they’re still going to have to know, I think, what formal logic is. And that’s like a piece that’s missing, at least in US education, that’s usually ceded to mathematics. But the way that you go is you go through algebra two to pre -calculus to calculus.

And maybe programs are different these days, but when there was the you should learn to code movement going on in the arts in particular, there was always this duality about it. Not everyone needs to learn how to code was the kind of resistance movement, but what they were really saying was not everyone needs to learn to write code to the extent that they will become a software engineer.

But there are plenty of professions and plenty of times in life that use formal logic, even if it’s veiled under a different set of language like lawyers, for example, lawyers are very good about creating, there’s a formal language, and there’s a set of semantic concepts and semantic layers that they use that are specific to their profession.

So I think that those are actually the core skills of what we mean by coding, which is not necessarily encoding something in the future, but being able to speak to a machine and give it conditionals, create like a kind of semantic back and forth where you and the machine begin to speak the same language about something and can make that actionable.

William 19:59
So that’s what I’m really looking forward to. And I think that software engineers, there will be an equivalent that smooths over everything below the value line, just like trying to get libraries to compile with each other.

But then you’ll be able to speak about the concepts more and more, you’ll be able to speak about the concepts that are valuable in the context that you’re in. And that could be you’re having fun with a particular AI capability, or like you’re trying to ship business logic, and you don’t want to get bogged down with with sort of the the intrinsic difficulties of the mechanics of gluing together like lower level lower level systems.

Violet
Well, one thing that I think is interesting is that because we’re moving more towards that higher level of abstraction, for folks that don’t learn how to program, let’s say not even like code, but in the future are not learning those same skills, um, in this future, that’s much more based on the AI understands our intent, uh, Hey, I need, um, this was an exercise I was doing the other day.

Um, because I work a lot with sensors and, okay, like told chat, GPT, you’re a Rube Goldberg machine. These are the different, um, IOT sensors you have to work with. Okay. Let’s say we just tie that together with some function calls.

So it can do this automatically. I connect it with IFTTT and it all of a sudden it can control these sensors. You’re a Rube make sure that my dog doesn’t eat my plant. Okay. So I just give it some really high level intent for the design.

Violet 21:44
Then it came up with all these different, I didn’t actually string together the function calls, but it did come up with designs for what sensors to string together. Okay. We’re going to use a camera.

We’re going to detect when your dog is in this area, when it’s by the plant, and then we’re going to play a sound or whatever it was. And so, and this is, yeah, exactly. So I think, um, there is something there’s one, I think it’s fascinating that we can in the future potentially program just at the layer of intent.

But I also think that there’s this danger, which I think is what William’s pointing to that if we don’t understand more fundamentally, the ways in which it can control and change the system, it will get into these predictive channels or silos that it’s like always thinks a certain type of solution is the right solution.

Violet 22:48
Oh, what you need is like, spray your dog with this automated water squirter thing or whatever it is.

AB
around the whole world to keep a dog from eating your plants. Yeah. Versus there’s a dozen ways if you think about it. Yeah.

Violet
Yeah, absolutely. It never uses web sockets. It does some other thing, or whatever it is.

AB
love the conversation. There’s a lot of tools I’ve put in the 101 Resources article, but one of them is one of the premium ones:

Dataiku is the tool where I learnt what the difference is between no code, low code and pro code. Yes, those words all rhyme which makes it easy to remember. No code is you just rock up and do stuff and it does it for you automatically. That’s equivalent to asking GPT to just do it if you can copy and paste it. Low code is often graphical, lightweight, connecting things with lines on a thing. Under the hood there’s all the regular code. It’s just a link output of that to there and it’s actually quite nice. For some people who are visual and me amongst them, that’s quite nice. I can see that my code is the same as if I wrote 10 ,000 lines of code but I know it’s working and often see red lights and green lights makes people happy. Pro code of course is the real thing under the hood.

AB 24:08
My grand question to us all really quickly is what do we think the ratios are going to be today versus in a few years time? What’s the ratio of no code, low code, pro code getting smaller? What do we think is going to be a relatively mid -level future?

Violet
Is this for most people?

AB
The moral majority. Yeah.

Mirek
I’d say it depends on the area that you’re in, say, web design, right? And this is coming from somebody who spent 15 years in the industry making websites, making the internet. I wouldn’t do it today.

You can go on something like Squarespace that sponsors half the YouTube, and you can just tell it what you want your website to be, and you want an e-store or not, or this kind of things. And it just generates something that’s good enough. That’s perfectly fine. And you tweak the design of it, the look and feel, but that’s pretty much it. It’s not reinventing the wheel, and I think that’s fine.

So I would think in these kind of applications, sort of natural language programming, if you want to call it, will be faster, much faster, by if you’re looking for something like creating new product that involves lots of hardware, lots of different things, lots of open source, lots of things that are heterogenic in its nature, and it’s kind of hard to keep them together in the first place, that’s going to be much more difficult.

Violet
I like the scenario that in the future, um, folks take English and then they take English programming or something, or, you know, this would be in an English speaking country and insert other language for wherever you’re at.

So I like the idea that almost like Excel made it so a lot of people started writing like little mini programs. I kind of imagine a world like that. Everyone’s kind of got, um, general knowledge. Yeah, I took this in high school, like, or whatever.

We have some like general kind of language programming. Um, hopefully like reasoning. That’s what I hope becomes the future. And then, um, maybe folks get more comfortable with some of, um, a very kind of shallow layer of programming and different domains.

So like, I just think it’s fascinating. Um, I’ll look up how to do something in After Effects on YouTube, and it’ll be like a seven year old teaching me how to do, to do something. I actually hope for that future where there’s kind of like niche bespoke, additional things in programming that all of us learn that might be more attuned to like your domain or something.

William 27:20
I still am very inspired by Star Trek in a lot of ways. So I look forward to, I still look forward to an assistant indeed, but like having, like those scenes where they have a back and forth with a computer and what they’re really doing, I think in the, at least in the next generation is they’re using the computer as a means to think about the thing that they want to think about and let the computer think about the things that are necessary, but that they don’t want to think about.

And so being able to kind of inhabit that layer of abstraction that you want to inhabit for whatever reason, it could be utilitarian. It could be you’re a business person trying to use a low code system to automate something because it’s just, you know, it’s just too costly or too, like too involved to get somebody else to do it for you.

Or it’s really about like some area that you enjoy, like you’re, you want to learn about hardware, but you need something to help you like bridge the gap of Ohm’s law and some of those, like some of those things. Like I want us with technology to inhabit more of the things that we want to do in our brain and with the time that we have.

AB 28:44
Yeah, nice. We might start to wrap up. I might just bring it all back to the start and violate your medium post. The very bottom is you’ve got a question of, you know, bring the cup downstairs. I’ll draw parallel to that lovely thought experiment, I think by the Aussie academic, bring me a spoon, which is basically if I ask a five -year -old to bring me a spoon, it’s likely that a spoon will appear.

https://bringmeaspoon.org/

We want a robot to do the same. So you’ve basically reiterated that now with even not just a robot, but I want things to understand me. But you’ve also asked for help. Do you want to ask the internet for what kind of help and assistance you’d like in this domain?

Violet
Yes, please help me! I’m hopeless. Um, no, I’m serious. Um, uh, we are working on ideas around a natural language programming environment. I really want to see programming tools for natural language. Um, I don’t think we, we have tons of examples of code editors, different types of programming environments and playgrounds. Um, and we’ve seen chat GPT slapped onto all kinds of things. Um, people are putting it in unity and Roblox and whatever.

I don’t think we really yet know what a programming environment is going to look like. So if you have any case studies or projects you’ve seen related to natural language programming, if you’re thinking about how semantics relate to program and space, um, if you’re just interested in the research, please send it our way.

You can find us on LinkedIn. That’s probably the easiest. Um, but yeah, we would love to see anything related to natural language programming.

AB
Links in the show notes for everything. Mirek, I can see your brain working. We’ll come back in a separate episode, but I really want to explore the way that you three, and I’m pointing to the screen, which doesn’t really work in an audio format, but love to explore the ways that you guys are actually talking about your work, Mirik, of making the way to command control robots machines using an XR interface, and then going one step further, and yes, but it was even interface -less completely.

I take it a mind-melding session to do the Star Trek Spock analogy. He’s probably warranted, and gee, I’d love to be on the fly on the wall when you three talk about that. Done, it’s gone all round.

That’s a commitment. I’ll hold you to it. Well, thanks, everyone. Oh, awesome. We’ll pull up stumps there. That phrase doesn’t work everywhere around the world. That’s all right. It’s a cricket thing.

We’ll leave it there. Some people have to go put the heater on and have dinner. I’m going to go hop on a motorbike and go cruise Western Victoria, so it should be fantastic. Thanks for listening. We’ll catch you very soon, and by all means, Links are in the show notes, and we’ll catch you on the next episode of SPAITIAL.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

To absent friends.

The post Episode 012 – Spatial AI for all / the future of coding appeared first on SPAITIAL.

Recorded on March 8th, 2024

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From Violet:

Oh my – half-funny/half-serious/half-bad-at-maths – is THIS the future of Television? Sadly… we think they’re onto something, especially when they release the version that lets you watch TV with a buddy, all in virtual space. A must-watch.

https://sandwich.vision/

Create your own cinematic fly-throughs of open world location, using Google’s own 3D Mapping and Satellite Imagery library, and keyframe your way to stunning visualisations. As mentioned in the Podcast – this is so good – we want this for our TVs to play on loop as background art!

https://www.google.com/earth/studio/

From William:

Meta comes to the party with open code, papers and an insightful rundown of how their new V-JEPA AI/ML model learns – and can be used as a foundational model for your projects, too.

By using unlabeled video feeds – then removing (masking out) portions of the video, Meta has arrived at a model that can reconstruct temporal- and space-sensitive guesses at how to fill in the missing piece. The good news is that this model approaches the world of ‘spatial intelligence’ from yet another angle, giving more tools to help understand the relationships between video of the world… vs the real world – and how to go back and forth between the two concepts.

https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/

From AB:

Just ONE DAY after we finished the 101 Resources for Learning Spatial AI – Helena and Daniel O’Donohue have launched their new Podcast, focssing on how to Market, Brand and otherwise smartly-and-elegantly Promote your own geospatial business – in style : )

Episode 1 is here: https://www.marketinggeospatial.com/scaling-your-geospatial-brand-with-thought-leadership-content/

So first up, it’s a ‘GPT’ – a custom-chat interface bundled into your OpenAI Pro Access plan (circa $20/month) – but NOT available in any other way. What it is, is a pre-prompted chatbot that is predisposed to handling and talking about geospatial and geographical concepts and geoscience terminology. Backed by ChatGPT4, it’s quite impressive, and in my trials trying to break it/find it’s limits (because of course that’s what us human do to ChatBots)… it’s more focussed than your regular ChatGPT session, but not Professor-level by any means.

https://medium.com/data-storytelling-corner/geogpt-revolutionizing-geospatial-data-analysis-with-ai-78e3f7e4c1e1

Also compare and contrast to Geology Guide (https://chat.openai.com/g/g-o91x8Zr2A-geology-guide) – a less-excitingly-named GPT, but it it more into the data wrangling and AI, than the science ond history of geology. Your mileage may vary.

To absent friends.

The post Episode 011 – Spatial AI News of the Week appeared first on SPAITIAL.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

AB
Well g’day and welcome to SPAITIAL. This is the inaugural episode of a guest interview. Inaugural. Am I going to already question why is that word ‘inaugural’? Can you have the word ‘augural’? Anyway, it’s the first ever guest interview.
I’m joined this morning to my right, Helena, there in New Zealand. Good morning. How are you faring?

Helena
Yeah, good. Thank you. And very welcome to Scott. We’re really pleased to have you on.

Scott
Yeah. Yeah thank you so much. Yeah I am really, really happy to be here and actually honored to be your very first guest.

AB
The very first. In this country we’d call you the guinea pig. All good, there’s nothing to be worried about.

Only the two of us having a chat to you, we haven’t got the full team. We thought it might be a bit rude to have five people asking you questions at the same time. So there are two us here this morning, loving to having great chat with you. But look, if you have a few seconds, can you tell us a bit about yourself, a quick bit of your history, but also what you’re doing in that part of the world?

Scott 01:36
I’ve had a long history in the geospatial field. It’s kind of like kind-of weird how it all just worked out this way.

Even when I was a kid, I’m a maps guy, always staring at maps. Also, too, I am always kind like spatially aware. I always want to know what direction I am facing. There has been times in my life like you get out of the subway in New York City and you don’t know quite which way you are facing and it’s really disconcerting for me.

So anyways, you know, I have a long history in geospatial. I got my bachelor’s at Penn State University. After there, I worked at MapQuest .com for two years. Then I went, got my master’s at University of South Carolina.

So I kind of like made my way down the east coast of the U .S. Um, then, uh, so I did my master’s on – I always say like at that time, I don’t know what you’d call – like mobile mapping? So this was a very, very early – *early early* stages of GPS enabled smartphones.

This was the prehistory of the Palm Pilot, 2D mapping. So there was 100 by a 100 pixel black and white screen. And I was able to put some imagery on that so it was kind of cool – like it actually worked and it picked up the GPS location and showed me you know based on where I it showed you how really you know images that you could kind of make out water bodies and stuff so it’s kind of cool so anyways then after that I worked for Esri so when I when I was getting my Master’s I took programming course Java at the time and And then that kind of helped me, also too, I wrote this application for my master’s in Java that helped get this job at Esri as an application developer in the DC office.

Scott 03:34
That was a really exciting time, working for some really like exciting clients there. And basically after that, so I went down the East Coast and then I came back up the east coast to my hometown, which is in my Alma Mater.

I was a researcher then at Penn State University, my alma mater, for basically like 15 years. I worked at a research center there called GeoVista Center. It was really a wonderful job. I love this job!

AB 04:03
Scott, what years are these? We’re talking late 90s by this point in time, or 2005-2010?

Scott
Yeah. So, so I started there – GeoVista Center – started in about 2005, Spent about 15 years there, also in the process. So that was great because it was, the GeoVista center was multidisciplinary.

So we were based in geography, but then we had so many collaborators from, next door to us was the College of Information Sciences Technology. We had computer engineering collaborators. We have landscape architecture collaborators, sociology collaborators even towards the end, a lot of folks who were handling some of the big data.

We kind of like a umbrella organization called, what was it, ICDS? What is it? Basically, it was of promoting big-data & data science. So we had a lotta amazing experience and I learned so much.

Basically, it was of promoting big-data & data science. So we had a lotta amazing experience and I learned so much. I was really blessed to be around so many smart people with so many creative ideas.

AB 05:11
So 15 years of research in the mapping in the geospatial world, that’s a phenomenal length of time. You said research as well as teams coming in, your background at ESRI must have given you a front row seat.

Were you pushing technical boundaries or were you pushing customer questions? Were you sort of trying to go into uncharted territory? And I mean that in a bad way with maps or you trying actually do larger data sets and get ready for a more public mapping world?

Scott
Yeah. So basically all of the above, I’d say. And you mentioned about sorta my experience at Esri and it was kinda like once I came, it was like, so we had this really fascinating project. It was the very first project I worked on.

This was, you know, keep in mind, again, this was like, maybe what, 2005 or something like this. So we had this, it was we were doing a collaboration, NSF funded, but then we are doing a collaboration with Port Authority of New York, New Jersey for like crisis, the, you know managing large scale crises.

They basically handle all of the security for the the area of the New York metropolitan area. So then we were building this system where it’s a large screen display. And at that time, we had this camera that recorded people’s gestures and speech.

And basically, you would point to this large-screen display and you’d say something like, there is an event here, please draw me a buffer around this point, this area here.

And so it was really kind of fascinating that basically when I came in, boom, right away, I’m kind like building the mapping interface for this.

So it’s really kinda cool. And it just, so right now, like large language models, super, super popular, everybody’s talking about it. So, it reminds me of the missing part of that particular project.

So, back then, so the speech recognition was really crude at the time, and also too, you would have to use really precise words and then once you would say something, it would recognize the speaker and it was kind of just like we had a whole bunch of sort of like if -then -else statements. If the person said this and this, then perform these actions.

AB 07:44
You were almost saying code, like a “go north”, “open this”?

Scott
Kind of like hard coded, yeah. But now I feel like, with large language models, this can easily interpret now what kind of a natural language, what you’re trying to say, and it can even give the instructions on doing the command.

Maybe the only missing part is like kind what people are working on now is connecting that large -language model to to action performing functions. So, yeah, so anyways, I’m telling this story, but so I don’t know if this answered your whole question, but basically then to summarize the way I felt about this research was we had our funding agencies coming, you know, the various funding industries like we had NSF, you Know, National Geospatial Intelligence Agency, DHS, but we have a project for centers for Disease Control and Prevention, CDC.

And basically, they kind of like have their problem. And it was kind like our job to try to figure solutions for their problems. So it’s like really super like, you know, stimulating and I love the, I loved all the meetings with, once again, like I was really blessed to be around some super smart people at the time.

So I learned, learned a whole lot.

AB 08:59
Absolute joy to beyond the leading edge. I mean, to being in a place where that was the question being fired at you, every alternate week, would be quite phenomenal. Can you look back at that? I mean, 15 years is a long time – a gold watch comes at 10 years? I can’t see a goldwatch on your wrist right now, but that’s okay – 15 years is a lengthy bit of time where you’re not only cemented yourself as being an expert in your subfield, but that whole team must have just been a go-to for, I take it, bigger and bigger questions.

Were you starting out with the vocabulary from the customers of, can we do this? Yes, we’ll show you. But, towards the end, it was like, you know, how big a question can you ask? And were you taking larger and larger chunks and steps and going from regional to nation -based to what I’m assuming now is a worldwide view of some of these tasks?

Scott
Yeah, so this is an interesting question. So there’s a couple of points about this. So one, we’re supposedly the experts. So for when we’re guiding them in a “this is what we think we should do”. And then you kind pointed out at the end, in a sense, one of the reasons I fit in well there was I had a lot of experience with web programming at that time. And that basically, and then in the process, I learned databases, we had projects where we were collecting millions of tweets at a time.

So I learned all this stuff about how to handle big data. So, I was pretty good at then scaling up the projects. We’re going to first focus on a particular area, but then, okay. Now we want to scale it worldwide. So this was something, and then another point to this is, you were saying about how far do you go? When you do science, you always got to be really kind of careful because the more you’re kind of bringing in much more variables and then when you come to your conclusions then it’s a little bit harder to say with confidence that you’ve kind of done what you did.

This was always like throughout the process. It’s always “let’s try to keep a focus on this”. If we accomplish something else, it’s cool – but try and keep the focus of the main problem because this is the actual problem we’re trying to address.

AB 11:30
Yeah, gotcha. So there were ‘bright shinies’ everywhere. You probably had a blackboard or a whiteboard of things you would like to tackle in the future. But every task you were being put forward, you will try lock in most of those variables and just focus us on the task at hand, here and there and take slow strides, but in looking back, you took large strides towards making maps that, you know, really are there for larger teams.

Scott
Yeah, and so this is something like it makes me, you the way you’re talking, it make me think of, it didn’t always happen that way because for example, I mean, at that time I was doing a lot of coding, right?

And now at this point in my career, obviously, you want to code for a purpose, right? You want to solve an end goal, you wanna solve and end real world goal a lot of times.

Nowadays, looking back, there was many, many times, I kind of did stuff because I was like, this is really, really cool, you know? But in the end, it didn’t really serve that purpose.

You know like an end goal. So, oh, okay, this is really cool to do. But we really need to keep focused on what’s our end goal? What is the problem? The real world problem we’re trying to solve. And so this is another thing, like in the process of while I was a researcher there, so I mentioned that we were based in geography department, but then we had collaborators from next door, which is the information sciences and technology.

Now, I think it’s still called the College of Information Sciences and Technology. It’s multidisciplinary itself. My degree is actually in informatics. It’s multidisciplinary itself. But one thing I really, really liked about them, they were focused on using technology to solve real world problem.

So this is something that was really ingrained in me, even in my PhD is you really got to keep a focus on: who’s your audience? What is the problem you’re trying to resolve? How is it going to benefit them? How is going benefit society?

Helena 13:39
Yeah, I think especially in geospatial, there’s a tendency to be very technology-driven. So I’ve seen it referred to as like the ‘geospecial product trap’, where people build cool applications and do cool things with the technology, but it’s not really addressing a real -world problem.

And so then you’ve got this tool or you got an application and you realize no one actually needs it because it wasn’t actually problem driven, it was technology driven. And I like how you really come from that other angle, Scott, which is, you know, a breath of fresh air, especially in this like, heavy technology field that you’re operating in with the AI and the geospatial, but you are saying, let’s not be driven by the capabilities of the technology, lets be driven the problems that we can solve.

I think that’s a really powerful position and it’s really a position where you’re going to be able to help so many more people than if you just kind of get hung up in the technology and think like, this is all the cool stuff I can do and I could do anything for anyone. But then it is not really actually focusing on anyone’s problem in particular. So I know you’ve been working a lot in health field, I don’t know, do you want to get into that a little bit and tell us how the story continues from after your time at the GeoVista, was it, to what you’re kind of doing now and more kind of how you are bringing together these different crowns that you’ve got like the applied research strain and the geospatial strain in the AI and how your weaving all of that together.

I’d love to hear a bit more about that.

Scott 15:22
Yeah, yeah, certainly. So like I said, I really loved my time at Penn State. So many nice memories, so many great colleagues that I still keep in touch with. Some amazing people, maybe I should mention actually, some names.

I don’t know if any of y ‘all have ever heard of Alan McEachern. He was basically my mentor. And one thing I, so there was lots of things I loved about Alan, but one thing in particular. So he began his career as a cartographer. He got, you know, he got his bachelor’s master’s PhD in cartography, and then he became one of the world’s leading figures in Cartography. But then as he progressed in his Career, technology started improving.

And he just really, really he was a lifelong learner, which is something I took from him, You know, I took it to heart really and I feel like I’ve kind of done this in my career, my crew as well as like, you know. Whatever the technology comes I learn it so despite the fact that he really you know we would be in meetings and he’s talking about these programming concepts.

He’s talked about this data base issues he in Then towards the end of his career He even became so qualified that they became an adjunct like i said at the neighboring, um, you, know college of information and sciences and technology so So anyways, maybe another guy, a couple other people, Anthony Robinson, he’s amazing guy.

He’s head of their now the Penn State online, not only as a professor, but online learning. Was it master’s in GIS for online program? And then Brian Tomoshevsky, another, I feel like I don’t know, Maybe I shouldn’t have named names because now there’s so many I’m going to leave out.

I can’t forget about my advisor for my PhD, Prasandra Mitra. So this is kind of another thing. He came from the opposite way. So his degrees were in computer science. He got his PhD from Stanford in either computer science or computer engineering, complete. You know, what do you say? Like a tech geek, and then he worked for Oracle for a while. But then he so he came from other perspective that then He learned so much about some of these geospatial issues and now he’s like, you know, writing papers and all about geospecial problems So it’s this is what I was telling you about.

This multi -disciplinary environment was really wonderful. I left there with mixed emotions. I basically left there for to marry my my now wife – and hence why I’m in Kuwait. This environment really set me up that now I just want to continue challenging myself, trying to solve real world problems.

Scott 18:16
So you know Helena, You mentioned health, and technology towards health. So there was two, let’s see, two maybe even three projects while I was at the GeoVista Center. So we had one for the centers, the CDC, Centers for Disease Control and Prevention.

We built for them both the Pennsylvania Cancer Atlas and the – in turn, it was like the U .S. Cancer – we modified it, US Cancer Atlas. So this is like an online – AB, I mentioned to you about the action script in Flash. So it’s built in flash and action scripts. This was that time.

How do you present the data – amounts of cancer rates, but then also all of the demographic data? How do you present it in a way to a user that they can understand it, they can filter it they, can search it and, and try to gain knowledge from, from the visualization?

Then we had another one, it was called Health Geojunction.

This was taking large amounts of online media, news articles. So basically like news, articles, extracting place mentions, key topics, what are the topics and trying to then map. map. So doing some geo-parsing, basically.

We’ll extract the place names, put those place names on a map, and then providing, once again, a visual analytics system to let people visualize where are the diseases happening, where the connections going on between various diseases, the placements that are co-occurring.

And then finally too, there’s one other, so a project I worked on for quite a long time, it was called Senseplace 2 & Senseplace 3. So this was a really interesting part. It actually, it really successful.

Recently we won this award for one of the paper from IEEE. We wrote a paper. It was amazing that we got recognized – it was called the Test of Time award.

It’s 10 years since we wrote this paper about our research. It was kind of like whoa, I can’t believe it has been 10 years. But basically, this project, there was kinda like it stood the Test of Time.

It gets cited a lot, and it generated a lot of discussion. So what was about kind have a similar concept about analyzing text.

So at that time, Twitter was getting really, really popular for things like people would talk about all of their societal problems on Twitter. We built a system that was really designed to extract meaning from this incredible amount of Twitter data you would get. We focused on crisis management, but then really it could apply to almost any domain.

Scott 21:36
It can even apply to businesses these days, like a large business wants to analyze, okay, our customers in this area, how are they responding to our products? How are their responding our competitors?

Things like this. So it was kind of interesting. So basically, given my background, multiple projects focused on health, basically technology towards public health, you know, improving public health outcomes.

So then recently too, I got a project, I was working with the World Health Organization. So this was once again, so this was some folks based in the African office. Congo is the regional office, so basically their, what do you say, domain is the vast majority of the African continent, it, excluding like some of the countries in the north, Egypt, excluding Egypt Libya, a couple of countries.

They had this data set, I forget exactly what it but basically, it’s a case. Oh, yes, yeah, it is. It’s comprehensive. comprehensive way. When when you hear data scientists saying like good, you know, you really need good data.

They had good data, so this is a really good good thing. So it’s basically like suspected case counts and death counts for almost all of the diseases that the infectious diseases that go on in the continent for every week and for every district.

So not at the country level, not at state level but at district level. Kind of equivalent of a U .S. county. And so we did some exploratory analysis and we, you know, we did spatial autocorrelation. One thing was, like, we were able to find different, all of them had a strong spatial correlation.

So it’s not really not, it is not hard to kind of like think about this, that if you have cases in your neighboring district, there’s a good chance it will affect you. But what we were able to do too, is quantify it.

For the different diseases, there is a different amount. And so we’re quantifying it, quantify the differences for diseases. But then also too after you do this, you get to local spatial or auto correlation.

When you’re getting your hotspots, your cold spots, your outliers. If you’ve got a cold spot, but then you got high district among them. So it was really, really neat kind of work. I don’t want to talk about this too much because we got a paper still in review.

AB 24:30
Well, that’s fair. We can’t talk about something that hasn’t quite hit the airwaves yet. But just to roll back a second. So everyone knows at the last few years, the dreaded R value, you know, the COVID years have taught us to watch that number like a hawk.

Was this research actually trying to correlate not the virology anyway, how bad the viruses were, but the speed and the density and trying to actually put time and space to that R value, was that kind of the focus of that research?

Scott
One focus was the spatial distribution patterns. The factor is that we chose the model. For example, we choose precipitation, temperature, population density. There’s a couple other ones and mainly because like for one, they, oh, what is the population near a water body?

Scott 25:28
Because water bodies are very important for disease transmission, either directly or indirectly. So then, so what are the most important factors for the various disease, for this spread of the diseases?

What are the most important factors? Then the other thing is making a prediction. So making up prediction on cases, based upon these criteria, these factors, precipitation, temperature – all of these… will your district have cases or not?

Scott 26:07
So this was the thing basically making a prediction. We were able to do really quite good. I’ll have to be honest with actually, predicting the number of cases, we didn’t do so well. This is like, you know, if you’re familiar with machine learning, this kind of like the, it’s called regression. So predicting actual number cases. It’s a little challenge. And to honest too, I mean, we’re doing this over, obviously the African continent’s vast, you know, vast.

It’s really not surprising. But when we turn this into a machine learning classification problem – are there cases or not? – then we started getting really good success.

It’s a little bit setting up the machine -learning problem a little differently, than we were able to get really success, basically in the goal of, So, if you’re able to predict cases or not, kind of at that local level, but over a large geographic scale, and on a weekly basis, in a matter of milliseconds, then you are successful at this, it will be really powerful.

AB 27:24
That answers the question I was just going to ask. Was this work to actually try and get close to real time and be of use tomorrow, the next week? Or was this one of those deeper questions of let’s look at previous years data and make broad sweeping recommendations for next year’s possible viruses?

Was it short, sharp and in this case binary, or was it that long running question that you had time up your sleeve?

Scott
Yeah. So the model we built, I mean, it’s more about the near term. Also too, when you do machine learning and you create a model, if you want that model to predict a long time in the future. I mean, are things change or environment change, things like this.

So you kind of got to update the model as you go. You got a retrain and all. So basically it was intended. The model that I built, it was like, you know, say, for example, if you have if you have OK, next week, it’s going to be raining in this rain prediction.

You know you have all these variables, you have a prediction, then you’ll be able to predict case counts. And basically, with a little bit more work, I wouldn’t see any reason why it wouldn’t be successful like a month in advance, two months in events.

AB 28:35
It feels like the global weather forecasting trend, by means every nation has their own body that looks at the next five days, seven days. There is a growing trend of worldwide weather models and look, they’re not always better, but sometimes they are surprising.

But it’s just that concept of extrapolating up from a large area to the world used to be impossible. We definitely won’t be doing that, but that’s now becoming potentially tangible. So yes, when you’ve got a graph database, a chart of the whole world, potentially one day there might be someone down the track who decides, “hey, let’s go for broke and do the whole world in one fell swoop”.

Scott
You touched upon the value of our work. I mean, it’s not like what we were doing was really groundbreaking in itself to predict disease, but the fact that you could do it at scale and in fractions of a second, I mean this is what’s really kind of like the key point to it.

And it is not just simply like a prediction, and it isn’t just, oh, we’re going to predict this, what you can then do is give a prediction but do some modeling.

Helena 29:49
So just to clarify a little bit, Scott, because I guess not everyone listening to the podcast come from the geospatial background, since we’ve kind of got also the scale on a different scale when we’re talking about the indoor navigation, robotics, the architect, and so on.

So from a geospecial perspective, we have obviously got those different data layers that you’re talking about. We’ve got the geography, so we’ve got water bodies, and we have the human factors, we got different populations, and the layers with the education level.

So we have all these different layers that we’re layering on top of each other for a typical geospatial analysis. Then you are bringing in the AI component. And so, can you explain a little bit maybe about where kind of geospatial starts and ends in the AI takes over?

Like, is it that you’re basically, you can, your modeling the as is with typical geospatial methods, and then the machine learning AI does the predicting, or is there not a clean cut like that? Or how do they play together to go beyond what’s possible with like, pure geospatial capabilities?

Scott 31:09
Yeah, so this is an excellent, excellent question. So I’ll kind of talk a little bit about the current state, but then I will kind of mention where at least some people think things are going. So right now, I mean, most of the, like when we do geo AI, we love to stick on this geo in front of everything.

So, maybe it’s like lack of better term, ‘Geo, AI’. I don’t really like it, but I don’t know what else to call it, but in the end, the AI model is the same as everybody else. We’re using RNNs, CNNs. All of this stuff. It’s all the same.

Also too, for some of the challenges like computer vision, doing machine learning on satellite imagery, it’s basically some of the same models that you’re going to do computer vision on on regular pictures, really not too different.

So for my all kind of comment about the project, you know, this particular recent project is so for the geospatial part, I used a lot of geosphere spatial part for pre processing for. For one, for one the spatial autocorrelation, the special hotspot analysis, Because to be honest, that did lead into us choosing some of the factors that we did choose.

Also too, for example, I mentioned, what’s the population near water bodies? So that, you kind of did like some, your standard way back buffers and overlays, overlaying, getting the precipitation, but then bringing in the waterbodies from open street map, doing a buffer around them and making some calculations.

In the end though, all of this sort of geospatial processing, in the end I had kind of like a structured table. The result of this was a structure table, so it’s like basically for every district for every week, we had the number of cases, the amount of the average precipitation, the average temperature.

So it’s like basically a Excel table kind of in a sense. And so then the AI came in, I treated it basically as a tabular type of AI model. So one thing I also considered, but we weren’t able… I didn’t try this. I wanted to try this. So I treated this particular problem as, you know, like I said, trying to get things into a structured table treated as that type of machine learning problem. But in my mind, if some of the data… No, actually, it went down to the disease data, the actual disease. This was aggregated by district. However, in my mind, if we had a little bit finer spatial resolution, similar to the temperature data, the temperature datasets precipitation, this was basically from your NASA, you know, satellite data.

Scott 34:29
If we have the case counts, but on this kind of like resolution or similar, then you could actually treat it as a computer vision problem. So this is kind like, I’m trying to to describe for this particular project, how the geospatial and AI fit in.

Now, kind of like for the future, so I’ve heard a lot of talk in the kind of geospaceal, geo AI is getting like AI models that inherently no space. So for my, once again, my particular case, in being in this structured table, it really, maybe the model could learn Um, this particular district is near this, but, but it wasn’t very clear. If I would, if I were to treat it as a computer vision issue, uh, a computer version problem, maybe it would have been a little bit more spatial where the model is able to know the, you know, the pixels around it at all. But, but I think what, where some of the AI models are going is being able to inherently know geographic relationships. relationships, not only nearness, but some of the other geographic relationships that all of people in the geospatial…

Going back to what is it, the first law of geography, which is… Even for this WHO project, it was really… It applied when I told you about spatial autocorrelation is that, you know:

Paraphrasing the first law of geography is that everything is related to everything else in the world, but near things are more related to you.

Scott 36:14
And so if we have AI models that kind of like, account for this, this natural, then I feel like they’ll be able to solve a lot of problems.

So doing it the way you were using the GIS world to prepare your data for a regular 2D data set and that gave you the results and the tools and leavers for your clients to pull to be able to test and use different scenarios and what could you call that a good 90 -something percent, excuse me, a 90 something percent answer which gave people confidence, the ability to play.

AB 36:58
You’re thinking ahead to different modes, different ways to increase, you know, one more signal level, one more level of confidence in a data set to incorporate richer data, more variables. Is that all on the drawing board? Is it a wish list? is that you’re going to pass the can to someone else in 10, 20 years and they’re going solve it. How soon can you see these sort of problems going from that regional level down to the next level of cell?

Scott
And another thing just too, which was valuable from this machine learning. So machine learnings, many of the models, they produce what’s called feature importance. So how important was that particular factor in to making its prediction?

So this was another thing that that we were kind of so it’s not it doesn’t always so it basically you got to keep in mind that this was for the machine learning model, not necessarily the real world.

But it gives you really, really good hints at what are the most important factors for making that, you know, out of rainfall temperature, it quantifies the importance. So that, once again, gives you a lot of information on a more knowledge on what you try to tackle.

What are the factors you really try to address first?

Helena 38:21
I’m also really curious, Scott, about how soon you think that the geospatial, sorry, the AI model that knows space or that understands geographic space, How far away is that, would you say?

AB
Go on, make a prediction. This will be in big bold text on the episode post : )

Scott
I don’t know, please! No, no, I’m not going to pin myself any, uh, I dunno, when things are going, all I can say is probably faster than I think, actually.

Helena
And what, what problems still need to be solved for that to become a reality? So I know we’ve talked on the podcast in the past about like the issues with AI and space in general, But in the context of the geographic space, I would think there’s a lot of specific rule sets and so on.

But what are the hurdles that remain until there is a model that can actually do that? What’s the big missing piece?

Scott 39:19
To be honest, I don’t know. I feel like I can’t answer that question. So basically, this might be a question for, so while I was at Penn State, I was also around some experts in what you say, like spatial cognition.

How do people think spatially? How do you people understand their place, where they’re located? Things like directions, wayfinding, things like this. So I would say you get those people and you let them talk to the actual people building the algorithms, and then you can find sort of the missing piece maybe.

Helena
Yeah, it’s a really hard one, isn’t it? I mean, I know I focused a little bit on my research about formalizing place, you know, as in like how humans experience space being like places. We don’t think in coordinates.

In terms of places, we think, in terms of neighborhoods. We think of, kind of personal reference spaces and that’s really the incredibly hard concept to turn into zeros and ones as you need it to be able to kind of um teach it to a GIS system for example.

So it’s just one of those big problems where personally I always thought without knowing so much about how the AI works in that sense that potentially large language models might give us a new avenue to explore that concept.

You know you mentioned before with the Twitter, the sentiment analysis, I don’t know if that kind of large scale language analysis of place sentiment could potentially be an avenue that would help formalize the concept. Do you have thoughts on that?

Scott 41:09
Yeah, you’re really, so like I said, one gentleman, Alex Klippel, now he’s at a university City in Germany. He was originally from Germany, he came, he was my colleague for a long time at Penn State. Now he’s back in Germany. I think now though, it’s kind of like getting into virtual reality. But he had a lot of this, you know, kind like what is place also to some other folks that came through. Brian Tomaszewski, now he is at Rochester Institute of Technology.

So when we went, yeah, exactly this when we were doing this sense place, that’s the name actually, the name kind of came about. Actually, Brian Tomaszewski was the one who kind of coined it, sense place.

And then we he did SensePlace 1. And we, I mean, together with us, but then it was like, then we built sense, place two, since place three. But it’s kind of like we had this, that was one of the overarching things, I’m not sure we ever really kind of, like answered any questions towards that.

But this was, one thing is that if you get a whole lot of text, and people are talking about places, they’re talking about other things, and you’re starting to get in natural language processing, sentiment analysis, emotion analysis.

Some of the other natural light language, processing concepts, people mentions, you know, named entity recognition. So people mentioned disease mentions you know whatever the name density is. You we always kind of like had in the background minds, this is a really good opportunity to kind of like try to conceptualize exactly what you’re talking about.

How do people define plays? And also too, like, you know, we mentioned this earlier, a lot of the research in this area is at a really, really fine level, doing things like user studies, a very narrow population.

Scott
But can you scale this up? So can you scaleless to the world, some of those ideas. So this was something, yeah, that was… We talked about it a lot. I’m not sure we really kind of tackled it directly, but in my mind, this is a really good opportunity to use big data to try to understand how people actually conceptualize place.

Helena 43:30
Yeah, absolutely. And I think it’s also interesting that you mentioned scaling it out to a world level. I think that there’s a whole lot of other issues wrapped up in there that, you know, possibly people don’t really think about, like, the different kind of cultural impacts of language.

I don’t know. Perhaps, I mean, if you’re living in a different culture, I’m sure you’ve observed that people talk about things quite differently in different cultures, like places. The concept of place is very different across different cultures.

And so what happens when we analyze these, you know, big kind of English texts is quite easily that we kind paint a very Anglo centric world view of what place looks like without really kind recognizing that cultural nuance.

And I think there’s always that danger. And potentially with like this kind of large scale analysis would even have to look across different languages, across, you know, different cultures and really create something that brings in more than just the English.

Scott 44:43
It reminds me of another project we had. So this particular project was focused on wayfinding. And actually, I think Alex Klippel was on this project. So he was really interested in way finding.

How do people get to where they want to be? How do they, you know, kind of like do it in their minds. So how, for example, in the US, we have we have our MapQuest route directions. And it says go go on this street, take a right, you know.

Go down this distance, then take it right. Sometimes we use landmarks. But then, then like I said, Prasadja Mitra was on his project, he’s originally from India. He’s like, yeah, We kind don’t do that there.

You know? He was kind like it’s all about landmarks, you know, it is all about landmark there. So, how people, even for a task like this, this is just a small part of place, wayfinding, is extremely different depending on where you live in the world.

In Kuwait too, they have street addresses, but honestly they don’t use them much. So they have like a actually it’s really I like their address system. It’s really kind of intuitive. So, they, have although honestly it is a small country, so it, is it a little bit easier to manage.

SBut, basically they have the, they have Kuwait City, the areas. So Kuwait city, The different kind of suburbs, Kefaun, Haldia, but so, so basically you say, you would say something like I live in Kefaun and then Kifaun is divided up into block.

Every area is divided into blocks. So I go to Kefaun, I lived in block 2, and I’ve lived on street 64, and live on house 2. So this is say your address, kind of. You wouldn’t say it kind like a US way.

But honestly too, it’s really, and then the street signs, all of the areas you’re trying to put a little map on. You’re drawing on this road, lock two, here’s block one. It’s kind of like, I think they did a freaking job with that. They lit out the roads too actually, the highway system really well, because it is kind like you’ve got Queen City, you got First Ring Road, then you got Second Ring. Then you have Third Ring road and it kind just goes out.

And then also too you’ve got like these spoke highways going out which is like what is it 30, 40, 45, 50. It’s really kind of intuitive but this is all of this kind of stuff is, like you said, it’s like really, really different depending on where you were, where are you in the world and one.

Scott 47:34
So one thing I learned from my really really smart colleague at Penn State was everybody everybody plays differently you know like there’s so much variability in how you you kind of like are you a spatial person?

I gave that example before about like if I don’t know the direction I’m facing I feel like uneasy because I am a very very spatial person.

– I used to play a lot of golf, and I remember playing one particular golf course where there was like two holes and a very windy way through the trees and you couldn’t see the others a lot of times. You can see other holes but I never played it before and I didn’t know what direction I think and really uneasy the whole time – I felt was really kind of like something just here because I don’t know where I am I also have to know where am and what direct which I’m really I feel like I really good at.

When I hear my father-in-law, he’s like, “how is he even going to get around Kuwait City”? It’s really no problem for me.

AB 48:48
Very nice. Yes, Scott, I do share that same point of view and I wonder actually if that’s common or it’s uncommon. I generally know which direction I’m facing vaguely and it is really nice to be able to if I am ever riding or driving somewhere.

Going somewhere for the first time is a regular normal challenge and the Melbourne thing is which car lane do you have to be in five Ks before you get there? That’s our normal problem. But I’m always fine the second time, which is always a nice sort of vibe.

But I must give a shout out to my cousins who were sailors. I can tell you right now I am kind of facing South – but my cousins were crazy wild: they knew they were facing “south, south west”. And it’s like, okay, that’s a level of a distinction that is a super power that you need to try and find a use for!

Scott 49:33
I’m much more reliant on Google maps. It was like before it was, like once I drive somewhere, boom, I know it all, you know, I would drive from Carolina, from Pennsylvania, South Carolina, no problem. I want to drive 10 hours. I don’t even need to, you don’t know directions. I know where all of the basically the turns – but now with Google Maps, I feel like it’s taking me longer to know the route and I am using maps longer until I actually know to go places.

So I think like I’ve a little bit too reliant on the technology. Maybe yeah,

Helena
That definitely happens!

AB
Scott thank you so much for your time. Absolute pleasure to have this long chat with you. It’s been fantastic…

Look, your application for Unpaid Intern at a SPAITIAL has been accepted, but we’ll have to get back in touch with you whether you can fill the role… But your qualifications… only just pass. To be honest : )

Scott
Ha! I love talking about this kind of stuff. I like talking this stuff, and also too, it’s I learned that you’re doing… You touch the part, or you are a little bit more focused on the spatial part versus I’m little bit more geospatial.

But honestly, I feel like what you’re doing here with SPAITIAL, combined technology, combined with AI, and I mean, I love talking about the kind of stuff. And also, another one of my, what do you say?

You said about something ‘a touch controversial’ earlier. This is one of mine controversial things, is that I think in the geospatial world, we play up the importance of the geo-spatial part a little bit too much, I feel, because I think like so many of those problems could actually be solved if you treat it as Cartesian space.

You know, instead of like having to deal with the curvature of Earth, all this stuff, maybe this is something like the geospatial people, they want to, you know sometimes a little more and a lot, don’t get me wrong, in a a lot of cases is really, really important.

AB
Oh, for sure. There’s always the purists versus the pragmatics.

Helena
And then there’s always a ‘spatial specialist’, isn’t it? You guys heard of that? The spatial specialist. It’s kind a of a geospatial thing.

Scott
I remember when when Google Maps came along, and I was you know, I was in in geography in all and oh my god, how could they possibly use Mercator Project? Oh my gosh, like, you know, I mean, I understand Mercators Projection has, has its problems, but you know because they were able to treat, basically treat the world as X, Y, they, were doing able, to do so many amazing things because they simplify the problem and then they’re able to, do some amazing, so this is kind of just one example, I feel like.

Helena
And the vast majority of Google Maps users probably don’t even know. Yeah. It does. I think most users of of google maps, they have no idea about projections anyway, so it’s really kind of, you know, back to what we were saying before, it solve a problem for your user and they don t care about, projections that needs to be abstracted under the hood, which Google maps has done a fantastic job of.

Scott
Most of the time when you’re using Google apps, I mean, you are looking kind of like local, so you aren’t really even going to notice the difference. I hope I didn’t upset any of my former colleagues when I made that comment.

AB
I tell my teams all the time that there are no Oscars or Emmys for coding. Yes, there probably are a couple of awards in deep dark places on the internet. That’s not to be a cop-out – but if it works and it’s reliable and you can sleep at night and it’s elegant, simple, and does the job… the job done.

We can solve some gnarly problems with a lot of code and as long as we’re okay with it, that is the job. So it is great to hear that pragmatic sense coming through.

Scott
You and I talked about this a little bit yesterday. Both of us kind of like having coding in our background. It’s like no matter what, you’re going to have some issues. So you kind of got to like pick your battles.

So what are the really, really major problems? Am I able sacrifice speed? Am able the sacrifice something just isn’t quite vital, but I am able solve a bigger problem?

AB
Brilliant. Well, Scott, thank you so much for your time. We’re going to stay in touch. And by all means, we’re gonna come back when that paper of yours is announced and ask a few more questions about it if that’s all right.

I know we are leaving the audience hanging. Come back for the next exciting episode of “will Scott be able to talk about his forthcoming paper?”

Bye!

Scott
Masa lama.

To absent friends.

The post Episode 010 – Geo AI in Public Health with Dr Scott Pezanowski appeared first on SPAITIAL.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

Welcome to SPAITIAL. This episode is a deep dive all on its own. This week we’re covering the rather gnarly but important topic of how do you get started with Spatial AI?

The field is changing so rapidly. If you’re already on the leading edge, then congratulations, The burning question of today is how do you get there? How do get close? Close to – I won’t say leading-edge – close to watching the edge, but then a deeper dive. How does someone fast follow? How do you catch up? Open question! Who’d like to go first? I can see a lot of people with a lot experience here in this chat.

AB 01:10
Violet: could you give us a bit of a brain dump of your experience of watching this field and then go around again for catching up? Go for it.

Violet 01:19
Sounds great. I kind of stumbled into this field because I started in architecture which has a lot of 3D stuff. It has a lotta modeling and simulation. But then ended up getting really interested in the way we interact with physical spaces.

So I wanted to get out of the digital simulation world. I had been at Google on this product for generative design. And so that was all simulation-based. But I was really interested in sensors and the tangible world and physical computing.

So I chased after that really, Creating a new class for that so that I could start to learn more about it. And so I guess my biggest advice for like making the jump into something like Spatial AI is I think you need to find what you’re really interested in and carve out that niche for yourself by just starting to learn more about it, work on the tools.

Like for me, when I started getting into this, I don’t think I had much upper hand other than the fact that I had been working in the simulation space, but I didn’t really understand sensors or how to merge the physical world really other than any experience I have from fabricating things in architecture.

So, I think the best way to ramp up is just to start playing with the tools. So for us, it’s spending a lot of time having ideas about specific projects. “Oh, could we start to use something like OpenAI to have more natural interactions?”

“Oh let me play with this gesture library and see if I can combine that with speech.” And so I think you have to allow yourself a little bit of freedom to be a novice. There are so many people being novices trying to learn how these things behave.

And we have really just poke and prod at the tools and what talks to what and what does and what emergent properties does it have. So I think you just kind of have to chase what you want to do and get muddy.

So I think you just kind of have to chase what you want to do and get muddy.

AB 03:45
Get muddy indeed. Helena you’re definitely possibly in that muddy field you’ve been playing in the geospatial world. Not only have you been able to go ahead and do some deep research but you also then take your clients through that kind of journey.

How do you sort of get them pushed up that hill without – you know – dragging them up the hill. How do you do it politely?

Helena 04:08
So it’s a bit interesting, because I’m actually quite the opposite approach of what Violet’s just outlined with playing with the tools and kind of taking that technology-first approach as a kind of a front to what my personal philosophy is always a problem-first approach.

So, for me, it’s less about the technology, it is less the tools, It’s more about what problem are we trying to solve.

So, for me, it’s less about the technology, it is less the tools, It’s more about what problem are we trying to solve. And that’s always what’s at the heart of the conversation I have with clients when they come to me and they say, like, “Hey, I’ve got this really cool idea.”

And I got to say 9 out of 10 client conversations that I had these days are people or startups – like individuals, smaller companies, or larger companies who are kind of pivoting or adding a sector who are going into AI.

I’ve only really had one conversation lately where it’s been about branding that hasn’t been somehow related to AI, so people are seeing the opportunities everywhere. People from various sectors, like AI in education, AI and street lighting, AI this, that, and the other thing are just from this week alone conversations I have had.

And, you know, it’s always really nice and easy to get hung up in all the cool stuff that you can do. But the way I tend to handle it is to pull it back and say like, okay, I know there’s so much cool you stuff you could do, and there is so many potential in there about what problem are we addressing, what problems are solving, because otherwise we fall into this, in the geospatial world.

It’s been described as like the geospecial product trap where you create an amazing product but no one really needs it or wants it. And I suppose some innovations come about that way. And it definitely has its playground. But when you want to make money with it, when want build a career off it when you wanna build the brand off of it then you need to be super clear a problem you’re actually addressing. So that’s where I start.

So we really drill down into what’s the need and how can you address the needs and how could AI come in to help you address that need more efficiently. And most of these business ideas or kind of products are based around the AI component of then addressing a need and more efficient.

And that’s what we then drill down into and say, okay, what do we need to consider? You know, what are the hesitations in the audience? What’s the objections we to overcome? What is it that people are scared of? What are people’s preconceptions where they think they are an AI could never replace this or that certain thing, or you know being worried that it’s going to take their job or on the other hand being like wanting it to do too much.

Helena 06:58
I’ve also had that conversation this week where someone said like then you the AI people who he’s gotten on board to help create his product idea, they want to integrate far too many AI stuff and actually completely take away from the experts who are also involved and so it is really then about how do you have those conversations with the different stakeholders on the team to ensure a balance of kind of the human component, the AI component.

What problem are you addressing? How can you make sure that AI is staying in its lane and the expert is saying in their lane, and bringing it together to create something that’s going to actually drive humanity forward? Does that even make sense? I don’t know.

AB 07:42
Oh, it does. I’m hearing the same theme, even though different ends of the stick potentially I’m hearing balance and awareness and play and by the sound of it ‘knowledge breeds knowledge’ and perhaps both of those are in that same bucket.

Mirek for you, how have you sort of gone to getting to where you are now with background skills but also how would you encourage someone to fast follow if they wanted to, yes: catch up/overtake – what are the the tools and techniques that you’ve found the most useful to you?

Mirek 08:24
I don’t know if anybody would find it interesting, but I agree with both Helena and Violet that you need to look for a problem, what also look at the type and sort of try to merge these things together.

My work is on the intersection of augmented reality and the and human -mission interaction and robotics. There’s quite a lot in all these fields to sort of learn that you need to understand, or you to have a buddy or a teammate who understands that kind of thing deeply.

There’s quite a lot in all these fields to sort of learn that you need to understand, or you to have a buddy or a teammate who understands that kind of thing deeply.

And so I started in AR before it even existed. You couldn’t get HoloLens even. It was this thing that’s coming next year or the year after. I still wanted to get into the field, So I just took whatever was available and I hacked it and modified it, and as long as I needed to have something I could work with, it’s a painful process, it is not fun, most of it you don’t see results immediately, it can take a really long time to actually modify something, or I have a platform that you’ve created, especially when you’re approaching it by knowing nothing and, you know, having to learn everything as you go.

AR or VR, that’s no longer the case, you can just grab Oculus and your golden, you can just, you know get that for a few hundred bucks, and Unity, and then you can start prototyping and you can learn from others and see what library is available.

Your mileage may vary. It depends on what you doing and whether it’s already, whether the components are already out there. If they’re not, you have to get your hands dirty in any case. Or you might end up with something that has been done hundreds of times already and nobody cares.

If you just take whatever is available and then mesh it with other things, you’re most likely not creating anything unique. I think for that you really need that deep technical knowledge to be able to actually both understand the system and able to sort of imagine what you can do with that.

In robotics, the learning curve is crazy steep. You have to have some general understanding of electronics and how these things works. If you want to get into that, get an Arduino, that’s the fastest path to like get some really basic knowledge and then build it from there.

If want be playing with these AI systems, I reckon, and I didn’t get that far yet, I still have to sort of, you know, catch up on that. Everything is ROS based in this industry, that’s my understanding.

Mirek 11:18
And ROS is this (it stands for robotic operating system) – it’s essentially a messaging system, so you have multiple nodes running on multiple pieces of hardware, talking to each other. one drives your motor, one reads your sensory data and something else and meshes it together and does the decision -making.

If you want AI plugged into that, you’d better understand this environment really well and that may take a long time really. It’s the roast in particular is like after a week you you kind of get the idea how it works like you know the overall principles but then six months later you’re kind like still there and you feel like what you are doing but just be really it takes a lot of patience and a bit of determination to sort of understand these concepts to be able to then innovate on them because that I assume is what do you want to do, you want build something that isn’t out there.

And if you just take things that kind of exist and put them together, maybe it’s not as unique as you’d like to. So my recommendation is just equip yourself with a lot of patience and just set out a time to learn new things because you need that.

AB 12:37
Yeah, I’m hearing a thread emerging. I’ll wrap it up in a tick, but I am definitely hearing thread. I will put that in the parking lot, put it into any other business metaphor you want. William, to you though, your journey and then your way forward? Can you sort of chart as a here to there kind of pathway?

William 13:00
Oh, my journey – it’s been an interesting one. Perhaps I’ll say serendipitous. I’ve been tinkering with electronics and writing codes since I was probably six years old. And when I went to college and grad school, I wanted to do something that was simultaneously technical, but also creative and also tangible.

And so that led me to architecture. And you quickly realize there isn’t actually that much technical in it, at least when when went school. But we saw the emergence of that technology and architecture is always a bit behind the curve on adopting particular types of technology, which wasn’t the case in the early days of what we know as the modern profession, say the late 19th century and other periods where architects were actually inventing new means of drawing, new needs of communicating and representing space and translating that into the real world, things like projected, prospective projection and descriptive geometry, things that.
But I did not like the profession itself. I was more enamored with things like design techniques that were emerging from other fields, and that we’re finally integrated into architecture in the early 2000s, things Like evolutionary design.

William 14:32
And so my thesis involved writing a genetic algorithm to optimize acoustic reflectors, things like that, and there was a small cohort of us who really love this idea of writing code to generate form and design buildings, so how could you start with something that was relatively primitive or random and then evolve it into something meaningful?

Then outside of school, after I graduated, I ended up as an academic for some time and so taught structural engineering and also tried to incorporate some of these newer techniques emerging like what we call parametric design, so tools that take a form and essentially extract the numbers out of it and then you can play with the number and see what that results in.

And so using things like finite element method to analyze structures that you might generate, and you kind of tweak the numbers again, you get a different shape and then you analyze it again and iteratively do that.

But then I ended up sort of going back and forth between something related to space and something related to AI or technology in general. I ended up at a friend’s startup that was effectively a machine learning and data science startup where we were building a platform for a unique programming language that would allow a data scientist to write small chunks of code but execute them at scale on the cloud.

Then I was a product lead at a company called Florida, the head of product there, where we engaged with VR. And so that was like, marketing experiences in the browser for real estate brokers, Where we had an Oculus and where we, had an experience there.

William 16:40
This is before my time, but they built an experience with the Oculus that would allow the largest development in North America at the time which was the Hudson Yards in New York City. They allowed the developer to lease the retail space by having an Oculus headset and having them walk around the mall and essentially pick which retail spot slots that they wanted.

So that was rather remarkable, I think, at the time. One of the products I worked on and helped to build was a generative floor layout product that also leveraged the 3D technology we had at the time. And then I went into cloud computing, and then went in to sustainability tech. And now here I am again recombining space in AI. Now where AI is actually these advanced capabilities of artificial intelligence are far more popular and far accessible.

Just two years ago, things like BERT and GPT-3 were really niche developments and you had to be in the circle to know about them and really appreciate them.

AB
And you could run them yourself.

William
Yes, totally. Now that these large language models can demonstrate this reasoning capability, I think it was just the right moment again to take my fascination with space and design and combine it with AI and just ask the question, and like, can we do this?

Can we can do something meaningful and what’s the potential of it? And so the half of me that’s the product manager from cloud computing and from real estate tech is saying, yeah, we need the right problem to solve.

Don’t just throw a chat on it. But then the researcher designer technologist side of me is like okay, but we we need to incubate this in some way and provide some wisdom so we can explore. And finally, the one piece of advice I would have is like you everyone needs a project that they’re passionate about to lead them into something like this which is really complex and really technically grounded and but also has a lot of potential when I was in architecture school there was no there was very little time you had payoff with the work you were doing you would you would work on a design for weeks and weeks on end only to be eviscerated in your jury But with programming, every few minutes you could have a win.

Something worked, and it was magical, and you can see it happening, and so that journey, I think, is really important when you’re trying to deal with something new.

It’s like you have to enjoy the work itself, and that has to motivate you, and the big payoff at the end of you sold to the client, you shipped the product, or whatever, is part of it, but you need the project so that when you get up in the morning, you really look forward to sitting down and struggling through the hard parts.

AB 19:44
I hear you big time. If I can summarize a couple of those things and we might wrap in five, is that spatial AI is definitely a niche field within the fields that we generally play in. Spatial computing is obviously 3D and has all the different complications of the 3D modeling world and environments and deeper, denser data in more dimensions, literally.

So spatial computing versus regular computing: data science is rough enough and a great learning curve. It can be tables, it can 2D data, but then it gets to 3D and then time-based, well, 4D, whatever you wish.

It makes it rougher, so that is a steeper learning curve or you’re combining fields there to even start with. The geospatial world, if it’s 2D – it is great: there’s a lot of learning and there’s a, you know, it’s its own field and has been for hundreds of, well, for 100 years.

But to go into third dimension, suddenly you’re playing with 3D tiles and elevations and trying to do data science on, slopes and falls and things like that. It does make it harder to go in to a third-dimension, but that’s probably the defining factor that all of us can either visualize or contend with or our brains are kind of okay to it.

I’d probably suggest that those who have played games since they were little kids, probably are very familiar with navigating 3D worlds at pace. That’s probably been a helpful tool rather than a hindrance. So there you go parents, let your kids play games.

The thing that I’m getting, amongst all of us, is that probably being self-aware is probably the number one thing to sort of get ahead the fastest.

We’ve talked about playing and iterating. We’re talking about as a customer or as person trying to find ways to how can I do things and to honor Mr. Rumsfeld who did the classic known unknowns quadrant thing. You don’t know what you know.

If you can’t ask for things that you don’t know about, so you’ve got to go in baby steps to ask questions, then that leads to more questions. Your vocabulary gets larger as you go down the path. So just starting is probably the number one lesson we could probably say.

I’m also hearing that having a buddy, having someone to bounce ideas off is pretty wild.

Either your code gives you the feedback that it worked is good, that’s a good buddy. Having a real buddy to push you along is fantastic. Having someone to bounce ideas off is quite brilliant. Mirek, I’m quite fascinated by a subtopic that we’ll probably come back to is, are you your own customer or do you see a need and then the build it and they will come and try to be your customer. Is that good enough in 2024?

Mirek 22:23
I think you have to start somewhere and William actually put it put it nicely. Serendipity is quite important in this process. So you’re digging into into something that you find interesting. Engineers like to engineer and you know, understand things and thinker and build stuff.

Sometimes you cross a whole new field that you had no idea existed before you started. And it’s gonna take you ages to become an expert on that. So it helps to have somebody find somebody who can help you with, you know, solid block of things to do that need to be done, or you just need to teach yourself how to do it.

I tend to do it the later way, I like to understand things deeply, that takes a lot of time. them. So I start by, you know, usually the process is like, here’s a crazy idea. I need some peer approval for that.

Just tell me I’m not going crazy, because then I am going to spend a year actually building that thing. So it helps to bounce ideas of people. And sometimes you really hit a wall and you need to talk to people who’ve been there before, in this particular field and to help you with This is not how we do it, you just take that and this is how you do it.

You had it all backwards, that sort of thing, because oftentimes you’re stuck in your own head and digging in the wrong direction and it helps to talk to people. Also, the process leads to, in my case, I just build a product by just digging into ROS and robotics and I discovered something that’s really difficult to do.

And I built a thing, tool for that, that I’m using myself. I am going to be releasing that as an open source. That’s something I didn’t plan for at all. It just happened because I, there’s a robot behind me that have been using.

I bought that one, learn about, you know, robotics and all this, but now I modifying it. And there is a bit of hardware. I’ll break you on that. Our listeners definitely won’t appreciate. This is like that allows it to go up and down.

Yeah, but this is where you need to get because that robot is now too small and limiting who I needed to do. I need to some physical modifications of that and that takes you to whole, you know, other places but it’s a slight diversion but serendipity is an interesting property of this of this process and you’re just digging in various different directions.

Sometimes you come across something that ties all these things together. It’s useful. It solves a problem. And you had no idea the problem was there when you, when you started.

AB 25:15
So I’m using the phrase self-aware in a positive sense. I’ve not trying to be negative in the slightest. We probably – all of us, and this is the, probably one of my, uh, life lessons – is that it’s good to know how you like to learn and don’t try and fight that.

If you need to be tactile or you William Violet, can we all have help with the… how do you pronounce it? Icky Guy? (Ikigai: https://lauchlanmackinnon.com/rethinking-ikigai-how-to-find-work-you-love-and-make-a-difference/ )No? Yes? Kind of.

It’s the classic Venn diagram which you’ll see in meme format on everywhere, but it’s basically a VENN diagram of, yes, three or four things that you love, things you’re good at, things the pay well and things I guess the world needs as the new addition to the VENN diagram.

AB 26:21
And there’s no peril wherever you want to sit with your passion, your playing, your learning. But some of those places that you sit are great and that’s an open source project or a weekend project. Others are this is where you want to head to.

So for me being ‘self-aware of’ I can sit here and say “weekend project – this is just a throwaway”. But these ones here over in this hand wavy area? that’s where I kind of want to be. And if you can ever find things that pay well you love you know the old adage of you’ll never work a day in your life.

That’s the hope for us all. We might leave it there, everyone. We’ll catch you next time on SPAITIAL, where we’ll come to you with News of the Week, then on a separate episode, we have the next topic of The Week.

AB
Cheers all, thanks for your time. Bye. Bye

Violet
Bye Bye

To absent friends.

The post Episode 009 – Getting started in Spatial AI appeared first on SPAITIAL.

Recorded on March 1st, 2024

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Geospatial Marketing/Branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From William:

GEAR: Generalist Embodied Agent Research

At GEAR, our mission is to build foundation models for embodied agents in virtual and physical worlds. Our research agenda encompasses:

https://research.nvidia.com/labs/gear/

From Violet:

“Toyota Research Institute is developing robotic capabilities that amplify, rather than replace, people. We’re on a mission to help with everyday tasks that require more than just our hands and fingertips. Our research platform, Punyo, embodies this mission. The Punyo team is focused on bulky object manipulation using the arms and chest to complement TRI’s other efforts in fine robot hand- and gripper-based dexterity. We are developing hardware and algorithms that enable truly capable robots to help with large, heavy, and unwieldy items.”

https://www.youtube.com/watch?v=FY-MD4gteeE

From AB:

MoonHub – https://themoonhub.com/ – a UK-based VR training company that just received another round of funding – still early stage – to make training ‘stick’, by putting the trainee in the scenario, and scoring them on more than ‘knowledge’, but also reaction times, spatial accuracy, spatial awareness and initiative.

BlueRoomVR – https://www.blueroomxr.com/ – a “semi-lo-fi” medical training room for medics within the military – the single dummy patient can be configured to have any number of ailments in any number of dangerous scenarios, and the two medics together problem solve the scenario while wearing headsets. All the ailments are overlaid on the training dummy, but the feel is real, often with real blood veins to find.

“Semi-lo-fi” in that ‘lo-fi’ is usually the term for very simple cardboard-based mockups – maybe some fake switches – onto which you overlay your complex interfaces/controls in VR. This gives the user some surfaces to use for tactile feedback, but only minimal. By ‘semi’, BlueRoom is going a little above and beyond ‘cardboard technology’, using a state-of-the-art medical training dummy to give a positive feedback cycle into the VR scenario, all in real time.

From Mirek:

Today, Hello Robot is announcing Stretch 3, which provides a suite of upgrades to what they (quite accurately) call “the world’s only lightweight, capable, developer-friendly mobile manipulator.” And impressively, they’ve managed to do it without forgetting about that whole “affordable” part.

https://spectrum.ieee.org/hello-robot-stretch-3

To absent friends.

The post Episode 008 – Spatial AI News of the Week appeared first on SPAITIAL.

Recorded on February 23rd, 2024

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From Violet:

From this Finnish hardware/software design group comes a tantalising preview of “The Ring” – a prototype device with finger pressure sensing, so that the thumb-mounted controller “becomes a creative tool for the world canvas”.

https://soundxvision.io/

From William:

A prescient white paper/video coining the term ‘General World Model’ for a spatially-aware multi-modal large model… whew – quite the definition, but essentially what we at SPAITIAL are commonly referring to as the next iteration of text–>images–>video–>spatially-aware large models.

Will the term and acronym (‘GWMs’) catch on? Time will tell. the concept is rock-solid. The marketing… still needs some thought.

Watch along, however, and stay tuned as we review Runway’s efforts in future episodes, in light of their new catchphrase.

https://research.runwayml.com/introducing-general-world-models

From AB:

Aria is a dataset for researchers and spatial hackers alike, that just had a major uplift in both volume of data captured – and more importantly: richness of dataset including new data types and annotations.

Starting from the point of view of ‘me’, this dataset follows one or two people through various everyday activities – but at each millisecond, capturing: images, video, sound, positioning, gaze tracking, object detection – as well as the surrounding point clouds of the space the person is in, and their movements within in.

Why all the fuss? some commentators suggest that this kind of dataset is perfect for training your very own ‘Jarvis’ – an AI that knows *you*. More neutrally, though – this dataset should breed a new range of understanding and embeddings around the ways humans navigate / where they look / how they validate their actions / and the differences between intent and actual movements.

https://www.projectaria.com/datasets/aea/

To absent friends.

The post Episode 007 – General vs. Local AI appeared first on SPAITIAL.

Recorded on February 16th, 2024

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

Helena Merschdorf
Marketing/branding at Tales Consulting.
Undertaking her PhD in Geoinformatics & GIScience.
 

Mirek Burkon
CEO at Phantom Cybernetics.
Creator of Augmented Robotality AR-OS.
 

Violet Whitney
Adj. Prof. at U.Mich
Spatial AI insights on Medium.
Co-founder of Spatial Pixel.

William Martin
Director of AI at Consensys
Adj. Prof. at Columbia.
Co-founder of Spatial Pixel.

From Violet:

You’re likely well aware of text-based LLMs to pour out large blocks of code in response to your requests – but most of those LLMs exist within their own sandbox – a walled garden, for you to copy’n’paste those blocks of code out from, to make them into real applications.

Well, no longer. Open Interpreter begins to cross the streams of a virtual [code] assistant, and can offer to help you work *across* your current applications, for a wider range of tasks – on the fly. Granted, every command or code output is met with a prompt asking: “are you sure you want me to do this?”. This sounds like both a great way to put a human into the loop in the interim – AND a great way to have a user interaction to allow cross-application scripting to occur in the foreground.

Demo here: https://github.com/KillianLucas/open-interpreter/#demo
Product page here: https://openinterpreter.com/

From AB:

I’m calling it: this model will be plug’n’play in a robot/car/phone/device near you, within the end of the year.

This computer vision model – not just a concept, but code. And demo. And paper. AND freely downloadable model! – takes any image as an input, and returns a depth map – a greyscale/colour-scale mask of what it sees in the image, in terms of near or far from the viewpoint of the camera.

In essence, this does what previous a fleet of discrete RGB and/or LiDAR cameras would have previously done – estimating a scene in 3D space – but with a single camera as the input.

The model is so temporally stable that it’s been shown to process the still frames of a video, in sequence, and keep consistent depth masks across the flow of time. Amazing.

https://depth-anything.github.io/

From William:

In this YouTube short from Boston Dynamics, we get to see their Atlas humanoid robot begin to have working hands and fingers, AND we get our first view from Atlas’ own internal cameras, as the robot overlays a digital twin of the car part that its currently handling, over the actual part – showing the improvements in real-time spatial understanding. A fascination watch.

https://www.youtube.com/shorts/SFKM-Rxiqzg

From Mirek:

A must-watch! 20+ android-like robots – on wheels – all performing various tasks around an office scenario, and all controlled by a central neural network.

Unless Captain Disillusion debunks this one in the coming weeks, we have to assume that it’s real – and if so – well *dang*. Low-cost (ish), but genuinely-human-replaceable… for some tasks, at least.

For bonus points: see how they plug AND unplug themselves from powerpoints when they’re low on juice!

https://www.youtube.com/watch?v=iHXuU3nTXfQ

Luxonis – OAK LIDAR/RGB cameras for robotic applications: https://shop.luxonis.com/collections/oak-cameras-1
Bring me a Spoon – the thought experiment that is the litmus test for robotics & AI: https://bringmeaspoon.org/

To absent friends.

The post Episode 006: Spatial, AI & Robots appeared first on SPAITIAL.