This week, AB has an amazing chat with Paul Stevens, CEO of Voltitude, a company focused on exploring and unlocking the potential of the stratosphere for research and commercial applications.

Paul discusses his background working on the Zephyr program at Airbus, developing solar-powered high-altitude pseudo-satellites. He explains the challenges faced in designing aircraft that can maintain their position in the stratosphere while balancing the delicate energy budget. Paul and his team at Voltitude aim to overcome the vulnerability of these aircraft to gusts and turbulence during ascent and descent, expanding their operating envelope.

Voltitude is currently exploring the use of small, low-cost, biodegradable latex balloons to carry meteorological sensors (dropsondes) into the stratosphere. These balloons can drift for several days, dispensing dropsondes that transmit atmospheric data as they descend to sea level. Paul discusses the regulatory framework for high-altitude balloons and the potential for using machine learning and AI for mission planning and route optimization.

Connect with Paul on LinkedIn at: https://www.linkedin.com/in/paul-stevens-b07b0382/ or on the Voltitude web site.

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/_d5wYy14mx4

Paul discusses Voltitude’s current projects, including the use of small, low-cost, biodegradable latex balloons to carry meteorological sensors (dropsondes) into the stratosphere. These balloons can drift for several days, dispensing dropsondes that transmit atmospheric data as they descend to sea level. He also mentions the regulatory framework for high-altitude balloons and the potential for using machine learning and AI for mission planning and route optimization.

Paul discusses the possibility of miniaturizing super-pressure balloon technology to create a more sustainable and tactical delivery system for environmental sensors. He explains the differences between zero-pressure and super-pressure balloons, and the challenges involved in designing a lightweight, long-endurance system.

Paul discusses Voltitude’s efforts to improve the resilience of highly aeroelastic aircraft to turbulence and gusts. He explains their approach of using sensors to detect deformations in the aircraft’s shape and actuators to control the lift distribution, restoring the desired shape and flight control authority. Paul also mentions the potential for using AI to preemptively adjust the aircraft’s shape based on sensory inputs.

AB
Well g’day and welcome to Episode 25 of SPAITIAL. If you’re watching this, it is a lot darker, all my coloured lights are a lot brighter, something’s happening. Yes, it’s my evening. Why? That’s because I’m talking to a very special guest today.

A big welcome to Paul Stevens – coming to us from sunny UK. Yes, I get to use those two words together in the one sentence. Paul, g’day, how are you? Welcome.

Paul
Oh hello, thank you very much for having me. It’s great, I’m very well, and it is indeed sunny. In fact, we’re having a little heatwave, so a lot of confused British people.

AB
Okay ‘heat wave’… I must ask the question. Weather check, what do you mean by heat wave per se? What’s a UK heat wave?

Paul
Well yesterday it was 31 degrees. Which is very uncomfortable for us British folk.

AB
That is “everyone to the beach”, “everyone grab the flannels” and start to, yeah, okay. Well done. That does count as a nice heatwave. Well done!

Paul
Well, that’s saying is something from somebody who’s used to heat waves.

AB
True, true, well, days of 40 aren’t too bad – it’s when you get five days in a row and when the can of beans in the middle of the house in your pantry is also 40 degrees, when the heat is just leached in from every side, that’s a good heat wave, but still, 31 in the UK, you get top marks for that.

Now for the record, Paul and I met a couple of months ago, we actually met at a, let’s call it a conference in, let’s say, northern Europe, that’ll do, but we actually, like ships passing in the night, we both, I think, heard each other’s five-minute intro lightning talk and then we’re in different streams the entire few days and on the last day, we must catch up, we must have a chat.

This is that chat, this is the time to actually finally do a final deep dive to actually ask all the burning questions which you didn’t get to ask back then.

So, Paul, you are CEO of Voltitude, which is a lovely company, which has an awesome name and I love the fact that you’ve got yourself, Paul Stevens, and then your two of your partners are Steven and Stephen, your meetings must be ‘Steve-o’, ‘Stevie’ and pretty problematic?

Paul
Yeah, we need to work on our diversity statement. People other than Steve are allowed to join Voltitude. Although it may surprise you that there are a total of four Steeves in a 12 -man team.

AB
You’ve hit the KPI for the year, I think even for the decade, so well done!

Voltitude does some astounding work and it’s definitely in a area of geospatial AI data science that is literally on the edge. You are playing with the highest of altitudes. You are playing with the highest of latitudes. Can you give us a easy then a medium and we’ll save the hard explanation for where do you play? What are the sort of things that you are actually doing?

Paul
So to tell you a story of why we are obsessed with the stratosphere and what you can do with the stratosphere. I’ll just fill you in a little bit about my background, how I got to here. Awesome.

So prior to 2020, I was working for Airbus on the Zephyr program. So Zephyr is a solar powered, high altitude, high altitude pseudo -satellite. And what they mean by that is that it is a very special type of plane that can cruise day and night only on the energy it’s collected from the sun through very efficient solar array and hold that in extremely high specific energy density batteries.

And by building such a craft, we can effectively create a near perpetual flying machine with interesting properties. It’s a bit like a satellite in that it’s got a very large field regard, but it has aerial resolution.

So it’s an interesting blend between the resolution capabilities of aerial sensing and the huge swath and low cost per meter square of space. And so we were developing this platform for Airbus. And I must say it was a two decade obsession to make it work.

Paul 04:56
And I was the head of design. So I had a career high in 2018 when we demonstrated all the key enabling technologies that come together and made it possible to design an aircraft that could hold itself in the stratosphere, cruising day and night and balancing this delicate energy budget.

So you might say, well, why have I changed, disrupted the ecosystem and disappeared from what was an amazing program. And yeah, it was simply two aspects really. One, I was really interested in opening up the stratosphere to other applications, which were perhaps not only defense related and not only connectivity related.

I was really interested in Earth observation. And number two, although we have demonstrated the energy budget aspect of designing these aircraft, which I’ll explain in detail in a moment, although that aspect had been solved and we could now hold a plane in the stratosphere day and night for months on end, the type of aircraft that can do this are extremely vulnerable to gusts and turbulence when they climb to and descend from the stratosphere.

And this vulnerability I saw as the Achilles heel for the entire industry. And we thought that there was some, the next challenge was to solve that, but without detriment to its performance in the stratosphere.

So, you know, I love, I love working for Airbus, but there are certain things you can’t do in a big corporate environment. And so my friends, my colleagues, my design team, basically we’ve joined up in altitude.

We’ve made a clean break from Airbus and our challenge is to open and unlock the stratosphere to a whole range of new applications and to overcome this Achilles heel for the industry so that we can expand the operating envelope of these unusual aircraft.

And so that’s, that’s how, that’s the story of how I got to here. And so I’m joined in altitude by my former design team from, from Airbus. We’re still on talking terms with Airbus. We’re not persona and vatius, fortunately.

And the journey since then has been to test the utility of the stratosphere for interesting new applications. And one of the first applications we wanted to explore was getting really high resolution meteorology data from remote areas.

And so we took a step back from fixed -wing solar electric flight, and we have started to explore the benefits of really, really small, low -cost, largely biodegradable natural latex balloons, and their ability to float for many, many days in the stratosphere and reach, just through drifting with the winds, reach remote areas, and to then dispense tiny meteorological sensors, it’s called dropsondes, which fall all the way down to sea level, transmitting their observations back to the balloon. And then that data is then disseminated via SATCOM, and we receive that, do some quality checking, and then send it out to our data customers.

Paul 08:18
And so it’s a really simple, it’s a reinvention of a very old idea. So cast your mind back to the 1940s, in the thick of World War II, perhaps, and as a terror weapon, the innovative Japanese were launching arrow -triggered, vented balloons to drift in the jet stream across the Pacific Ocean.

Yeah, across the USA. And drop incendiary devices onto the USA. I don’t know for certain if anyone’s ever killed, but as a terror weapon, it was incredibly effective. Gotcha. So we thought, well, you know, let’s not terrorize people, try and do the opposite.

And let’s try and solve societal issues and help improve the forecasting of extreme weather events. And so what we’ve done is fit a standard meteorology balloon, latex weather balloon with a vent and a ballast bag so that it can control its height.

It doesn’t go up and burst. It controls its height. We can set a height which corresponds to winds that will are desirable for our drift direction to take us to an area of interest. And then we can drop tiny, tiny dropsongs.

Our dropsondes are about 20 grams in total. There are some insects that are heavier than these dropsongs. And these dropsongs are stored for many days or weeks at a time in the stratosphere. We resuscitate them, we warm them up, we charge the battery, and we dispense them.

And about 20 minutes later, they fall all the way down to sea level. And all that way, they measure temperature, pressure, humidity, wind speed and wind direction in very high vertical resolution. And all of that data is then collected. And we send that to our data end users. And it’s all in near real time.

AB 10:16
Look, I must say, I’ll put the link to your products page in the first thing within the show notes. Stratus Sond. StratusSonde, am I saying that correctly?

Paul
Yes, a combination ‘dropsonde’ and ‘stratosphere’.

AB
So sold on trademark like that, but the image is there. I mean, you really are showing pennies and, yeah, pence there for size. So how long can that parent balloon stay aloft? What’s what’s the sort of the mean range of days or is it measured in weeks?

Paul 10:48
So those balloons are really basic. It’s a very high value system. It’s a bit like the explanation for why do they have finite life? If you can imagine this ball of buoyant gas, every day when the sun sets, that gas ball cools and it loses buoyancy and to prevent it from sinking all the way down to surface, it has to throw out some things it doesn’t need anymore.

And so we dispense harmlessly, we dispense sand. So sand trickles out of it and falls away. And that reduction mass catches the drop in buoyancy. And so every day, a bit like compound interest, negative compound interest.

You’re just paying it forward.

AB
And eventually you get to a point where, gotcha, gotcha. It runs out of ballast. Gotcha. So, um, days or cycles, do you mean days or sun cycles?

Paul
Yeah, we we get about in the tropics, we get about five sunsets at high latitudes. We get a few more maybe in theory, we should be well in excess of 10. But we find that the UV degradation of the latex envelope then causes it to burst.

And so really, you can only really count on about five or six days. And in that time, you’re going to be able to drift to somewhere interesting. And, you know, regularly distance your drop songs and we carry up to 10 drop songs on these balloons.

So it’s a and they’re really small, you know, one person can launch them, it’s very low logistics footprint. And so any if you’ve ever watched me trolls as small just standard weather car is exactly the same.

AB
Any priming of the, what’s the, is there much of a CPU or a black box or is it just comms and hardwired and it’s just a slave?

Paul 12:59
Yeah, there’s a little computer on board which manages everything, there’s a default mission plan which usually works for most missions, but if it doesn’t, once airborne, our operations base in Farnborough can send it a new mission plan, and so really it’s a case of somebody who really knows nothing about the system can inflate a balloon with buoyant gas and release it.

They don’t really need to be able to program it or anything like that. It’s very hands -off, very simple, and then it’s all managed centrally, and indeed, we don’t even have somebody watching it all the time.

We have a computer generating a mission plan for each balloon, and that’s worth talking about because it’s crying out for its own talk, which is all about machine learning and artificial intelligence on planning routes through a complex wind field. You have the platform.

AB
You’ve got time, you’ve got altitude, you’ve got the cycles to contend with – you haven’t got battery life per se – but how do you chart a four dimensional path – a complex route?

Paul 14:10
I’m going to disappoint you really badly here because we’re a bunch of Luddites and what we do is a large scale Monte Carlo simulation where effectively we tell synthetically a balloon to change height to all possible altitudes within a certain resolution and every six hours it changes back to all other possible altitudes and this Monte Carlo simulation of all possible outcomes for this balloon over the next five days is computed.

It’s an end to the power of wide problems. But with modern computing, this is done within a few seconds. So there’s a probability map of where the balloon could go. And so we highlight where we want it to go and all paths of truth that lead to that location are identified.

We can wait each branch and say, well, what’s the common factors here? Which branches in this decision tree hold the greatest likelihood of getting to where we want to at the right time?

AB
Mission ‘set and forget’ you can tell it what the mission vaguely is and unless it goes outside those parameters because of Hurricane X comes along chances are it will converge close to that mission profile

Paul
And of course, you know, the world’s heartbeat for meteorology is about every six hours there is an update to the global forecast. And of course, in six hours, the balloon has made some progress along its planned track.

And so every six hours, we automatically rerun the mission planning system with fresh data, with progress. And that is then said, you know, progress and fresh data and differences are fed back in to create an enhanced track prediction.

So in a sense, we are vectoring in, as time goes by, but the whole complexity of this is is is rife for modern thinking and modern machine learning principles and artificial intelligence and, and so on, so forth, to be thrown at this to because as we as our constellations get larger, it’s inconceivable that we should be doing this on a balloon by balloon basis, and that there are certainly better ways to do this to coordinate the

AB
How many do you have up at any one time? If it is in known numbers, it’s a great thing to always ask a potential client, what number would make you change your thinking? So 10X, some people go, oh yeah, 10X is fine. Okay, well, if that’s fine, how about 100 or 1000 x?. At what point in time would you have to transition to using it like a, and here’s the gnarly question, what’s the collective noun for a group of smart balloons?

Paul
I’m not sure, but I know what the collectives now for Radiothorax scientists are, and I’m sure it’s probably something like a disaster of transparent engineers, I don’t know. Our busiest time of year is the tropical cyclone season in the Atlantic, and we’re hoping to branch out services into the Pacific.

But at the moment, we rarely have more than between five and 10 balloons airborne at once. So with the numbers nice and low, we can take this rather clunky methodology and keep it going in an automatic sense that humans really don’t have to, well, aside from launching them and deciding at an executive level what the mission should be, humans are no longer interacting.

The drops ons are scheduled, the path of the balloon is automatically being modified and evolved in response to new forecast data and progress. But the answer to your question is, at what point does it become, you know, do we need to evolve?

It’s really, really soon. If we are successfully expanding services to the Pacific, we really should be doing it straight away. And related to your previous question, which is, why are the balloons only five days endurance because of this compound interest effect of cooling on the buoyant gas?

Well, there are other innovations in balloon technology which could benefit from being reinvented in a much smaller form factor. And we’re exploring really small super pressure balloons, which is another type.

So what I’ve been talking about is a zero pressure balloons, the balloon which has a vent in the bottom effectively. And if it wasn’t for the fact that latex is elasticated and trying to squeeze the buoyant gas out of itself, if it was just a parting bag, this would be called a zero pressure balloon.

And it’s one where each day that cooling effect causes it to have to dispense ballasts and retain its buoyancy. A different type of balloon called a super pressure balloon is one where the envelope and the gas inside it is under a small pressure.

And that pressure is large enough such that even during that cooling cycle as the sun sets, there is still a positive overpressure in that envelope, even at the coldest point of the night. And by achieving this, the balloon does not change density.

Paul 19:49
And so they’re often called constant density balloons. And what we have there is a really long endurance balloon able to float almost indefinitely if it wasn’t for a fusion of buoyant gas through the balloon film.

And so this is if you think of perhaps some of the really large balloons you might have seen doing scientific missions operated by the likes of Aerostar International or even the Chinese spy balloon that we all came to early with 10 years ago.

Those were examples of super pressure balloons, very special super pressure balloons, which are able to fill an emptier ballonet inside them and to control their height and benefit from this remarkable feature, which is that they don’t change their density on the sunset to give really long endurance and altitude control.

And so one of the technologies we’re exploring is what is the smallest balloon of that type that you can design practically and has all the requirements.

AB 20:58
And is it still the on back of a single person, back of a car, the graphical, comic-style Swedish furniture maker instructions – not a list of instructions a mile long? Is that, so the super pressure balloons, are they managed by teams on the ground? Are they tenfol- sized today?

Paul
At the moment, really the smallest commercially viable super pressure balloon carries about a 50 kilogram payload underneath it and is a really large envelope. We’re talking maybe 15 to 20 meters in height and a really huge volume of buoyant gases used in that process.

It requires a substantial team to launch these things. I would say it’s still pretty tactical. You could turn up with a Land Rover or a truck with all the equipment needed to do that, as well as another truck with all the buoyant gas cylinders.

It’s a 10-fold, maybe even 100-fold increase in the effort to get one of these things airborne. They’re a mature technology. I think the world record is about 250 days, which is really impressive and hats off to those people.

I think the average is more like 60 days, but it’s a really interesting field of endeavor. High altitude balloons are afforded an unbelievably welcoming to aviation via the regulatory community. The regulations are common sense, not the sort of thing that you find in conventional aviation.

They are very operational risk assessment. It’s very lighting. It’s all about the operational risk assessment and ensuring you operate your balloon in a way that is safe for the general public, knowing that all balloons end their lives in what aviation would call a catastrophe, a crash.

They will always fall from the sky, hopefully under their parachute. That’s the end of their life. They’re a single -use system. Remarkably, the regulatory world has embraced this principle and afforded huge freedoms for innovation.

We see the efforts to miniaturize super pressure balloon technology to something that’s really quite small and tactical and as sustainable as possible as a potential new and disruptive delivery system for environmental sensors of the type that we do.

AB 24:08
10 or 20 times the number of the songs the the small yes you would and then you would need better comms and better mission planning to To actually push them in the right direction for longer. So you

Paul
And it’s a real challenging system engineering optimization process. An altitude changing super pressure balloon needs a compressor and a compressor that works up there in the stratosphere at 30 millibar.

And it needs solar arrays, it needs rechargeable batteries, and it needs a mission system. And it needs this high tech envelope, which is ultra thin and ultra lightweight and able to withstand the mechanical strains of this overpressure that keeps it in this constant density mode, but also allows you to fill and unfill a ballonet to deliberately change its density, to change its altitude and actually navigate.

So there’s a minimum capital of mass of system to go into this product to make it viable. And in the system engineering optimization space, it’s very easy to chase your own tail into nothing and find that you produce a brilliant system, but there’s no payload capacity or there’s no excess power to provide for a payload. Correct, it’s almost the opposite of that.

AB
The famous line of ‘the best part is no part’. That’s true but when you’re left with just a balloon you’ve removed the bits that make it viable!

Paul 25:48
And it’s probably no surprise to you or anyone listening that the available mass fraction of a system like this, available to its payload, gets smaller, the smaller the overall size of the platform.

So at some point, it is just not possible with current technology to make it smaller than a point of commercial value. And we’re exploring that point at the moment with some projects we’re running, and exploring whether or not they offer disruptive new services for Earth observation and meteorological data collection.

I should mention that there’s a sort of regulatory, I’ve said the regulations are very lax, there are regulations, and a really convenient threshold is for meteorology boons is if you can operate in what’s called the light category, which is where your total payload mass has got to be less than four kilos.

And payload is considered to be anything that’s hanging under your balloon. So it’s not just your, your box of tricks that are paying you your your livelihood, it’s all the things that everything else.

Yeah, all mechanical things and everything, all the critical systems to make the balloon work. And so four, four kilos is a key threshold, because the world in 1945, at the International Chicago, civilization organizations inaugural conference, set out some rules of the air.

And they said that balloons of a light category used exclusively for meteorological purposes, can be allowed to cross international state boundaries without prior permission. Okay. And so this is like a revelation of global freedom for a balloon in a weight, the light category can go anywhere without prior conditions.

You must ask for permission to launch it from your launch state. But but once it’s airborne, it’s allowed to go anywhere. So, so being in the light category, an engineering of balloon that can be so light, that it can do all these amazing capabilities in that tiny weight constraint, suddenly opens up global freedom.

And so that’s the, that’s the exam question. I don’t have the answer, yes, whether it’s a commercially viable solution. Well, we’re doing some great prototyping. And we are optimistic that there is a system hidden in there somewhere that allows us to open up a new application to the stratosphere with a 10 fold capability.

AB 28:41
Existing’s like looking forward to that and also you mentioned earth Observation so of course dropping little droplets, you know, 20 gram things behind you that can sense is still you know Astounding I need to talk to you about comms in high Altitude and high latitudes, but what sort of earth alps would you be thinking?

How do you put a… You don’t want a wide-angle camera you want the longest super zoom in the smallest package are there Throw away cameras of that nature or parachutes and you retrieve what’s the what’s the size weight and power?

What’s the what’s the delicate balance you’re aiming for in that kind of world?

Paul 29:22
Yeah, I should say we’re probably not seriously exploring conventional Earth observation using the principles of remote sensing, a long focal length optical system staring down the surface and retrieving imagery.

We do use our balloons for test and evaluation purposes and indeed we have a steerable parachute. So when we carry a high value payload to the stratosphere, we can conduct multi-day tests at a really low cost because our balloons are so cheap.

And we’ve got a reasonable chance of not just navigating the balloon to within land, but then releasing this steerable parachute and navigating that back to do a spot landing and retrieve a high value payload.

So you can imagine the industry of fixed-wing solar electric, high altitude satellites is still in this infancy. And there are a lot of payload developers exploring the solution space, which is a really demanding size, power and mass envelope.

Paul 30:45
And there are very few successful operators of these planes at the moment. So there’s a very long waiting list to get to the front of the nose of one of these aircraft with your payload. And that’s not very helpful if people want to do a real world full system test before the embarrassment of its failing on one of these precious flight opportunities.

So there’s a mini market in providing test and evaluation services on balloons. And I would say that’s probably the only time that we remotely get close to what you would describe as Earth observation using multispectral, hyperspectral sensors.

However, I would say that our efforts and technology relating to fixed -wing aircraft to expand their operating envelope, which I can talk more about, and to get those aircraft more regularly transitioning through the troposphere to and from the stratosphere more successfully opens up all kinds of commercial applications, which currently are priced out because the platforms are just so vulnerable and they keep crashing.

AB 32:07
A two-stage, a high-density atmosphere, low altitude transitions to a different mode of working or a way to protect or pardon, that’s the toughest power to weight ever surely?

Paul
Yeah. So I’m hoping you’ll love the simplicity of the solution that we are pursuing. So when you analyze why these aircraft crash when they’ve encountered gusts and turbulence, it’s very rarely that they were not strong enough.

The problem they encounter is that by prioritizing within their design, this energy budget in the stratosphere, the need to be able to cruise on so little power. And we’re talking powers equivalent to human powered flights.

I’m sure you’ve seen these athletes pedaling as fast as they can to get airborne and cruise just above walking speeds.

AB
Ah! I went down the YouTube rabbit hole a few weeks ago, I’m gonna say Japan and I’m gonna say 60 kilometers? I watched a bird man cycling and he just kept on going. He actually had to stop because his muscles were cramping up.

Paul
It’s amazing – the English Channel used to be the reference mark for for success But these modern athletes doing this sport are doing you know, 60 kilometers, which is huge a huge distance But they are super human.

They they’re able to produce a lot more useful But so it’s a very similar problem, how do you how do you close that energy budget in the strategy? the answer is fly with minimum power required to cruise and Casting your mind back to your your school years.

I’m sure you’ll remember that drag is proportional to velocity squared. And if you can still remember these classic physics equations power is drag times speed so a Velocity squared for drag times speed again is a velocity cubes term So the power required to cruise is a V cubed relationship So the in the biggest lever you have is to reduce V right?

So you have a really slow flying aircraft up there in the thin atmosphere. It has to fly pretty fast to generate lift But it’s still you know It’s still gonna get there and the design space solution is a really it ends up with a really wide wingspan very high aspect ratio and incredibly light and very very low wing loading and With a distributed mass so it is a you know There’s no central point like a passenger carrying aircraft whether the fuselage let everyone sits and a strong stiff wings holding that you can’t do that with these aircraft You’ve got to distribute all of your your mass uniformly across this this flying wing.

And the net effect is it’s it’s incredibly easy to deform it’s it’s displays very strong aero elastic properties Okay, what I mean by that is the serious wings twisting.

Paul 35:24
Yeah forming bending torsional effects and And that’s a byproduct that comes from prioritizing the minimization of mass on your aircraft and And the issue there is that when a plane of that type encounters a gust or some turbulence It might be designed with a little bit of dihedral the dihedral is a sort of v -shaped your wing Great and it gives a gives you all aircraft role stability It’s a it means that even with no flight control system on board a role will generate more lift on the on the downward hanging wing Naturally restoring it to stable flight and And so most aircraft are designed with a shape in mind that gives them their optimal flight control authority And the problem with these really aero elastic aircraft is that when they encounter, you know strong or moderate or even light turbulence the effect can be the shape changes completely right and the the the aerodynamic derivatives the stole the Stability characteristics of this can you say lost momentarily or for yeah.

Yeah, it could be lost for many tens of seconds We’re talking about a very slow train crash Yeah, and and the problem is this this is these aircraft once they’ve lost their stability characteristics They they find themselves in a kind of divergent runaway situation Where they might have lost role control.

They might be entering a spiral dive They’ve got no way of coming out of it because the shape has changed to one of zero role control it might have been turned to instead of having dihedral I’ve gone to and he drove and And so this for aircraft Yes And so just gets faster and faster and faster until eventually it just breaks up There’s there’s too much too many aerodynamic forces on it and the structure fails and so You know the step one in our solution is to tell the aircraft what its shape is.

So MEMS technology is super light super lightweight as a key enabler for the solution is to cover your aircraft in Sensors that allow the aircraft to know its shape Wow and the in the second Attribute of our system odds our solution is to give the aircraft an ability to change its lift distribution across its wing To allow it to vary its shape back towards its idolized shape So if it finds itself encountering a gust that deforms it in some way Maybe makes it bow in the middle or even accentuate We can get the aircraft to sense that deformation has occurred and actuate devices which will change the lift distribution to restore the shape, to give it flight control authority back to control its flight envelope.

AB 38:37
getting a simple descent of mechanical help in those extreme situations to pull it out of the deadly dive?-

Paul
Actuators usually cost you mass, and people are familiar with some of the actuation systems on a conventional aircraft. I’ve probably heard of ailerons. So, ailerons are a trailing edge device. They sit on the trailing edge, and when they move, they impart a torsion into the wing.

Now, normally wings are stiff, and all the planes that you and I travel on, for all the tempers, they’re really stiff, strong aircraft, and ailerons are great for that. But on a highly aeroelastic aircraft, like a solar -powered HAPS platform, ailerons are terrible, because as soon as they deflect, they impart a torsion, the wing will twist, and they can result in

AB
You can’t about turn it into an oscillating mess and yeah

Paul
It would be a disaster. And the only way of making such devices work is to stiffen up your structure and add more and more mass. And eventually, you can no longer close that energy budget in the stratosphere.

Paul 39:52
So we’ve been thinking about ways in which we can control this lift distribution, control the lift generated by all parts of this wing without imparting torsion into the wing. That’s the key factor. And I’m sure you’ve heard of a spoiler.

So a lot of cars have spoilers on, they dump lift. When I was 18, I had a very, very underpowered car. And I think it was a spoiler. I didn’t need it. But I thought it was cool. So spoilers are really great at destroying lift. And they’re really simple actuation devices. They don’t need to be very small. actively put some pretty much a binary off on off on click it into the air stream to just suddenly bring things back in milliseconds

Little fences can pop up here and there to disrupt the lift, and so for very little sophistication in your actuation system, you can really have quite profound disruption of lift for very little mass or electrical power, and that’s why dumping lift is a really great solution to the exam question, which is how do you dynamically vary the lift distribution of an ultralight air elastic aircraft to control the shape of that aircraft as it transitions through turbulence and gusts and other features.

AB 41:22
Do you train such a system? Are there known rules for such a system? Is it is it a known set of ‘if this then that’? or does it have to be put through simulations? And dare I say it a small AIML to try and simulate this thing up in this up in the upper reaches?

Paul
There is certainly a solution derived from conventional control theory, with a like a super loop around your conventional flight control loop, which is maintaining the shape of the plane.

However, the really sexy part of this is that there is the potential for AI to preempt based on some sensory inputs, particularly if you’re able to sense disturbances before they’ve even reached the wing.

Paul 42:11
So just like a meter in front of the wing edge, you know, there is the potential to have a signal that says the last time this happened, it was strongly correlated with the following events. So why don’t we pre -end that.

AB
Things like radio interference between the nose and the wingtip things like that or?

Paul
Potentially, yeah, this is an area that we are road mapping into our technology. At the moment, we’re trying to get the conventional control theory solution working in an optimized way. But using old school engineering, the results are dramatic.

We were able to halve the bending loads in a structure, compared to a system without this technology. And we’re able to double the speed envelope. So we are able to control an aircraft back inside its normal cruise range from a position that would have been disastrous without the technology.

And so suddenly, this means that these aircraft are a lot more robust. They still fly very, very slowly, because the drag is still the enemy of the power budget. But they can withstand moments of very high speed flight and safely control themselves back inside that conventional flight envelope.

AB
And while the effort to make them still might be intense, they are long lasting, but the fact that you will have the confidence to know that it will be up in the air for longer and survive squalls and survive the unknowns better, that certainly makes it a much more viable per payload proposition.

Paul
I should admit, though, that like everything we do, it’s a reinvention. And anyone listening to this who’s a keen bird spotter may have watched large wingspan birds soaring on the winds. And they may have noticed that when they hit a gust, they let their wing deform.

They let the wing take a new shape. The bird’s bones connected by tendons and those tendons allow the bone to let other joints deform. If they didn’t, they would break their wing. The loads would become too great.

They’d snap a bone, they’d die. So they let their wings deform into new shapes and shapes which would not allow them to control their flight envelope.

And a bird is covered in sensors every single feather. Every single feather is a sensor for them. It connects to their nervous system. They have acute awareness of their wing shape. They’ve got acute awareness of the lifts affecting their wing.

Paul 45:01
And so they follow exactly the same algorithm. They sense that a deformation has occurred. And their priority, when safe to do so, is to restore the idealized shape. Then to restore their flight envelope and then to restore navigation on their mission, which was probably to hunt some small rodents that they were chasing.

And the effect interrupts why it seems to be that they’ve lived for another day.

AB
I think that gives you a definite good method of name for your next project for the I mean we’re talking Condor We’re talking we’re talking the largest of large birds. So Paul that is a phenomenal phenomenal journey from weather balloons with smart devices the Drops ons falling on ahead the size of a less than a coin.

So we should be too worried going from that to from zero pressure to Stable pressure brushes gotcha Cheers and then going from the Chinese spice at Spy balloon size down to can your company make something that is you know truck single-person deployable?

And then not only have you come from the largest solar-powered long lasting astro spheric You know fixed wing you now try to already design the next flavor of you know Better faster cheaper or in this case longer lasting and you know better bang for your buck More resilient more resilient able to leap tall buildings and survive for longer Paul.

That is an astounding journey. I don’t don’t think we’ve actually touched the sides of it I actually love to have a chat to you again in a little while if that’s possible and we’ll come back and talk about the communication sort of things you’re doing what data you can actually play with and Catch up and see where you’re going on both those two next -level challenges that you’re facing

Paul
Sure, it’s been a pleasure.

AB
Where do people find you, where’s the best place? We’ll have links to Voltitude, and to LinkedIn for you. Any other places people need to reach out and find you?

Paul
Well, I’m usually fairly vocal on the conferences I attend. I go to a lot of meteorology conferences.

AB
I must say, I’ve been living vicariously through your LinkedIn feed, Paul!

Paul
The EGEU is an annual one – the European Geological Geosciences Union – is one I frequent and anything to the tropical cyclones. I go to those, so if you’re interested in the stratosphere, meteorology, all those things I’ve been talking about, look out for me on LinkedIn and through our website, I’ll be posting which conferences I’m going to.

I’d love to meet up with anybody who’s interested in having a proper, a proper, you know, geek to geek conversation about this. So you can probably tell I love to talk about this. It’s a real passion and I love the physics.

AB
Such an amazing field – and you know you wouldn’t start as a teenager saying ‘I want to do this’, this is obviously you jumping from one insight to another to another – you’ve obviously seen what’s coming and you’re getting ready for it.

It’s heartening to hear, Paul – thank you so much for your time.

We’ll leave it there and we’ll catch up with you hopefully in a couple of months definitely when you’ve got the next product ready to go but also we do want to talk about data and the data science coming off these devices

Paul
Thank you so much.

AB
Well, that’s it from Episode 25 of SPAITIAL, We’ll catch you on the next episode. From myself here and probably the plethora of cats that’s been walking behind me if you’re on YouTube we’ll say farewell and we’ll see you on the next episode.

Bye bye.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

To absent friends.

The post Episode 025 – Unlocking the Stratosphere with Paul Stevens appeared first on SPAITIAL.

Is point’n’click end-to-end low-code spatial AI… possible? What about *no code*? Spoiler alert: the answers are yes and yes. Knowledge of development patterns and code syntax and how many words you can type per minute are no longer barriers to entry for most of us. It’s time to get our hands dirty with spatial data!

This episode features Grant Case, the Vice President of Sales Engineering at Dataiku for the Australia Pacific Japan region.

AB and Grant discuss Dataiku’s AI platform and its capabilities in handling various data types, including structured, unstructured, and spatial data. Grant highlights Dataiku’s ability to cater to different user personas, from low-code and no-code users to pro-coders, through its intuitive interface and integration with open-source libraries.

On a wider note, we explore the advancements in large language models (LLMs) and their impact on data analysis, particularly in the spatial domain. Grant shares examples of how Dataiku leverages LLMs and digital twinning to enhance data understanding and decision-making processes. The conversation also touches on the role of Chief Data Officers, data governance challenges, and the trade-offs between building custom solutions and leveraging existing tools.

Connect with Grant on LinkedIn at: https://www.linkedin.com/in/analyticseverywhere

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/1EU042y4_7A

Grant explains Dataiku’s AI platform, which caters to different user personas, from low-code and no-code users to pro-coders. The platform aims to bring these diverse users together across multiple technologies, allowing them to work in their preferred manner. Dataiku has been recognized as a leader in the Gartner Magic Quadrant for its completeness of vision, particularly in catering to low-code and no-code users.

The conversation shifts to the advancements in large language models (LLMs) and their impact on data analysis. Grant discusses how LLMs have opened up new possibilities for unstructured data use cases, such as natural language processing (NLP) and spatial analysis. He provides examples of how LLMs can assist in tasks like understanding business locations and mapping data.

Grant highlights the concept of digital twinning, which involves creating virtual replicas of physical systems or environments. He discusses how digital twinning can be applied to various domains, such as disaster recovery, infrastructure planning, and manufacturing. Grant also shares examples of how Dataiku leverages LLMs and computer vision for spatial data analysis and decision-making.

The discussion touches on Dataiku’s integration with open-source libraries and its deployment options. Grant emphasizes Dataiku’s ethos of being open to both proprietary and open-source technologies, allowing customers to choose the best solution for their needs. Dataiku supports cloud, on-premises, and hybrid deployment models to cater to different organizational requirements.

AB and Grant discuss the challenges of data governance and the role of Chief Data Officers (CDOs) in organizations. Grant acknowledges the ongoing struggle with data quality and governance, highlighting the importance of proving the value of data and AI initiatives to secure a seat at the executive table.

The conversation explores the trade-offs between building custom solutions and leveraging existing tools. Grant advocates for evaluating whether a solution provides a competitive advantage or solves a unique problem before investing in building it from scratch. He emphasizes the importance of focusing on value-adding activities rather than reinventing the wheel for solved problems.

Grant shares his thoughts on future developments in the AI and data analytics space, including the concept of Retrieval Augmented Generation (RAG). RAG involves combining LLMs with an organization’s own data to provide more contextualized and relevant responses. While RAG offers a way to quickly derive value, Grant acknowledges its limitations and sees it as a waypoint rather than the final solution.

AB
Well g’day, and welcome back to SPAITIAL. This is Episode 24, coming to you after, yes, a minor hiatus again. Apologies, I do often have things called ‘work’ and out-of-town-isms. Apologies, but we’re back on a regular schedule with this episode and one booked in for next week and the following week.

With me today I have the great pleasure of chatting once again, not with SPAITIAL, but back in old-school territory here with Grant Case. Grant Case is Vice President, Sales Engineering at Dataiku, Australia, Pacific, Japan.

Oh look, your title is long and varied. I’ll let you introduce yourself. Grant, welcome to SPAITIAL.

Grant
Thanks, Andrew. Hi, everybody. I’m Grant Case. I am the Regional Vice President for Sales Engineering here at Dataiku. For myself, I work in the Sales Engineering / Solution Engineering space, where I spend most of my time with clients across the region, but particularly here in ANZ, where we talk to organizations, both large and small, in and around analytics and AI.

I’ve been in with Dataiku for the last six years, but I’ve been doing everything, analytics, AI. Before we were calling it AI, it was all statistics, right, Andrew?

AB
It’s just math in the end. It’s just math.

Grant
It’s just math. It’s ones and zeros, but I’ve been doing it for the past 20 years across multiple different industries and quite a bit of that. Spending time within different domains, whether we’re talking about NLP, we’re talking about just machine learning, but also GIS has always been a very interesting background and interesting set of projects that I’ve worked with.

So happy to be aboard.

AB
So six years. Six years at Dataiku. I must say at the outset, I’m going to do the Australian way. I mean, you do the North American data.

Grant
I come from Queensland, right? Northern Queensland?

AB
VERY far north Queensland. But obviously from the US. For the record, Aussies versus Americans, you know, the ‘data versus data’. So we (we Aussies) have problems talking about data, the character in Star Trek versus data. That’s always wrong. But at the same time, we can, we can figure out the difference between ‘routing’ and ‘route’, which is nice. So we, we lose on one tech term, but we gain on the other. So that’s exactly, it’s a net neutral.

So that’s nice. Six years at Dataiku. I think I was chatting to you pretty much the month that you started. This is back before COVID BC and before the current sort of situation. You were, look, Dataiku was one of those tools – is one of those tools – that for me is still pure magic to sort of, you know, Vaguely quote Arthur C. Clark, you know, technology that’s in just indiscernible from magic is just, you know, a joy to use. And that’s really what it is. It’s a walk-up tool that does everything data related.

And yes, ‘everything’ is a big claim, but it really is. I’ve been describing to people with the catchy non catchphrase of it’s the no code, low code and pro code. It sort of doesn’t lock you out of doing the hard way if you want or doing a low code, which is the title of this episode – and a topic we will definitely come back to.

So if we do manage to low code is the graphical point click, no less powerful, but certainly if you need to do something quickly, look that goes to that goes to that, but the, uh, thing that really blew my mind and you’re probably going to blow my mind even more cause I haven’t caught up with the Dataiku world for a year probably — is the no code just press a button and have everything done for you? Do you want to give us a rundown on where Dataiku is and where it sits? What the other, yeah.

Grant
I’m gonna put you right in front of a client because you did a great job of it. So as a platform, we look at organizations today and try and understand that there are different levels of maturity, the different personas.

So anyone from just someone who just needs to consume the dashboard all the way up to someone like yourself, Andrew, that’s getting in, messing, fine tuning on different NLP algorithms, they all need to work together, right?

So Dataiku is the AI platform to bring all of those individuals together across multiple different technologies. So nobody, everybody works the way that’s best for them. Obviously, just from a pure number standpoint, we have more low code and no code users.

That’s just the way of the world, right? But the need is for everyone to be able to access. So yeah, so we’ve actually just announced as a leader in the Gartner Magic Quadrant this past year and are a few months ago and most, we are the furthest along on completeness of vision in many ways because of what the title of this episode is talking about those low code and no code users and how do you bring some of the sophisticated things that we were doing, you and I were maybe doing five years ago, maybe even two years ago in code. And now it’s run of the bill, anyone can do it.

AB 05:59
And those three different levels of approaches are vital because, yes, even if you have someone doing the pro code, the full on way, had you passed that down neatly to people who just want to get in and use the damn thing.

And then there’s the moral majority who don’t even know that they want to use it. You can have them in the no code mode where they just press a button and it’s done for them. They don’t even need to know the magic is just there.

And I think that’s, yeah, the realm of the data scientist/data analyst has traditionally been a bit of a lone wolf, not by choice, but by I’ve set up my environment, you know, how many years ago would have been my hotted-up PC with a water called GPU was the way to go. Those days have gone. Thank goodness. Then it was okay: I’m buying 35 minutes of time on a beast of a machine in a cloud. And now it’s like, okay, I’ve got to play nice with the rest of the organization.

Quite a quite a challenge to flip that around. So it’s nice to just have a place where people can do what they need to do. And if need be, and I’ll say this nicely, get into what they need and then go back to work as opposed to spend all day going.

Grant 07:18
Yeah, for, you know, and to be fair, especially as we talk in these different problems, right, problem sets right now, what ends up happening. So I’ve been around this block enough times to know that the things that the pro coders are used to do, it was only them.

Now that becomes run of the mill. And now those folks don’t want to deal with that. So, you know, if you, over time, then it becomes, hey, it becomes a problem. I need the data or need you to do this or add this in.

And to be fair, Dan Kenny, who is a good friend of mine, and in the United States, he is the commercial data science leader for all of the handsome pharmaceuticals, the division of Johnson and Johnson, he makes a comment, you know, especially pro coders, they want to do one turn of the crank, right?

So they want to get it to run. Yeah. And then they want to go away. But if they’re sitting there maintaining it, what happens, they get bored and everybody that wants to do something, they got to do it, you know, and they’re banging on the door, right?

AB
So I’m not a builder or an architect, but – I’ll again take out the dagger from the back in a second – it’s not nice to have to be a plumber or a tradie to have to maintain the thing for the next 20 years. Yeah, it’s nice to be able to drop it, leave and then move on.

Dataiku has traditionally set and its sweet spot is tabular data, Excel, spreadsheets, databases, text numbers, the whole lot. Again, brilliant example that might blow some minds is you can load up and do your data engineering mashing on the fly fixing of data and just say, I’m going to go to lunch, but I want to figure out why my sales or my widgets are working and do all the models in the next hour, even though you probably know that half of them are ridiculous, but do it all. And by the time you come back, it’s all been done. Sometimes it can confirm what you’re thinking and you get internet points.

Sometimes it can actually validate or invalidate what you’re thinking and you might learn something. So tabular is home turf without fail. I know you were just moving into multimodal, into images and rich data, rich media, especially with NLP and audio there a few years ago.

AB 09:40
Can I stretch that question into how far are you now leaning into richer multimodal, deeper nested data asterisk into spatial data?

Grant
I think that’s a great question. And, you know, we always hear about the 80% of the data is sitting in unstructured locations. I think the one thing I’ve seen over the last 18 months, and this was with GPT -35 coming on, I had started to play with GPT -3, uh, when it came out and I was, it was really blown away.

But once it got into, you got into a service and open AI, really releasing that, that’s when pretty much all of these unstructured use cases completely opened up, right? Because this was something that was really hard, took a lot of effort and really expensive.

So you and I were both working on a use case a few years ago, where you’re, in the back corner and doing something and it’s really edge, now that same use case, you know, I can pump that into, you know, llama three one out of the box and I’d probably get better results out of it today, right?

With no fine tuning. So to me, the, what we have seen over the last 18 months has really opened that up. And especially when it comes to spatial, right, where, when we talk about spatial, the first use case I ever heard about, you know, true, really interesting use case for me, I worked personally, I was doing, where do I put business bankers across the country?

Because in a former life I was working for Citigroup and where do I was taking all of this tabular data and trying to bring that together and understanding, but think about that, you know, that same problem.

Now I can actually just show it them. I can show something like the map, right?

AB 11:44
…and say here’s the continental USA, how that indicates density, coverage, yada, yada, fine with population and GDP, where my next five pins are going to go. It’s going to have a red hot guess and chances are it’ll get pretty close. Pun intended. It won’t be all. It won’t be hallucinating. It won’t give you all in Alaska, all in Hawaii. It will actually give a reasonable answer for the human to then come along and assess and figure out whether that’s an excellent guess. A lot of questions might get you there. By the time you’ve, you know, had your first cup of coffee for the day, problem could well be solved.

Grant
And again, I think that is the key when we’re talking about this, is the lot of the legwork, especially on the spatial side that you would have to do to get this stuff ready for a traditional machine learning model, even, you know, 18 months ago, two years ago, is now gone.

AB 12:47
So even doing boring – adding lat-longs to a tabular dataset and starting to say, okay, now draw me a map, now get, you know, a few years ago, that would have been incomprehensible. The data chain would have been dodgy, hard work, and three weeks later, you would have been tearing your hair out.

Grant
I’ll give you a good example. So I was working with an airport client here in the region. And part of the proof of concept is like I needed lat longs for a number of different airports across the region.

I gave it to the, I just pumped it into the LLM. It’s like, we don’t have to go search for that data. Obviously it’s out there someplace, but it’s sitting inside the large language model. And you just ask, here’s your row of data.

Give me that answers in and around it. So for me, especially when it talk, when we talk about that low code, no code user, and that use, and that was me at the beginning, a lot of beginning of my career as an analyst is how do I, as you say, how do I get my job done and I leave and I go off, right?

So the LLMs have really opened that up. And especially for any sort of GIS analyst at this point, you know, it used to be, to do this sort of work, you’d have to go have an Esri or an ArcGIS to do a lot of that kind of low end work.

Yeah, we saw a little of that visualization come in with the tableaus and clicks, and I could start to overlay a couple of maps, but now. I can’t, we can skip those tips. Yeah, you can skip them. And that’s amazing. That’s amazing.

AB 14:27
Yeah, I love the fact that GPT 3 .5 was good, was great. You could ask it to write the entire essay, all of what it wouldn’t. Go crazy by the end, it could do three, four, five thousand words and keep on point, but it was slow.

You could almost watch it as it was forming the words. At the start of this year, we basically went to Crazy Fast and I must say that’s been another epoch that we’ve talked about here on the show, previous episode, where ironically now it’s absolutely human to try and catch up with what you just asked.

You can say give it to me like this and bang. Okay, give me a few minutes now to figure out what you just did and ask the next sensible question. Before you had time to formulate the next sensible question, you almost had time to think about five questions in terms of going so slow or smaller chunks.

You can now on Anthropic Claude: You can say write me a web app on the side that does this and I can deploy it to blah, blah, blah. That almost is three months old, that kind of technology. We haven’t seen the same from OpenAI. Sure, they’ll all be copying and releasing any day now.

I think to your point, these epochs, the epochs are increasing in, you know, decreasing the amount of time between them, right? So it used to be two months. Now we’re seeing it every two weeks.

AB 15:55
Yeah. The chat leaderboards for LLMs are taking off. They have subcategories in them for math, for text, for understanding. They haven’t yet, although I’ll start a petition or I’ll start a bit of a trend for spatial, but certainly spatial in has, it’s got a few meanings, so by means happy to take any of them.

One of course is that geospatial mapping, laying out, understanding concepts. You would have to run to a graph database earlier to figure out this is connected to this. You don’t need to know the highway between town A and town B, but there is a line between them.

There are relationships between these dots of the map. They’re not just random noise. We’re now going to not just two -dimensional spatial, but three -dimensional spatial. What is behind that? Can you figure out what’s behind the chair in this image?

Things like that are now getting towards possible. Is data Riku going down the path of open format, open APIs, and connecting to anything and everything? Where is its special source? Where’s the sweet spot that it’s coming up with its, I guess, architecture?

Grant
And I think that’s a, you know, for us, and this has been a long term strategy. So you can go back 10 years from our founding. And at the very beginning, it has always been, how do we make the best use of whatever compute resources, you know, storage is available for the client, because our secret sauce is getting every people, people together, and actually being able to use and do things.

So when we talk about, especially, you know, open source capabilities around GIS, if we’re talking about data sources, if we’re talking about LLMs, there is a there’s value, especially for many of our enterprise clients being able to do all.

So right now, I’m working with a customer in the Singaporean government, they are hyper, you know, hypersensitive about being in the cloud, they need to be completely on prem. Well, you’re not going to be able to use open AI, right, that are anthropic, but you are going to be able to use something like, you know, llama three, one.

Yeah, right. So yeah, so for us, it’s playing all of the fields, knowing full well that, you know, as a as a technology, obviously, we, we sell our wares into organizations, but we make use of a lot of open source capabilities at last count, I think we had about 349 open source libraries, packages and applications as a part of our acknowledgments.

But that also goes to our ethos is, you know, hey, we don’t necessarily we’re selling our wares, and we’re not open source ourselves, but we also have to be able to be as open as possible to others. I know just from an ethos perspective.

Yeah, you know, if, if you go down, if, as I tell many, if you go to scikit learn, it’s a website, and you go scroll down to the bottom, there’s all, you know, here’s Microsoft and Nvidia and a couple of others.

Grant 19:10
Here’s a little known fact about Dataiku: both it and scikit learn started in Paris. That’s where all where we started. And that’s where our four founders, everybody knows each other that ecosystem. So so again, I think for us, and, you know, to be fair, anybody, you should be open to both those closed proprietary and open source.

And I see the value in both. But, you know, as we’ve seen, this stuff’s moving so fast, you got to be able to go both.

AB 19:41
That’s it, the rate of change. Yeah, so by staying agnostic, by staying ready to just embrace and roll with, you know, all of the different possible options. I can recall a version of data IQ I play with early installed locally.

I guess that’s still achievable, but not as the main course. Is the mainstay cloud one, two, three, or is it, yeah. Still both. So again,

Grant
So we now have cloud. And again, I think this is indicative of the entire ecosystem, right? So you have to be able to do both. Some organizations have completely gone to the cloud. So one of our partners, Databricks, or two of our partners, Databricks and Snowflake, they’re both cloud -based systems and they don’t wanna do anything on -prem.

But we as an organization know that you need a cloud instance for those who need to start and go. And guess what? You can be up and running in 20 minutes and starting to do your work. Flip side of that is you may wanna run on your local machine, as you well did.

And then you can run in the cloud. So it just, and run on a self -hosted or even on your data center. If you’ve got a Hadoop instance still out there and gotta help you, if you do.

AB
OK, so, but COBOL is finally out, is it?

Grant
Yeah, it depends. There’s still people using it. I don’t think anybody’s doing anything new.

AB
I did end up playing with a COBOL, a client who was using COBOL. And my first question was, hang on, are we talking green screens or orange screens? I said, oh no, orange. Oh, wow.

Grant
Oh, yeah.

AB
…there is hope. It was pretty rough. That was a, you know, 30, 40 year old bit of technology that should have been upgraded, you know, 37, 27 years before. Good claim fun. What are the sort of things that you’re playing with?

What are the new things that expand your mind? I know, you know, the data IQ realm was one of the tools six years ago that really, you know, showed me that there are much faster ways or more the point, I’m very happy to bury my own, my old code knowledge, keep it as a pattern or as a, yes, I used to know that language back would all that technique backwards, but I was more than happy to say goodbye to skills in favor of reaching for the next skills.

One of the things that really have expanded your mind and, you know, mind equals blown for you.

Grant
Mind = blown for me, too. I think right now things so funny for me is one of the more interesting things I’m seeing right now is digital twinning. So digital twins were the idea and the concept to have just, you know, I’m mimicking whatever my infrastructure is and how do I start to interact with it.

GHad a great conversation with John Blick, who is one of NAB’s kind of thought leaders in this realm, probably about two years ago, and he talked about how do you encompass an entire knowledge base of someone into and make that available.

In effect, he was talking about this concept and idea of LLMs maybe a little bit before his time, but this idea of how can I encompass an entire, you know, system itself or even a person in a way that makes I can start to interact with it.

To me, that’s probably the most interesting aspect I’m seeing right now.

AB 23:18
…as in a replication of a city, but as of a… It could be both. A systemic. Yeah.

Grant
So if you think about it, I can digital twin my network, I can digital twin a city, I could digital twin an oil refinery, I can now even really getting into the concept of digital twinning an individual.

So think about, yeah, that’s the crazy part about all of this. So for me, being able to encompass all of that makes for an incredibly interesting. So you start thinking about how do I interact from just to kind of play in the GIS space, disaster recovery.

A tsunami hits Northern Queensland, right? You know, what infrastructure is impacted? You know, what are the chances or what are the probabilities that a substation is going to fail in this area? We saw this in Cairns, what?

Probably three or four months ago with all of the flooding, right? You know, a lot of what’s going on and what’s happening is being able to look in and understand the impacts of those well ahead of time.

We’ve seen that, especially in, if you go into SCADA systems, into places like oil pipelines, refineries, going into electrical systems, these are things that have always been there. Yeah, yeah. So to me, yeah, go ahead. This is going.

AB 24:47
Beyond theory, I know in manufacturing circles, having sensors in places using AI to do outlier, when you have a bad day, you can circle on the chart. That was a bad day. Tell me when we are approaching a bad day again.

But those are full of sensors. They’re already connected by virtue of they’re the things that are turning things off wrong. How do you translate that up to, well bank sized or system or enterprise? Is that still, here you go.

How many years in the future are we talking or is here in?

It’s here. It’s now. So a great example of this is Western Digital is one of our customers. And they have a high speed assembly line, a manufacturing line of hard drives. So they have computer vision cameras sitting over the top, going through them.

And it’s immediate, you know, just being able to use the sensor, being able to do that computer vision, understand if there is, you know, some sort of circuitry that has, you know, a defect, being able to immediately pull that off the line.

That’s here today. There is, you know, steel manufacturers being able to understand based and this is all based in, you know, chemicals and numbers and all of that being able to understand, you know, what the hot roll is coming off and what is the purity of that steel and what’s necessary.

That’s an, you know, that’s being done right now, you know, by just modeling instead of somebody having to take a measurement.

AB 26:21
Yeah, so rather than having to rely on, yes, years of knowledge and a bit of a finger in the air and feedback that follows information, being close to real time and change the settings.

Grant
That’s, you know, you said like, Hey, without years of training and experience, I still think that that training and experience is still incredibly valid if it’s kind of the fact you have to know what the underlying math is to understand, you know, what’s impacting your problem, right?

So there’s still, maybe I don’t have to, it’s in many ways, yes, a lot of engineers can sit down and still do, you know, calc three and differentials and all of that good fun stuff. But, you know, there’s a lot of applications that can help.

AB
So it sounds like humans are rising up the knowledge curve and passing more and stuff, more, more things down into the data realm. What are the, what are the roadblocks to that? Data quality, data governance.

Same old, same old. Or can we just go straight to the LLM, a enterprise LLM and have it, have read everything that our organizations done and give a semi -decent answer to a really dumb question.

Grant
Yeah, well, again, I think, you know, it’s always garbage in, garbage out. Right. So, and to be fair, what I’ve seen from the LLMs so far, they don’t particularly do well with, you know, here’s a tabular set of data and go figure out the numbers.

If you upload something into an LLM, what is it going to do? It’s going to try to do a bunch of statistics and math and try to create those insights, right? Those are the sorts of things we can do anyway, but they’re not as great with math.

And, you know, with your context windows being so small, you think about a, you know, a good size Excel spreadsheet that would be beyond pretty much the context window of an entire model, right? Now start adding 250, you know, 250 sheets to that or, you know, doing a year’s worth of data on it and you start to up.

So to me, there’s still value there in understanding kind of both senses of what’s going on and what’s happening.

AB
Yeah, no, their windows are expanding rapidly, which is nice, but they’re definitely not database size. There’s a reason why we have large stores of databases and highly redundant. But I can recall that DataIQ would always take a look down in columns and have an educated guess at what things were.

Can it start to do that with richer datasets now? Can it sort of be looking ahead and saying, so file -based systems and interacting data and pulling that into, is it a central file format that it uses, or it just describes what it sees and the human to come along?

Grant
It’s a lot of description of what we’re seeing. So let’s take the concept of structured versus unstructured. We’ve been adding additional functionality into the tool to understand, obviously, the descriptive statistics, min, medium, mode, max, standard deviations, IQRs, all of that good fun stuff when we talk about numbers.

But what we’re seeing now is we’ve been adding this functionality as what you do with unstructured data. Because typically, that has been, what is the data drift inside of a column of text? That becomes an interesting question.

That’s impossible to ask.

AB
would be a rough one.

Grant
…but that is now, you know, available, right? So, we’ve got, you know, part of being able to understand an LLM is to evaluate the drift and what, you know, what it’s responding with, right? So, to me, this becomes the interesting aspect is now a lot of that feedback, you know, turning the unstructured into structural.

So, whether that is, you know, describing what it is or describing the governance around of what it is. So, you made the comment, do we need to know data or do we need to do a data governance? Guess what?

It’s always there. It’s always been there and it’ll continue to be there. I tell this story when I’m on stage and I ask, I tell, I went into a customer’s office and this, you know, CDO is telling me, you know what?

We’re not ready for kind of AI, machine learning, predictive analytics, because we can’t get our data right. We need to get our data right first. That was 2014. You know, that was 10 years ago. And, you know, it’s still the, I still get the same feedback from folks.

So, it’s an ongoing problem. But it is a show.

AB 31:38
I’ll not name an organization that I had contact with for many decades who was on, let me go, their fourth version of an enterprise data warehouse. That’s a fun story. Version two should be good, but version four, hang on, at some point in time.

Grant
Well, I think I did it this way, right? There’s a lot of, and this is one of the things if you, the average data executive is in seat for roughly two years before they’re moving on. So it’s roughly about four months as the standard deviation.

So if you think about it, the average, uh, that, and it takes you a year to be able to get to the nuance, to get the nuance, in effect, most people are going away and you start to understand why that is the case because people, it’s much easier to discuss, Hey, how do I get data from one location to the next?

It’s a solved problem, right? It’s an expensive problem. It’s a long-term problem and it’s a long running problem. But no, if, as you say, if you’re going through your fourth one, there’s a lot of people that are still going back in time and still trying to get value out of that.

AB
Is there a good news story with the Chief Data Officer, are they moving on and upwards or are they running for the hills and saying never again?

Grant
I would say, um, I think for most organizations, the CDO, um, and now a lot of times you’re starting to see if chief data and analytics officer, we’re starting to hear noises about the chief AI officer, all of these particular roles, they’re coming, they’re starting to, they’re not going anywhere.

I think the, especially for the CDO there, I’m starting to see them more underneath a CIO or they’re going to the office of finance. They’re not necessarily reporting all the way up. So for them, it’s, you have to show your wares.

Maybe you don’t, you know, you’re close to the table, but you’re not at the table at the moment. So, but for those CDOs, those CDAOs that are proving out value, guess what? They’re the ones that they’re the ones pushing these AI projects, you know, whether we’re talking about things in the spatial areas, such as, you know, doing an, you know, map analysis, computer vision, you know, even things as simple as, you know, boxing and understanding, you know, the number of cars in our competitors, parking lots, right? And all the way to, uh, the flip side of that, just the basic, how do I get data in and data out?

But you have to have all of that in order to have a self -sustaining data and AI strategy. And if you don’t, well, you’re going to be looking for a job in, you know, 16 months, right?

AB 34:43
I hear you. Last two questions were actually good around the topic. I think you mentioned it there for a second. Solved problem. I guess in our look, our both our bids, I think we paint on the gray hairs every morning, correct? Yes, they’re not really gray hairs. Good, excellent. There’s a lot of belief in starting from scratch. There is a good percentage of our population who says hardcore, no, notepad, insert tool here. I shall reach the book.

I’ll do it the hard way versus let’s skip 90% of that. I’ve been using the metaphor of God of the days to have to learn how to understand how the mystery of an internal combustion engine works before you can drive a car.

I just got rid of my last internal combustion engine vehicle, which is good, because when the change or light came on, I think it was the same episode. My mechanic was off the same mold. It’s like, you know, Jerry, Jerry, Jerry was, you should be stopping by the time this light comes on.

Then why does the light only come, there’s a level of understanding that is either you do want it and you go back to first principles. I know many people who enjoy that, but I’d love your point of view on the new benefit of just saying, given granted, if there’s a red button for it, I’ll press it.

If there’s no button for it, I’ll just ignore it. I just want to do the thing and I want to do the thing faster. Is that the kind of world that you were saying more and more?

Grant 36:19
I think it is, um, I talk, you know, it’s this concept of build and buy, right? So do I build it myself or buy it? And I did a presentation, you know, a few years ago, talking about that. And to me, it becomes, is it a bespoke?

Is it something that is going to give your organization competitive advantage? You know, is it something you can’t readily find out in the marketplace? Guess what? Go build it. It’s, but there are so many solved problems, Andrew out there that I’ll talk ETL for a moment, there are plenty of tools to do ETL and, uh, that will do it quickly, but some people will die on the, die on the Hill that they need to write it in SQL, right? That’s not, you know, that doesn’t return value to the business.

AB
No, it’s an admirable sideshow to the important superhighway that’s right there to your left.

Grant
Yeah. But again, if I’m doing that work, it makes sense for me. And I think this is where, if I am a leader inside of an organization, I have to understand what is valuable to my organization versus necessarily what is valuable to my people.

I’d love to have that interaction so that they’re the same. But if your team is doing nothing but building stuff, what else could they be doing with that? And guess what? If you can’t, then that’s going to be a problem.

AB
Speed is of the essence as we’re saying the gap between epochs in the field that we’re thankfully being able to, you know, take along the side. The field is reducing down to less players doing more of the things with a healthy open source copy paste of, you know, teacher student models actually models.

Grant
I got a good question for you. Do you think we’ll see more or less LLMs two years from now?

AB
Hmmm, there will be more by number. I think those top 5-10 companies will still be the only ones who can get speed to market. There have been a few initiatives of universities trying to pull together open source funding to, you know, do some mega training on the big foundational models.

I haven’t seen those come to fruition. And if they have, they definitely haven’t hit any leaderboards. Though I think it sadly is in the realm of Nvidia shares are probably the way to go. It’s not financial advice.

Grant
In a gold rush, you know, so pickaxe picks axes and shovels.

AB
Yeah, because we’ve gone beyond how to measure it, centuries eons of computing time to train these things. We’ve kind of run out of content. There has been a side discussion of what happens when AI starts to have to read previously written AI as new content fodder and is that gonna start a vicious cycle and race to the bottom?

Probably not. I dare say with having read the entire internet, we can do better with what we’ve got.

Grant
The average LLM sees the same amount of content that a four -year -old does. So think about that. So all of it’s been ingesting, you know, for everything, right? Yeah. So to me, that becomes the next, the phase of the LLM is computer vision, putting eyes on it, you know, cameras, walking around and understanding.

Then we open up a whole more can of worms. That’s exactly it. But if you think about that’s in effect what we’re getting today, what are we going to get in three years from now?

AB
Well, we’ve used, we’ve used the metaphor of the early LLMs, sorry, not LLMs, the early BERT. So I think we’ve forgotten half the name of these things. Yeah, that was the bi -directional. Transformers, yeah, CNNs, all of those. Could finish a sentence. It was handy, but it wasn’t very, wasn’t great. We quickly got to large models were as good as a primary school user, and then a middle school, and then finally a high school.

I think it’s safe to say we’re at academic tertiary level asterisk in many fields. But to then roll that metaphor back to, even with all that content of only having, you know, a preschooler’s level of content absorption, being able to extrapolate up to those levels.

Yeah, okay. I wonder what does happen in five and 10 years time when we’ve got, yeah, I’m drawn back to science fiction, as always. It’s 2001, Space Odyssey, HAL, not irrespective of the letters were one before IBM. We’ll leave THAT controversy a different day. But the Arthur C. Clarke’s metaphor there was that the creator of that supercomputer had spent almost a decade with it teaching, training it, talking to it.

So perhaps we just do need time and time will resolve itself. And the funny thing is less would be my prediction though. Surely we’re not going to have the plethora of copy paste. Each student decimated smaller versions of the large models will continue.

But I don’t think we’re going to have a random entrant from you or me in our garage going to take over one of the leaders.

Grant 41:58
I don’t think we’re going to keep the leaders, but one of the things I talk about is, you know, wisdom of the crowds with people. And I talked to folks as the price of the LLM API tokens will fall because they will, you know, that’s just the nature of, you know, unless you, unless energy prices go through the roof over time, the Silicon will decrease in price and we’ll be having, you know, H100s and A1, we’ll be able to get them a lot cheaper. What will happen is the LLMs, they don’t, LLMs don’t hallucinate the same way. Right. So if I can call the same question to four LLMs and then have a separate LLM, so in effect, chaining them together to get a response, I could probably be pretty confident.

So to me, that becomes that wisdom will be one of, you know, just how we use an ensemble of model machine learning models, we’ll be using an ensemble of queries back from an LLM to understand.

AB
So today we would have an LLM prompted, you are an academic, I want you to respond in this tone, having those being persistent agents and asking your own personal AI body of knowledge and then have them battle out for who gets to answer.

Grant
Yeah, well, even just being able to synthesize what four of them talked about. Right. So, and being, and then maybe fine tuning one that basically starts to resolve out when, you know, if there’s discrepancies.

So to me, that means there’ll be interest, even if you look at the biggest models right now, uh, they’re not just one model, they’re a collection of models underneath. Yeah. And we’re going to see more and more of it.

AB
Last question then for you, if I can. It’s a big one, which may not have an answer.

But I guess the last few topics, last few questions have been leading up to the concept called RAG, which the author acknowledges is if they had known what a concert was going to be, they would have given it a better name.

RAG is Retrieval Augmented Generation, which is not any help either. Essentially, it’s keeping your hot, hot, and your cold, cold, keeping your LLM changeable. And when that gets better next month, you can switch it out for the next one, but bringing your own data to an interface and then having you interact with it.

Right now, large models, any foundational models, don’t have a concept of, well, you or me. They’ve got a concept of people like T-shirt, green background, green T-shirt, chatting on a podcast. They won’t make a picture of Grant or myself.

They’ll make, we can prompt them to get close to Will Smith eating spaghetti, but it’s never, make me a video of Will Smith eating spaghetti. It’s, man, these features, spaghetti on a plane, go. RAG is the thing that actually brings those together and lets you have your own data set safe, brought forward, and then you keep on switching out what is the latest and greatest front end to be able to interrogate.

Open question to you, and is that something the data rock crew can help with to smash together all that digital twin of your own organization? Smash together with the latest and greatest.

Grant 45:29
So we’ve actually, RAG, when I talk to customers, RAG is in effect kind of, you’ve got your base answer, you’ve got your RAG, and then it’s out to fine tune if you really need to get it right.

Fine tuning is an incredibly expensive to do, but for most organizations, RAG is good enough. So last year, even with Dataiku, we worked with a LG Chemical in Korea for their health and safety data. So help it, you know, build a RAG inside of the tool in order to query that with their own external application.

So for us, this is, I see RAG, especially being a way to provide a lot of value very, very quickly to an organization, being able to ask those questions, being able to ask those questions very fast. I would say this, there is a question right now of context length, because in order to use a RAG, you have to, in effect, break up different elements of it.

So you’re chunking them or batching them for most part. And that, and in the end, what you’re asking it to do is do some, you know, basically do a Google search inside of it. Here are the five things that approximate your question, give that to the LLM and then the LLM responds.

It’s not a perfect, it’s not a perfect way of dealing with things. So for me, this is one of those where I’m gonna watch this space. We absolutely can do it. And I don’t think it’s going to go anywhere, anytime within the next two years, just because context length.

But is it the be all and the end all? I don’t think so. I think it’s a way point. It’s gonna be a, you know, it’s a way point in much the same way as settlers are moving, you know, in the United States to use my, they moved West and you’d have different outposts RAGs, just gonna be a really significant.

Grant 47:47
So for me, that’s, that’s what RAG is right now. It’s a, but it’s a way to get a heck of a lot of value very, very quickly.

AB
Yeah, but not the final result, gotcha. .

Grant
And more important thing, restricting down what the LLM can talk about, right? So being able to say this is the data, this is your data set to work with. That is probably one of the most important aspects.

AB
So this is your corpus.

Grant, thank you so much. Absolutely awesome to have a chat with you. We do need to do a follow-up by all means. Let’s get you back as a regular.

Grant
Sure, absolutely. At any point in time.

AB
Where do people find you these days? Or where would you like people to find you? Or any call to action you’d like us to leave us with for personal or data record?

Grant 48:34
So, if anybody’s interested, you can find me on LinkedIn. I’m typically posting something, you know, every few days or so that I find interesting. You may or may not, but that’s the best place to get in touch with me.

If you are interested in this space and your organization is trying to figure out, Hey, what do I just do with analytics all the way up to how do I get everybody working together, dataiku.com and feel free to try out, retrial of the tool or even if you just kind of a hacker.

Uh, I know Andrew, when you were going through school, you can, you know, you can download Dataiku into your, uh, your notebook and you know, work on it locally.

AB
It’s the closest thing to magic there for many many years There are other magic tools like LLMs being able to do all your code for you that you know I means that’s that’s the one to watch of a full -on architecture enterprise tool versus is there a magic? Invoke any of my voice assistants right now, but you know what happens when even that gets abstracted out But Dataiku is a brilliant tool and I heartily advocate for it. It saved my bacon many times.

Grant thank you so much for that. Enjoy the rest of your day. Cheers for the chat

Grant
Cheers. Thanks so much, Andrew, for the time.

AB
No worries. Well, that’s all of us from SPAITIAL. We’ll catch you next week with another special guest from a different part of the world. We won’t be daylight next time I’m chatting to you. For all of us here at SPAITIAL though, thanks for your time and we’ll catch you on the next episode.

Bye-bye.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

To absent friends.

The post Episode 024 – Is low-code Spatial AI possible? With Grant Case from Dataiku appeared first on SPAITIAL.

This week is a free-flowing conversation between AB and Arseniy Sergeyev – a true ‘young gun’ of the spatial computing world – and the co-founder of a startup working on Adaptive User Interfaces (AUI).

AUI aims to create interfaces that adapt to the user’s context, needs, and preferences, providing more relevant and personalized experiences.

Arseniy and AB discuss the current limitations of generalized interfaces and the potential of AI, specifically large language models, to understand user context and generate tailored interfaces. Arseniy shares his vision of a future where the world becomes the interface, with smart objects and environments adapting to individual users. He outlines his startup’s approach, starting with specific use cases and then building a foundational system for broader applications.

Heads up for a massive opportunity: as Arseniy heads to San Fransisco this week for meetings and networking, do note that he is looking for a technical co-founder – details in the show notes below!

https://www.linkedin.com/in/arseniysergeyev

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/VDGRFUI_Khc

Arseniy discusses the current limitations of generalized interfaces, which are fixed and do not adapt to individual user journeys and contexts. He highlights the potential of AI, specifically large language models, to understand user context and generate tailored interfaces based on free-text prompts and user data.

Arseniy shares his vision of a future where the world becomes the interface, with smart objects and environments adapting to individual users. He envisions a system where interfaces are generated based on user requests, and physical objects and digital interfaces seamlessly blend, providing personalized experiences.

Arseniy outlines his startup’s approach, starting with specific use cases like food and then building a foundational system for broader applications. He discusses the importance of understanding the user’s context, preferences, and available options to provide optimal solutions. Arseniy mentions potential use cases like choosing food, conference matching, and wine selection.

Arseniy emphasizes the importance of decentralized data processing and user privacy, aiming to create a user-centric system where individuals control their data. He discusses the potential risks of centralized data collection and the need for a more secure and privacy-focused approach.

Arseniy discusses potential partnerships with technology providers and the need for guidance and mentorship from experts in personal AI systems and human-computer interaction. He estimates a timeline of 5-10 years for significant progress in realizing the vision of adaptive user interfaces and personalized experiences.

AB
Well, g’day and welcome to SPAITIAL, This is Episode 23 – coming to you from deepest, darkest, emphasis on darkest in my part of the world. It’s a bit late at night. Why is it late at night? I hear you ask?

That’s because I’m talking to Arseniy Sergeyev, who is in Riga in Latvia. So in your part of the world it’s just turned over midday. Now I’m basing this timing on the fact that we’ve coordinated this and the Tour de France starts in a little while.

So I’ve sort of got my time zones really good, but Arseniy, welcome. Great to chat to you. Oh, look, I’ve got to say, and I’d better spill some beans. This is our second chat.

We were chatting actually at the start of the year, even before we were chatting about doing a serious chat, we were literally having a good open conversation about what is Spatial AI, where’s it, “what are you doing, what are YOU doing? Oh my God, that’s so cool.” The offer was made, I think, way back then.

And it’s great to be able to have time to circle back and actually find out what you’re actually up to. What you are up to is an acronym that is new to me. The words themselves aren’t new. The way it’s put forward is so, the acronym is AUI, Adaptive User Interfaces. And I’m mad keen to find out as much as I possibly can about it. What’s a one-liner? What’s the Twitter post? What’s the hashtag? How would you explain it to a grandmother?

Arseniy
Well, if we’re talking just about the adaptive user interfaces, AUI, it’s basically the interfaces that adapt to what you want to do what you need to do. Right? So they understand your context, and they give more relevant things for you to do, rather than just a general universal interface.

AB
That was the idea. So somewhat of the concept of a computer as a general purpose device where the screen changes for what we needed to do and ironically, inputs and outputs were left with mouse and keyboard, mouse and trackpad as the, probably the only physical remnants of here have a keyboard, here have a way to move a two dimensional thing.

But in that same way that to abstract and keep on abstracting even beyond what we’re playing with today, is it a function of I haven’t got enough screen space, so I had to make it adaptive or why is AUI, why is this way to be adaptive and context aware?

Why is this the future? What’s the real gem behind doing this?

Arseniy
Well, the issue with the current interfaces, and I’m talking interfaces in quite a general terms, but of course we are most used to smartphones, we are most used to computers. Any screen that we have is our most common interface, but of course there are more physical interfaces, knobs, some switches that you have on the surfaces of objects.

You have objects, you know, you’re putting a teapot on there, it’s also an interface. But I’m more talking, in this case, in this scenario, we’re talking about adaptive user interfaces in the context of present times, it’s mostly digital interfaces, right, on the screens, websites, apps, where we have.

And so the issue with the current interfaces is that they are quite generalized, so they are quite standard and quite fixed to what you can do in them as a user. And the idea is that the issue is that different users obviously have different user journeys towards what they want to do, and how they want to do it, right?

Some people want to learn more information along the way, some people won’t just instantly do something and just don’t worry about it, but the current state of the interface is doesn’t allow that. And the idea is to enable the interfaces to adapt more to your needs and to your individual context, so that you don’t have that hassle of going through someone else’s user journey where your user journey is different,

With AI, of course, this is finally more possible, and specifically with LLMs as well, because we can get a better understanding of the context of the user just from the free text prompts, right?

And that allows to get even more information, which wouldn’t be just static metrics and adapts already based on that.

AB 04:56
Wow. So being able to harness the interactions that are going on with the user and any system as part of that natural interaction, also being able to bend some of the, you know, de -emphasize, focus, highlight, deprecate, but also morph and change to be context switching.

That’s pretty phenomenal. Is this the holy grail? Is this a pipe dream? Is this being done in small elements right now? Or is this like a new thrust of a design mode that is everyone’s hoping will one day come out, but, you know, we’re in alpha mode right now.

Where is it on its own journey of context switching?

Arseniy
Well, I feel like that’s the logical next step for the computing that we have. Okay, so because there’s quite a big trend in terms of actually, there’s a big trend towards the computer disappearing. So the computer, the computer being all around us, the world being our interface, that’s actually also the future downloads.

And the current state of things is that it’s just it’s just very early in the beginning, there are a one example is brain AI, they’re doing really magical work, it’s really beautiful, and how you go from a prompt of what you want to do, towards the options you’re presented to fulfill a task.

So if you want to eat, you want to eat for a sign up for dinner, they have this demo, and it just generates an interface for you, it’s a generative interface with them. And so it generates you an interface with all the products that you need with a recipe, and you can buy through that.

So it’s not static interface anymore. It’s already a generative interface which generates based upon your request. And that’s already happening. They recently launched together with a T -Mobile, I think in Germany, and I’m mistaken, it’s a global company, but I think it was in Europe.

But the idea is that they made a phone specifically is a phone without the apps. It’s an interface that adapts to your request. And so there are some projects that are already working towards that. And there are a couple more that basically prepare the grounds for the data that is necessary to understand the context of the user and then adapt and trace to them.

So now that the grounds are being set, I would say, and with AI, with all the generative AI specifically, it now allows to really get into the context and not only get into the context, but also do something with adapting to it.

But otherwise, Facebook, Google, Amazon, they’ve been talking about ambient computing, ubiquitous computing for quite a long time. And then Google’s advanced technologies and projects department, they already started doing some work years from now, before now, so they’ve been experimenting with sensors, with radars, how you can mix interaction from the physical world with the digital interfaces.

And they are adapted to your context, right? So they’re not just fixed.

AB 08:08
I come back to the hardware sensor stuff. I know that’s something I’ll maybe throw dozens of questions at you, but just rolling it back a fraction, is it one way of looking at it potentially, is it the purposeful opposite of muscle memory?

I’m thinking of we’ve gone through the era of icons everywhere, dropdown menus everywhere. I’m now thinking of the Photoshop sort of esque suite of palettes everywhere. And if it could stop, please moving my palettes around, that’d be great.

But muscle memory is probably the thing that saves you. You know that’s there, if your keyboard, if so, if your fingers don’t have the muscle memory for the keyboard shortcut, at the very least your eyes can latch onto the icon, or you know that it’s the third thing down in the fifth thing or it’s tucked away under here.

All that’s great. And it’s great to be fast and proficient, but muscle memory is hard to get. And when they change things based on you every alternate season just because, yeah, that’s quite tiring.

Is this trying to be, ‘don’t try and learn something, I’m trying to be one step ahead of you’? So in the example of baking lasagna, not having a predefined ‘oh, they’re doing a recipe quick, bring up the recipe interface’, but all they’re saying to be doing something which will require a list over here, a timer here and, you know, pointers on the screen to say, put that with that.

Is it that level of low level fundamental building blocks that rise up to be smart, or is it actually today that mid level of, we have a recipe tool, we have a scroll list tool, we have a, and I’ll just mash together the mid level components until user is satisfied.

Where’s it sit on the perfect, lots of little good bits or just on the fly, it comes out of nowhere and, you know, lets you just do your job.

Arseniy 10:03
Right. Well, there’s a fine balance to strike so that not to make people learn something new, because just because we invented something new, and you know, you need to learn how to interact with it. That’s how many devices and interfaces are being created.

And it’s suboptimal. The idea is, first of all, is to reduce the number of actions that you need to do to get to your desired result. Because all those are micro frictions, right? So you know where the icon is, you know where the buttons are, but you do need to do those micro steps, right?

Press this button to proceed to the next step, right? What do you want to eat? All right. What do I want to eat? I select the meal. All right. What do I need to buy the next step in the journeys? You know, choose the groceries, add them to cart, pay.

So those are all micro steps that cause micro frictions. And yeah, removing those, this muscle memory, like not having to do it at all, is already a good thing in a sense that, you know, we’re not starting that much just by reducing those steps.

We’re not in a combination of different use cases in the day -to -day life. We are already much less stuck in the devices that we are at. We can raise our head, raise our eyes from the device, already a bit more.

That’s one thing. But on another hand,

AB 11:08
Wiggle your mouse to find your cursor. Where did I leave it? All those milliseconds not only add up, but they make you weary. Yeah.

Arseniy
Exactly. That’s why voice assistants are sometimes very useful that you don’t have to take into your hands the device You just say it if you have it already formulated in your mind, but it’s a different topic But then when we do keep some actions which are you know So you do need to press some buttons or do you need to show some gestures?

To kind of choose to make a choice make a decision And or we add some new ones we do want to rely on the existing muscle memory And that’s important so that we don’t Make it important by kind of seemingly simplifying the mental effort By creating something you wouldn’t want to add additional mental effort of learning something

AB
Gotcha, so fallback to conventions where that actually is the fastest way to get a part of the job done. But sometimes the whole might be radically different in different circumstances. Even if you do the same thing three times, you might get a different interface every single time.

Arseniy
And the question here is, how much back do we go in terms of what is this level of the actions of the gestures that we’re used to that we go back to? Maybe we don’t need anymore to remember how to click on the icons of the apps, figuratively speaking.

I’m imagining that we do need to go back to the most natural ways of how we interact with the things. You know, we’re waving or we’re sliding, basically, all right, those are already kind of screen native interactions.

But it’s just so they’re more natural. Just like, well, we’re getting one dimension.

AB 12:58
Because with 2D screens and flat surfaces with mouse trackpad, they are an elegant solution to have a 2D input for a 2D device that you’re trying to control. Perfect in many, many ways. Hence why it survived 40 something years.

Thank you Xerox (and I won’t get into the Apple debate) but thank you Xerox, Sparc Xerox. But once you remove yourself from having to be a two dimensional flat thing to how bad I rotated and pinch to zoom and twist your fingers on a flat screen, there are good conventions.

But what if you could be released from that? How do you see behind something? How do you grab? So is that part of the mix of already thinking outside the literal rectangles that we’re bound by?

Arseniy 13:49
I would say so, because we can use still those same interaction modalities that we are using on the 2D screens and interfaces on the same pinch and zoom and sliding and scrolling. We can use it more in the 3D world, but in spatial computing it doesn’t make sense, as I’m seeing it, it doesn’t make sense to put the same screens that we have on 2D surfaces to put them as Apple did right now for many apps.

It just makes much more sense for more natural and seamless interaction to have 3D modalities for interaction. You have a table here, you touch the table, you don’t touch a window that says, you know, a table.

For example, you do interact with the 3D objects, you don’t interact still in the 3D environment with 2D screens, as you do, because it doesn’t make sense.

AB
Yeah, correct. And it makes sense to transition from 2D to 3D to have lots of floating screens around. It makes your shoulders tired, pinching to Zoom and doing everything up here. I had some excellent play with the Hololens’ when they came out, version one and version two.

And they were good if I am on the YouTube version. They were in a natural in front of you, but you couldn’t go too far left, right. You basically had this. It was fun to watch different people sort of try and control things up here and hang on the field of view of sensors in a normal resting pose.

It’s lovely to see the Apple Vision Pro have a bit more range. You can be on the couch and just go sort of click, click and it should be able to pick those sort of things up. But to have your, not have your shoulders raised to manage multiple flat screens in 3D space around you versus I just want to have this screen where I would normally have it.

I’m drawn to the, I don’t have it here in front of me. He’s probably a little bit dead, but I am the owner of a beautiful big Wacom tablet. One of the big ones. It’s awesome. It plays as much as this desk does, but it’s glorious because with one big move, you could go from right in front of you to the whole screen would just become your desk and you could be drawing and shading and painting and doing awesome stuff.

But there weren’t many times I went from upright to desk mode. It basically stayed in the one position for most of any given month till it was unwieldy. But that different level of thinking is hard to get into.

And once you’re in it, hard to break out of.

Arseniy
Exactly, exactly. And then come the new best practices for good user experience. You know, you’re thinking about already more physical aspects of the interaction, which is a super important thing of interface design.

AB
How did you get into this? how did you start this journey? Was it the classical, was it from design first or human need first or from I’m a coder and this is broken, I need to fix it? What sort of avenue did you come into this realm from?

Arseniy 16:52
The quick answer is a combination. One thing is that throughout my life I always wanted to create something on my own, something to improve the world. Every Apple event was like a special happening to me watching all those kennels, one more thing.

AB
Yeah, so we’re both wearing the black turtlenecks. We’re not quite, but still, yep, definitely a design inspiration for multitudes. Yeah. But how did you get into this then specific field or how’d you get to see that this is the thing that I desperately want to do?

Arseniy
Yeah, and so basically, I’ve been inspired and you know, I want to create something as beautiful as functional like this and present it also to the world like in a very, in a very powerful way. So it really changes how we do things.

So we do things better. Then I got into in the university, I was in the retail specialization that we got into the era of psychology, people and it was super interesting. It was about you know, Kahneman, you know, the classics, you know, the nudging, how do you, you know, different bias, different hero stuff, behavioral things that actually can subconsciously nudge people towards one extra or another.

And it was super exciting, because in the context of retail, you could make the life easier for the users by showing them the choice faster. And for the businesses better, because they just have better chance of conversion.

Yeah, right. And that’s all right. I want to do something with that, something with understanding the user better and improve the experience for them. Then I also take some additional courses, then I, you know, worked as an e commerce with user experience optimization.

So all that kind of came together, building up some skill in that direction. But on a personal level, I think that was the biggest part, is one part of this inspiration. But then another part is just being crazily frustrated by how imperfect our interactions are with the world right now.

We still have to think about how I can do this. It’s like how to do things. And then we even, we thought how to do it best. We considered different options. We took everything into account that has to be taken into account to make it optimally.

We still need to do micro actions, either in an app or compare options with apps. Super frustrating. It does really take focus away from real important things. And that’s when it hit me that it just must not be that way.

And the world should be, it’s already going that way with all the big second and smaller companies as well, startups. But it needs to get there faster. So that’s how I got in there. And the ideas on how to do that have been in my mind, you know, for, I would say, two years, year and a half.

And yeah, in the end, I understood the right tool. It’s been too abstract. It’s too ambitious, too big. But then we started with something smaller, of course, going to that big ambition. That’s the story of how.

AB 19:55
Well, you currently have on your LinkedIn bio, in that classic mode, you have a startup where you are the co-founder. The company name is, well, you haven’t named it yet, and you’re welcome to reveal or you want to keep in stealth mode, perfectly fine to honour that.

So in classic startup stealth mode, are you building, are you prototyping? Can you envisage something that is really there? To have this idea in your head for two years sounds like, I don’t know… There’s a thing called the ‘idea maze’ and a creator or a founder comes up with lots of ideas and ideas are free. They’re cheap. You shouldn’t get people signed NDAs because an idea is worthless unless you go through this process of testing it against yourself for as long as you can.

And the good ideas fall out of that maze. They literally last the test of time and you’re still thinking about them, tuning them over, or they still are that bright light that you run back to even after multiple years.

So this is obviously something that in your brain you’ve probably got tired of, dreamt of, put down for a little while, come back to it. And this is now your sole focus. What is the, what’s this journey?

What’s the startup stealth to non-stealth journey that you can foresee and you’re willing to tell?

Arseniy 21:29
Right. Well, it’s indeed in the stealth mode. It’s I would say it’s like semi stealth, because we do participate in some programs, we do mentioned what we do in the name of the company as well. There’s mostly mostly stealth, not to not to be too early, before we have something solid.

That’s actually, but basically, the idea is to build out the system that knows the user, and then provide output for adapting the interface to them. And so at least showing the area already gets me to connect with the right people with the right competence, who is also who are also passionate about the field of, you know, making the interfaces more personalized, more optimal, in terms of interaction.

Okay, and so the path of that is I’ve been now for a couple years, like, all right, year and a half, it’s been a year and a half ago, it just started in my mind, quite abstract, I started putting it on the paper, conceptualizing of what does it entail?

How do we get to implementing this? And because I’ve been involved in commercializing science, basically, I’ve been involved in a platform that helps bring science from labs to market, mostly by connecting that with entrepreneurs to create deep tech startups.

So I’ve been exposed to the process of how do we identify the science, its level, its core competence of the research team, how do we bring that to the market? What am I missing? Yeah. Yeah. And from there, a big, big, good tool is being able to decompose the technology into its core components, which is your proprietary part, which is, you know, where you could cooperate with something which is open, available, and which is the special source where the real gem of the idea sits. Yeah. And then you can understand where you protect your intellectual property, where you focus your efforts on what your license or sell, etc.

Arseniy 23:21
And with that approach, I decomposed the idea the concept into the core blocks of the technologies. We have 5G there, we have haptics there, we have, you know, some some meta-materials, smart materials that make objects smart.

I’m talking now about the ultimate vision of making the world our interface, right? Yeah, but I had to start with something because it just, you know, it was blowing the mind because it was too big. So that’s what I started decomposing.

AB
“Boil the ocean”. How do you tackle something that you can demonstrate, gets the idea through, but doesn’t involve buying everything all at once and solving everyone else’s problems before you can solve the call problem?

Arseniy 24:02
Exactly. And that helped to narrow down a bit and to understand what to start with. And what I came to is that the world, so the special computer gives a huge opportunity, let’s say I think it’s 700 billion dollars in 10 years or something, but it’s basically special computer gives about interacting with the world as an interface, as I’m saying it, in different versions, either it’s AR, VR, or it’s you know just smart materials, et cetera, smart objects, IoT, but the idea is that it must learn the user first. And that’s what we’re like right now. And for that system when I decompose and understood what the core thing to start with is the system that understands the user, understands their basically preferences, their past choices, their current context on the individual level, and understands the options that are available to them.

And it processes that, it knows what information it needs for any given task in the user day -to -day. And it provides the best options based on that. That’s basically what we’re starting with. Right now it’s in the, we’re testing the core algorithm right now, building it up.

But the idea is that we need to find the core use cases. We have tested a couple of use cases. So then starting with day -to -day it’s like food, like choosing food, and you know what do you need to prepare for it?

Arseniy 25:25
The whole user journey. It knows what’s in your fridge in real time, knows what you like for sure, and knows what’s available around you. And it puts those blocks together to get you the optimal version, the optimal options.

And that’s just one use case that we’re starting with. We also did some conferences matching people properly based on their profiles. Wine choosing as well as a wine lover. It’s just, I’m not a wine professional, but I’m a wine lover for sure.

But it just overwhelms with all the choice when you want to experiment, you want to be too drunk all the time. So a couple of tries, but right now the next step towards going, you know, as we said, out of stealth or like into some real progress is really building a strong team to bring this forward.

And also why it was great to use this opportunity to talk to you because you do have exposure to really bright minds of people who are interested in human -computer interaction in, you know, in different forms and ways.

Right now we’re looking for a core team member, a co -founder, basically, who is, who would help us to architect the solution, basically a solution architect, trouble with background in machine learning, AI, and then probably back -end full stack.

AB
So basically, someone to code this with, to build this with, but also a user experience designer to make this user aware of where it’s going to lead to and not just build an API and say done my job yeah gotcha so that also needs to have the context of what’s coming down the line and a hundred percent buy into the fact that this is going to get gnarlier quickly so exactly exactly

Arseniy 27:01
Because we do we do have quite an early small team right now We just we need to move faster and not in a more profound way to to build the core core of the system right now And also of course the guidance people who are a bit further down the road in personal AI systems and Adaptive user interfaces or on the human computer interaction field Right now mostly about software, but later on as well hardware So this is basically we need guidance and mentorship for those people.

Those are those are basically two main Components for it to move forward really really well as I’m saying

AB
I’m glad to hear, oh, well, it’s good to hear that you’ve got a, you’ve brought out half the known world’s problems by not trying to solve hardware. First, you just stick with software. It’s cheaper. It’s easier. It requires ones and zeros and a battery or 240 volts or 220 volts, 120 volts, not many volts you want. Five volts, this one.

In this early stage, are you also going to try and limit the use cases down to, let’s say, adaptive user interfaces, but just for a smaller scenario of problems? Or are you going to try and do a foundational piece that tries to be that personal context tool with limited outputs for the first few phases of how it can help? Where’s the focus on the software of going strong in a vertical, in marketing terms, or to be going horizontal on the foundational piece so you’ve got your own secret sauce in how you filled up the knowledge of a person irrespective of wine, coffee, lasagna, choices?

Arseniy 28:44
Yeah. Well, we do start with something, with a certain use case. But in parallel, we do want to do the horizontal part. Because the horizontal part will be something that will be really impactful and enable much more seamless interaction with the world.

And that’s the vision what to build. We’re starting with the food use case, basically taking the whole user journey and all super seamless and making it ultra personalized and optimal to the user. But then in parallel, we want to build the core system that will be adaptable to different use cases.

Right. So this one basically will help us learn on, you know, what are the key things. And then based on that, we’ll be able to develop something, you know, already more foundational. And here is that understanding, you know, what what type of strength and what type of competence and the same would need, because this is a lot of research should be done there, right, for developing something foundational.

So that’s why we’re starting with something more specific than to be more informed and more structured when we’re doing something more foundational.

AB 29:53
That will let you, for instance, in food, learn food preferences, likes, dislikes, what you have on hand, what you don’t have on hand. If you’re in a certain part of the world, you will never have X, but you may have Y.

But that may give you deep insight into whether you need a personal context engine, the personal AI to be, you know, this size, mega size, or humongous size. It might give you the first glimpses into whether you can get away with a small little nugget or, you know, gem of an idea, or whether you need to go big quickly and perhaps the same in the large language model lessons learned in the last few years of, yes, language models going from small to medium sized is good, but at some point in time, there is a break point where they just get good. They do things that you can quite think of beforehand. So you might be looking for where are our break points, where is under baked to put the food metaphor back on the table, and where is, yeah, spot on that can actually start to give you the guidance so you can transition to more and more personal knowledge.

Arseniy 31:01
The ultimate vision is to have all the day-to-day things and why food also is because it allows to us to quickly to test people – people eat every day!

AB
It’s a rather common theme amongst, yes, people in this world : ) That makes that makes sense. And yeah, everyone’s got a bit of a preference. It’s nice to have that level of “what have I got on hand?” What’s ahead? Yes, yesterday. Can I have a, you know, cheese and biscuits for the ninth day in a row? Yeah, yeah, no, it’s time to mix it up. This seems like the dream and interface of really bad early apps that you’d take a photo with your fridge and it would come up with given these ingredients, you can make X and Y and Z.

It seems like the ultimate of that kind of tool.

Arseniy 31:50
I would think so, yeah. Because it is already possible. You kind of take a photo, I think, even with ChatGPT, and throw you some recipes on. There’s Samsung Smart Fridge that does the analysis with the cameras of the pretty contents and reminds you to buy something.

It’s kind of already towards that direction. But Samsung Fridge can analyze up to 30 or something products at the same time. They also need, ideally, to be weighed down very accurately for the system to recognize.

So it’s the precision. And I think, going back to our first conversation, it started, actually, it was connected with – I reached out to you, I think, after your podcast about local AI. So it was the statement that there are those large models, but what we’re really liking are those more specialized models based upon those that really know much better how to analyze things.

In this case, it would really help that if we had a model that knows – can much better distinguish between the products if they’re even piled up, not just artificially weighed down. But in this sense, there’s a lack of connection between all of that and there’s a lack of universality in terms of your scenarios.

So you might be sitting at home, you want to cook something or you want to order something. It’s not all connected. You have Uber Eats separately. You have your Smart Fridge or wherever separately. But the idea is to have this user journey of eating optimized for that user in a centralized manner.

So it’s probably a single interface, or if the person uses some service, it adapts based on that user profile, already switching the box with you as well. So that’s the ultimate, indeed.

AB 33:44
Now – you sent me some links, which I’ve done my homework on. So thank you for that. First time to give me some homework. Cheers! I saw some excellent TED talks and some lovely other great concepts.

One here is called pAI-OS, Personal Artificial Intelligence Operating System. Good to hear an Aussie voice on the end of it. It’s definitely thin and starting, but obviously I could see why you sent me the links to these sorts of things.

https://paios.org

I was having this chat with some of my teams today and we’ve gone from being a data scientist where you’d soup up your PC with the GPU and water cool it – and maybe overnight you might be able to bake a model and hopefully not, you know, use all the power in your street.

Then you went to the cloud and grabbed an A100 for 30 minutes and, you know, paid you $7 to one of the cloud providers and did the same thing, just more quickly. But now we’re almost leaving that up to the large language models, the big foundation models in every data domain are the place you go for image, for language, for multimodal, for coding.

And those large language models have no idea about you or me. I could ask it to draw two people having a podcast conversation, facing their cameras, you know, with a turtle tank in the background and yourself with black and white squares in the background.

I couldn’t ask it to draw ‘AB and Arseniy’. It would have no idea what that concept is. So being a general model means it just, you know, knows general things and can make ‘general stuff’ up.

I must keep the people who are trying to recreate the Will Smith spaghetti videos these days, because you can’t type in ‘Will Smith eating spaghetti’. I believe you have to basically prompt him into existence?!

So in that realm of we’ve gone from our own baking – our own models to do our own personalized things: “I don’t just have pictures of cats and dogs, I have pictures of my cats at my dogs”.

We’ve now foregone that to the power of these large foundational models with the asterisk is that we’ve lost that personalized sense of “yes, but it doesn’t know where I am”.

It doesn’t know what my local street offers. It doesn’t know where my closest X or Y or Z is. It has no idea even what the town is that I would name. It just goes, okay. I know the name of that town.

I can tell you facts, but it can’t tell you anything about it specifically. It’s just regurgitating general knowledge here.

Arseniy 36:33
So yeah, or you just you just prompted all the time with all the data you say what do I do in this town? I like this I like this I like this but you have to write it down So you just might might as well give up because it’s too long.

You just figure it out yourself what to do there

AB
Gotcha. This is the untold story of what happens behind the scenes in a lot of demos. So to try and replay it back to you, AUI adaptive user interfaces is not just the interface has smaller blocks, perhaps conventions and components where needed, where that is an elegant solution, but on the fly, being able to use Gen AI to say, I need something that does vaguely this, draw me one.

And if the user swipes left or swipes right, try, no, that doesn’t solve the problem. You could reiterate it again. But that’s the visual, not tactile, but that’s the bit that the user plays with. But the important mirror to all of that is having that context under the hood to be able to know what they want, try and guess what they might need, infer their location, infer their tasking, infer routines, infer person. That’s drawing good amounts of data about a user. Is that challenged then to try and come up with a standard or a syntax, or is that a someone else’s problem? Is that personal AI context your core problem solving, or you can outsource that to someone else?

Arseniy 38:03
Well, here, the approach that we want to take is not to store the data ourselves, not to be able to look into that for each of the people. Because with the data, we know the key question for everyone is, you know, who gets to see my data, etc.

And the key here is to keep it if we’re talking, if we’re talking popular keywords, buzzwords, we say decentralized manner. I’ve been in an interesting meetup in London recently on the centralized AI topic.

So it was blending a bit AI and blockchain, but it was a it was an interesting perspective on thinking we have AI on devices. So the idea there is to have at least bigger part of the processing of the data and storing of the data on edge, either on the user’s device or on the device that is, you know, there’s processing it what they’re doing.

And it doesn’t go to the cloud, or if it goes to the cloud, it goes with the apples, presume, presume we say approach that it’s, you know, private cloud, as they call it. So the idea is that we you know, there is no central one entity that gets access to this data.

Because imagine if you have all the behavioral and preferences and pasteurized data for all the day to day things for all the people in the world. This is the new empire. This is the new I would say, that’s, you can conquer the world.

eah, you can come to the world with that. And here the idea is to take more user centric approach so that they control who they give the data to. And they they have the control over it. Thank you.

AB
Nice. Now, as you were saying that before you said, AI with Apple intelligence, Apple’s new rebranding of AI, clever, also painful. But then again, they did have the eye for 20 years. So we’re all used to the eye, everything with eye rolled so many times.

This does sound like it’s it’s a great fit. I’m thinking when you said that privacy and security in that regard, my first thought was, was it the private enclave on your devices? That is the classic Apple portion of your phone, your personal computer, watches, has it reached earbuds yet? But still, actually, no, I’ve seen some patents. It’s early and may never happen. But I’ve seen some patents where you’re a future version of AirPods. Your buds one day may be sensing your blood, your temperature, your other stuff / going into your health.

Arseniy
Maybe it’s coming. I think I saw something. Maybe

AB 40:45
But you’re absolutely right, this does lend itself, lead itself to, and these conversations with those sorts of San Francisco, Silicon Valley based companies who have privacy basically everywhere. And to have that distributed intelligence, it’s, it’s still pretty radical that there are more AI processing chips on our phones than there are, you know, I think on the ratio of CPUs, GPUs, and TPUs or AI processing units.

I think most things are AI, like that is where the lion’s share of the processing power comes through. When you take a photo, you’re not actually taking a photo, you’re keeping a shut open for a very long time.

And if you can say what you’re taking a photo of and adjust things so that, you know, the sky is blue, people are vaguely have rosy cheeks and they don’t look like they’re bright green. You know, there’s a lot of work going on when you think you’re taking a photo, there’s no single film is exposed.

That level of local processing first, where that is not possible, go up to a cloud, private cloud, I’ll anonymize the request though, so it has less of a, you know, can be farmed less. This does end like to me that there is a certain company you need to be talking to at some stage, keen to deny that they’re on your hit list.

Arseniy 42:04
Yeah, absolutely. There will be a lot of integrations, first of all, to also be able to provide those options to the user and smoothly do them. Right. So there will be a lot of technologies that we will need to partner with or integrate with also to make it possible.

So it’s not just service providers, but also to the providers of the technology who who do that, for example, the security bit, right? Yeah, we’ll, we’ll, we’ll work with the best in class to provide the security and the optimization is there.

And then in a small fashion with other parts of the system as well. But I’m just seeing that the distribution is super important. And Apple is doing quite, you know, they’re, they’re the biggest terms of the, or one of the biggest with the devices with the user base.

So if we can get there to the devices, of course, this will be a win. But, you know, everyone wants to get that user. Yeah. Access to the user base.

AB 43:00
…the hardware, the software, and they’ve got a real big focus on the experience and making it seem worse. So it definitely leads in that direction quite well. But irrespective of that, being able to tap into any phone, I dare say a phone or a flat screen is going to be the interface for a while, looking through a looking through a phone lens in AR style, you know, panning it around the kitchen and having on that screen overlaid what you wish to inform the user they can do with things.

That’s the interim or long term technology until we all get again smart glasses and

Arseniy
I’ll say even brain implants, but this would further away, but not that far. It’s just a super direct way of how to interact with things. You just don’t need an intermediary thing almost. You don’t need it.

But it’s a bit further. It’s a bit of a different topic. But I would say even AR glasses, VR glasses, there is a lot of things that can be much more predictive towards the user, much more proactive towards the user if the system knows them.

And also, all such devices as the Rabbit R1, I imagine if it knew already your context, it’s already, it could at this point already connect to at least some services, which it couldn’t, unfortunately.

But I am hoping for a very cool over -the -air software update, several months that will make it magic. Right now, it’s a bit not that useful. But imagine if it really held information about your profile or could connect with it, that would be a whole different story.

You would have to describe all the context that you wanted it to do. You would say, order me food. It already knows what you want. Because surely you want it.

AB
I’m thinking your context doesn’t switch every alternate 10 seconds? Probably find cases where it does, but in general, for the next hour, I’m hungry, I’m in food mode. After that, I’ve got to do X or Y.

You don’t need to pull something probably every millisecond. What are you doing? What are you doing? What are you doing? You could probably, I last checked context five minutes ago. I don’t think that’s changed.

We’re still exercise mode or commuting mode or focused on work mode.

Arseniy 45:17
And then you have all the health data that’s coming in. It’s, it’s like all the different types of data, more and more, they’re coming into the system, they’re being processed. And then you create an almost, it’s like basically this, you know, digital twin term, basically will be a digital twin few that will be kind of adapting the world to you.

That’s the old computer. Yeah. And you have the objects with all the information and actions that you have, uh, that they have, uh, for example, the table may have some information again about groceries in your, uh, in your, in your fridge, uh, or maybe it’s a work desk when it has your email or something like that notification.

Basically it already adapts. It shows what you need to see as a person. Another person comes in, they see different groceries because they’re vegan. Uh, they come to the work desk. They, they don’t see nothing because do they have privacy access or like they’re very secure.

So in that sense, that’s the big division. I’m, I’m thinking the world is going, it should be going and it should be going at a faster pace. And that’s, that’s what I hope to contribute.

AB 46:16
What’s your prediction? Can I get you to do the Bill Gates “the world only needs five computers” kind of stupid prediction that we will hang you on a few years later? What sort of timeline do you think something like this is in its infancy and what time do you think this might really hit its stride? One year, three years, five years, or this is a 10 -year kind of journey?

Arseniy
I’m thinking it’s in the range of five to 10 years. Because right now, we are basically with all the base systems and solutions being built, it is setting the grounds for being able to get the needed data, process it, understand the context better, and then do something with it.

For example, I’m a big follower of the Archetype AI project. So the same guys from the Google ATAP, they went away. They built their own stuff. And it is super profound in the way of how you bring the context from the physical world into the digital one.

https://www.archetypeai.io

Arseniy
So you do have, you process what the sensors see. Basically, time series data, they process it, and understand what happens either from the cameras, from the sensors. And then their system makes sense out of what happens in the real world.

And then in the digital realm, you can do some calculations or some predictions, et cetera. So that really sets the base of where do we get the data? How do we make sense of the world? But then next step is what do we do with it?

And if they are already doing some stuff, so it’s really moving fast. So I would say five, 10 years, maybe it’s already too long. So five years or so.

AB 47:54
Yeah, so between five and 10, but if you were able to be positive, close to five, if there’s a lot of unknowns in there, then okay, close to 10, that’s wild. A problem I would like you to solve for me, if you are talking to any one infinite way company, Apple Watch, and here’s a real life problem that I hope your future adaptive interface will solve.

If you’re going for a walk, it does, if you don’t set things, it says, ah, it seems like you’re going for a walk. Would you like to record? Yes, yes, so, and it does that after many hundreds of meters because it figures out pretty confidently that you’ve been going for a walk for a while.

We live in Australia where we live on the quarter acre block and every weekend in spring and summer, you’ve got to mow the lawn, don’t you? So the great Australian dream is knowing that backyard and doing laps up and down, knowing the silly thing.

I’m getting up sweat, I’m breathing heavily, my watch should know that I’m exercising, but I haven’t left my wifi and I haven’t actually left a radius. So it never, ever, ever suggests, hey, it seems like you’re doing some medium level exercise, would you like to record this?

Of course, that is, for me, that’s a cracker of a use case for context. If it could know that every alternate, it’s on a weekend and it’s in winter, you don’t have to, and in some of you don’t have to, in Australia, the grass is dead, but in spring and autumn, fall, goodness me, the sound of lawnmowers on the weekend is what a shrimp on the barbie and lawnmowers is pretty much the classic Australian weekend.

That level of context awareness, it shouldn’t be hard to figure out that it’s on a day, that’s a Saturday or a Sunday and I’ve left X or Y, but my heart rate’s up and I’m doing something for an hour and, and, and, and obviously that level of context is easy.

If I can speak that context, you can probably put a rule somewhere and someone can find a way to action that with real code. That’s not AI, that’s fine. Real code works beautifully many times, but is that the kind of level of nuance you would love to be able to capture on a personalized level of different people in different scenarios and knowing routines, actions, spotting the patterns in daily life.

Arseniy 50:18
Yeah, absolutely. Yeah, I would say in our case, it will be more about if you see, if you’re going for a walk, it reminds you to get something it knows that you’re going for a walk, for example, it reminds you to grab some water or something.

So basically provide you additional things from the real world that you need to remember all for or to do. But in that sense, that it already knows your what you’re doing, what’s happening, goes, there are already systems like that, for example, the same architect, architect API, it, for example, you have a camera or some sensors that you’re working around, it understands what’s happening, you’re walking, probably sweating, because it’s hot outside, and you’re doing a lot of walking. So in this sense, it’s more about when you have certain, in our case, it’s more about when you have certain thing you want to do, it already predicts what you need in the world of all the things you need to take into account or do for achieving that in the optimal way.

It’s more about when you do want to have something. But also, after that, yeah, when it realizes that something is happening, it will most probably remind you that, hey, your heart rate is high, you’re sweating, grab some water, someone gotcha, gotcha.

AB
So less task focused, like I am going exercising or I am going to make a meal, but more delighting in the moment of, you know, it’s been, you’re in the kitchen, you know, let’s get stuff out and let’s get going or putting together parts of your daily routine that even you had forgotten about, but it’s able to suggest on the fly, build a way to show that to you and then back away.

Arseniy 52:09
Yeah, that’s exactly where interfaces are going. Basically, right now, in our case, also specifically the same thing, but basically, it’s in general. So first of all, it’s task focus, right now, it’s prompt focus, you have to write everything in the prompt.

All right, it will simplify how we get to the task results in the optimal manner. This is our current focus. But then all the other interfaces and for us also, the logical evolution will be that it’s already already knowing how you do tasks, what you usually when do you usually don’t do those tasks and how you do them, it really practically suggest, hey, yeah, I’m feeling you’re hungry.

Actually, I just finished the book I sent to you this interface design, it’s just came out, I just straightaway ordered it by Guillem Couche. So it’s a French name, but I’m, I might be wrong.

https://www.amazon.com/Interface-design-Creating-interactions-successful

Yeah, I might be wrong in pronouncing it. But basically, the final chapter is about what’s the future. And that’s exactly the productivity of the interface as being able to already know you’re looking at a banana specific example from the book, you’re basically looking at a banana.

All right, it already orders you the food that you like. So you don’t have to think about it, it already comes to you. And then it comes in, you know, when the human really blends with the machine. So of course, different sides to it.

Where do we know that it’s our decision, right? Yeah, that we processed rather than already the AI or the algorithm basically some kind of really subconscious thoughts, without us being managing to make that decision before it’s already delivered.

So but it’s an interesting thought experiment to think how far will we go? Also, no, you already wrote about the ultimate, ultimate vision of we won’t be communicating with our system, we’ll be communicating for all the decisions.

And then for example, even the romantic partners will be fine with that. So that’s a, we can go super crazy far in the future. But yeah, towards approximation.

AB 54:05
Yeah, you’re going to be doing the dark pedaling furiously underwater, but looking serene on top. There’s going to be a hell of a lot of furious pedaling and connections to be made at the back end. But the goal is to be as interface -free, context -switching, context -aware.

And as you said, reading your mind without having to read your mind, that’s an admirable goal.

Arseniy
And you know why? To live live, not technology. Because we’re too stuck in screens that we have right now, and devices overall. To do something. Why don’t we just want to do something? We spend a little time thinking about it and doing it, but really enjoying the results of that.

AB
So make it humane. Everyone has something which even though it hurts, it kind of feels good doing it the long slow way cause they’re in control, but gee, being able to keep up more and more stuff to be able to just, you know, and wave it and have it done is, you know, is glorious.

And we can do that these days because of, you know, CPU power to energy is climbed beyond belief. I was talking about systems today where, you know, if you have a data set not long ago, you would throw a few models at it and you would get good results.

Now you can cut it before you go to lunch, tell your meta AI, not your meta AI, but your mega AI to process it all the ways, like literally process it in all the known models from this list. And even though most of those are going to be stupid with what your data set is, with what the model is, there’s no harm.

One day you might be pleasantly surprised there was one odd idea that just, you know, was a delight that you hadn’t thought about because it was outside your knowledge base. And sometimes it does just, you know, to justify that what you thought was great, but that there’s far less penalty now for doing things aspirationally beforehand, having agents running in the background.

Yeah, our cents per watt per compute power that it does per cycle, it is just, you know, gone astronomically so much that the penalty is reduced to, you’re now in control of the next level of interface that I am looking forward to.

Lastly, can we find out the name of your startup? Can we reveal or links in show notes? I guess the question zero in this section is, how do we get in contact with you if not LinkedIn? What’s the way to reach out?

Arseniy
Yeah, I think I’ll leave my email for you. So sort of we, you know, just keep it as a as a as a contact email. Yeah, I think that’s the best way. Just write to me. I might might leave also the number, but I don’t think it’s necessary, especially given that I’m right now in Europe. So a bit too expensive to call.

AB
All good. Makes perfect sense. Can we do a follow-up? Are you doing an excursion to Silicon Valley, a extended stay, couch-surfing like the best of everyone to see how long you can last to funding and find partners?

Or when can we check back in with you to see what the next steps for both your co -founder has gone, but also how your meetings and conversations have gone? When’s a good time to check back in?

Arseniy 57:33
I think in a month or so, maybe towards September, because I’m thinking to go in a couple of weeks time, there is a Techstars hackathon on the physical and personal AI on this personal AI operating system.

So I’m hoping to meet some really exciting people there, connect with people, and also to meet with some of the guys already creating stuff on the cutting -edge market API from BrainAI, perhaps as well.

So whoever is in Silcon Valley will try to get a hold of it. But I’ll be there for a couple of weeks, I think.

AB 58:09
Legendary. Thank you so much. Absolute joy to chat to you now formally with the record button going. We’ve had a lot of this conversation, but before we even got into the detail here, many moons ago, but great to have you back on, to be able to talk to you at length.

From all your links, I will put most of those links in the show notes, I can possibly find. There might be a brain dump there, so apologies if they might be in alphabetical order, but that’s about it.

There might need to be some context switching there, too, to make that happen. Your details will be there 100 times, so it should be easy to catch hold of you. And by all means, we really are looking forward to seeing what this future brings.

The spatial computing world is probably one of the biggest doors that needs to be opened to solve finale problems. There’s been a lot of incremental and evolutionary growth in these fields, but it definitely does take a revolutionary way to do it.

Definitely, I’m wrapped here that you’re in the software space, you’re trying to do the human problems first, and figure out where’s your niche to be able to play with. I’m glad you’re not trying to humane pin rabbit.

Notwithstanding, the world has shifted that way, maybe it shifted back to, yeah, but that rectangle in our pocket probably is the best bank for our buck, and let’s use that for a bit longer. But I’m just super thrilled to watch this journey and encourage you to say, hurry up, we want what you’re thinking about.

So yeah, please make one, that’d be great.

Arseniy
We will, we will. And that’s why it’s my pleasure being here. Thanks for having me, because I’m realizing that I’m actually one of the several brightest minds that have visited this this this podcast, I would say.

So it’s great to be here. Thank you. It’s been a great conversation. So, yeah, it’s looking forward to getting in touch with people who could help us bring this faster, as you said. Thank you.

AB
We will get to work as far as we possibly can. This will be worth it. We’re probably either listening to this or watching this on YouTube. You’ve scrubbed right to the end to go straight to the who done it.

From us though, we’ll leave Episode 23 there and we’ll catch you next week on SPATIAL. Arseniy, you thank you so much. And to all, we’ll catch you next time. Cheers all.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

To absent friends.

The post Episode 023 – Arseniy Sergeyev & Adaptive User Interfaces appeared first on SPAITIAL.

AB and Dr. Mehdi Ravanbakhsh – an expert in the field of geospatial AI (GeoAI) – discuss a wide range of topics related to Mehdi’s extensive experience and contributions in the geospatial industry, including photogrammetry, remote sensing, and AI applications in various domains such as agriculture, forestry, fisheries, and insurance.

Mehdi shares his background, starting with his PhD research in Germany on road crossing detection from high-resolution aerial imagery, which combined photogrammetry and computer vision techniques. The conversation delves into the advancements in photogrammetry, from manual tie-point identification and feature extraction to the current state of automation using AI and machine learning. Mehdi highlights the challenges faced in the past, such as the time-consuming nature of manual processes and the limitations in creating large-scale mapping due to resource constraints.

The discussion also covers the applications of GeoAI in various industries, including agriculture, where Mehdi’s company, Mapizy, has developed solutions for pest control, crop monitoring, and farm insurance. In the fisheries domain, Mehdi shares his experience with a project funded by the Australian Marine Research Institute (AIMS), which involved automating the process of fish counting, measurement, and species identification from underwater videos.

Furthermore, Mehdi discusses his recent visits to countries like Vietnam and Indonesia, where he explored opportunities for national mapping organizations to benefit from Mapizy’s technology. He highlights the challenges faced by these organizations in creating foundational databases, such as national orthophotos and elevation data, and how Mapizy’s solutions can help reduce the requirement for ground control points and improve data accuracy.

The conversation also touches on the future of the geospatial industry, including the increasing availability of high-resolution satellite imagery, the integration of radar and LiDAR data, and the potential of low-earth orbit satellites for data acquisition. Mehdi can’t contain his excitement about the bright future of the geospatial industry and the continuous advancements in sensor technology and data processing capabilities.

Dr Medhi on LinkedIn: https://www.linkedin.com/in/mehdi-ravanbakhsh-phd-94674869

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/yjSRSUDOmpY

The discussion delves into the advancements in photogrammetry, from manual tie-point identification and feature extraction to the current state of automation using AI and machine learning. Mehdi highlights the challenges faced in the past, such as the time-consuming nature of manual processes and the limitations in creating large-scale mapping due to resource constraints. He also discusses his work at the CRC for Geospatial Information in Australia, collaborating with renowned experts like Professor Clive Fraser and Professor Christian Heipke, who played a significant role in automating the photogrammetry process.

The conversation covers the applications of GeoAI in various industries, including agriculture, forestry, fisheries, and insurance. Mehdi shares his company Mapizy’s solutions for pest control, crop monitoring, and farm insurance in the agriculture sector. He also discusses a project funded by the Australian Marine Research Institute (AIMS), which involved automating the process of fish counting, measurement, and species identification from underwater videos.

Mehdi discusses his recent visits to countries like Vietnam and Indonesia, where he explored opportunities for national mapping organizations to benefit from Mapizy’s technology. He highlights the challenges faced by these organizations in creating foundational databases, such as national orthophotos and elevation data, and how Mapizy’s solutions can help reduce the requirement for ground control points and improve data accuracy.

The conversation touches on the future of the geospatial industry, including the increasing availability of high-resolution satellite imagery, the integration of radar and LiDAR data, and the potential of low-earth orbit satellites for data acquisition. Mehdi expresses his excitement about the bright future of the geospatial industry and the continuous advancements in sensor technology and data processing capabilities.

AB
Well, g’day and welcome to SPAITIAL. This is Episode 22. I have another super special guest with me. I think it’s fair to say my guest here is ‘the OG’, the old guard, the original, pretty much the master.

Friends, love to introduce you to Dr. Mehdi Ravanbakhsh.

Mehdi
Yes, you got it right, yeah, absolutely.

AB
That was the one thing I was sweating on. Nice one. Excellent. I have already passed the test. I’ve hit my KPI for the entire interview. Mehdi, thank you for joining us. Absolute pleasure to talk to you.

Mehdi
Yeah, my pleasure. Thanks for having me, AB and it’s nice to be in your show and discuss ideas with your audience.

AB
Look, I have a full board of ideas. I have full browser tab of things to talk about today. There is a lot that you have been involved with: touch points, professional institutions, companies, startups, universities – the list is massive.

We will go wide, we will go deep where we can. We’re talking remote sensing, we’re talking GeoAI, which is a phrase we’ve used here on the SPAITIAL podcast before. And it is — to help define it from, I guess, a spatial AI or GeoAI.

I’m kind of talking Spatial AI as both spatial computing and geospatial mixed with AI, but GeoAI by definition is the mixture of geospatial with AI. Can you tell us about, goodness me, your wide and storied past, but how this field is the one that has, you know, your name stamped all over it?

Mehdi
Yeah, just a basic introduction to myself. Actually, I have 20 years of experience in geospatial. So very long time I’ve been in this industry, I did my PhD in Germany. And at that time, photogrammetry just started getting into the digital space.

And my thesis was in the area of people call it photogrammetry computer vision, because computer vision started taking off at that time. So there was a lot of research in this space in Germany for image analysis for, you know, building road detection, this sort of thing.

And photogrammetry was a different discipline. But that was the start of it that using computer vision and see how you can do in the geospatial space. So essentially, my thesis was in road crossing detection from very high resolution aerial imagery.

At that time, still high resolution satellite imagery was not available. And so the aerial imagery was around three centimeter resolution was really big thing at that time.

AB
Would that qualify ias anything we would call high resolution today or is that just latent background?

Mehdi
Aerial imagery is a completely different animal. You can see that still the companies that in this space are active, they provide very high resolution imagery. This is something that matched the quality of the drone imagery, especially in the urban environment.

It has an upper frame. But yeah, that’s actually the start of my journey into GeoAI- using image analysis, computer vision techniques and the range of the geospatial processing to be able to help geospatial communities and businesses.

Then I was offered a post-doc position at the Frontier SI. At that time was CRC for Geospatial Information, working with the great guys in this space like Professor Clive Fraser and Professor Christian Heipke.

https://www.ipi.uni-hannover.de/en/institute/news/news-details/news/honorary-doctorate-for-prof-dr-christian-heipke

Mehdi
These guys were instrumental in our discipline helping, for example, Christian Heipke helped the digital photogrammetry to the whole process of photogrammetry processing become fully automatic, essentially triangulation, ground control point connection, interior orientations, photogrammetry.

These are a bit technical, but the whole procedure was really long.

AB
Absolutely ground breaking. By all means. Yeah. I mean, the idea of doing that is to actually bake, you know, the number of CPU cycles is huge. We’re talking 20 years ago, 15 years ago, were you working on small patches in high detail or very wide swathes of land in moderate detail?

What was the sort of the limitations and the wild dreams versus what the reality was in the early days of large scale photogrammetry?

Mehdi 04:34
Absolutely. I think the limitation of the photogrammetry in the old days was because that was a manual process. So even finding the tie point, finding ground control point on the image, that was a manual process and triangulation, particularly that triangulation is the most important part, especially making sure that the final output you get from images and reflect the reality, or we can call it positional accuracy of the output.

So to ensure that this positional accuracy is at the best quality and accuracy, you need ground control point and toy point. Toy point essentially sticks images together and ground control point sticks the whole model to the ground.

So that’s the whole, I think one of the time-consuming part. And another important part in the photogrammetry procedure after making sure that the position of accuracy and orbit observation are correct is manually capturing topographic object or man -made or natural object.

And this is a very time -consuming process, especially just an operator sitting in front of the screen, digitizing building, roads, trees, anything on the planet. So that was the main challenge for many organizations that the most time-consuming part.

So as a result, most of national mapping organizations, they were struggling to create, let’s say, larger scale mapping for the entire country. AI actually changed this. Still, there is a bit of human in the loop, but this process has been automated to a large extent.

AB
It’s such a joy to be able to take lots of photos – almost cheating now if you take high res video – but to walk around with a phone just around you. I was in a museum (different part of the world) grabbed the phone out, there were some lovely sculptures. I just did a LiDAR on the fly scan of them and now I’ve got a lovely 3D mesh of them.

Mehdi
Yeah, it’s fully automated. It’s fantastic. I know.

AB
How easy is that now compared to, look, I’m still reeling at the fact you said manual tie points. That would have limited probably your scope, but also meant that you were the authority on probably strong coffee clear the whole day and just sit there and just match up images so that that rock is the same rock as I’m seeing in photo A to photo B.

Mehdi
Oh, I think it’s a significant difference. It’s now super easy. For the grammetry, let’s say after automation of this photogrammetry procedure, as I said in 1992, that was started. In 2008 and 2010, that was almost fully automated.

They are not feature extraction. Essentially, feature extraction means collecting topographic or other features like road and buildings. Not this one, but this triangulation and making sure that the final output is right for the purpose.

This part has been fully automated in 2010. People said that this is the end of the grammetry. If you have photogrammetry, you need to update your resume and then look for jobs somewhere else.

AB
I was about to say that it is NOT the death of photogrammetry. It is simply a democratizing, kill me for saying the word, but more people can now do it rather than the hands of specialists. There’s always a role for that.

But the scale, I’m gonna point out there’s a company based here in Australia, Aerometrex. They have just done most of the Great Ocean Road coastline, which – my wife’s a geologist, she was going, we were gonna ride our motorbikes down there for three months straight, drone batteries, a lot of coffee and coffee stops to recharge batteries.

And in the two or three years, when we were deciding what kind of batteries, what kind of motorbikes we would buy, there are firms now doing that on the largest of statewide, not thousands of kilometers, but hundreds of kilometers, digital twins.

Is that the natural pinnacle? What can you see as the next step from digital twin to GeoAI?

Mehdi 09:00
Yes, actually, I was mentioning that in 2010, then their own K, and suddenly photogrammetry has become a very sexy word. So, everyone’s talking about photogrammetry, drones, and the automation of the procedure.

So, again, even for example, I’m an adjunct professor at UW School of Computer Science. Even people from the field of computer, they say that this technology is amazing. They call this photogrammetry.

Even in the computer science department, they were talking about photogrammetry, how amazing this technology is. And that was the start of drone imagery, fully automated procedure. It’s super easy for everyone to capture imagery.

At that time, you see that a large number of drone companies came to the market and offering mapping and different types of applications, which was great, I think, for the whole photogrammetry community and mapping community.

Because even in some, I guess, respect, it replaced some ground surveying as well, which was a bit risky, time consuming, and costly. That was great. Now, you mentioned that there are a number of companies active in this space, mapping coral reefs, digital towing.

Yes, this is a new trend, especially in combination of photogrammetry, laser scanning. This is ranging technology that can be combined with photogrammetry. But the type of technology that develops over time, particularly in technology, I think one of the fundamental techniques is image matching.

This image matching ensures that you see the details in the digital towing. And there have been some great development and breakthrough in this technology, because previously, it was not possible to create that level of detail.

AB 10:55
I know! My credit card got hit hard when I bought my drone, but it’s still valid. We were able to get down to one centimetre resolution – granted my wife and I were playing with rocks.

Rocks have a great habit of staying still and being high contrast, so you can find them again. So it’s perfect fodder for photogrammetry of cliffs. But now we’re talking sub centimetre and we’re talking not only the quality imagery, but the precision and the ability to determine when it’s a natural object.

And perhaps with digital twins, the main peril was when you had something that was humans would know is flat, would always be wobbly wobbly and it would be hard to correlate. It would make everything into a mesh that was organic, as opposed to the grounds organic.

That building though, I want crisp sides. Is that part of that classification? And if this, then, you know, if it is something that I think is previously labelled, then I shall change my settings. Is that another meta automation tool on top of the photogrammetry landscape?

Mehdi 12:04
Yes, actually, I think this has been a challenge for photogrammetry for a long time when there is no feature like in the coastline and on the rock or on just the wall. For example, for indoor positioning, that always has been a challenge.

There have been a number of solutions in this space. Still, in such scenarios, people recommend to use photogrammetry in combination with ranging technologies. That gives some insight. But compared to the past, these days, technology is now doing much better.

So you have less challenging time working in the area with less features or less texture. So this is, I think, still, there is a challenge because we know that when you use imagery, you need texture.

This is something that AI algorithm are hungry for. They are looking for texture and pattern. So they don’t know which area to match to which area. So this is a bit of challenge. I think for such a project, I guess one of the recommendations is using ranging technologies or combining this with photogrammetry because you get the elevation data or depth map from ranging technologies.

You can combine with intensity from this.

AB
I’m about to say that the drone we bought hit the credit card medium was only sized, but 10, 12 years ago, an infrared camera would have required a second mortgage! I’m grateful to see that infrared spectrum along with RGB visible light spectrum is coming down into, I still think that one still hits the credit card a little bit, but it’s coming into prosumer into potential range to actually give us those other spectrums.

I’m going to jump forward to when you were involved with and use are still deeply involved with farming agriculture, smart farming. Have you seen that the potential for infrared with visible light allows for that full spectrum, more of that image analysis to happen beyond the human perception?

Mehdi 14:26
Yes, absolutely. I guess in agriculture, we know that this is really important to be able to collect it near and far for the health of the, for example, crops or trees or any vegetation on the ground in agriculture.

And I think the challenge in agriculture is especially last week I was in Vietnam and in Indonesia, we had a lot of discussion about challenges in this space. And previously, I had some projects in New South Wales in Goodowindi, essentially, for creating an accurate map of wheat, pest or disease.

I think agriculture is drawn out fantastic, but the challenge is the connectivity in remote areas. And we always, I guess, we recommend that there should be a solution around this. One of the projects we previously was involved was pest control through AI, essentially, rather than using a drone imagery, which requires lots of logistics, and you need to capture the data at – let’s say, a sub-centimeter / five millimeters – and you need to operate it in the cloud. A computer vision company needs to access this. And the whole process takes a couple of days or a week. And by the time that this map is ready, the map has already changed on the ground.

AB
Gotcha. Yeah

Mehdi
So one of the solutions that we recommend, especially in this space, rather than capturing the imagery of the entire farm, which is time consuming, you can just use everyday smartphone from farmers, put it on the wheel of the tractors, capturing every 10 seconds, single image.

And then that gives a hit map of the, for example, wheat or pest or any issues on the farm. Of course, that’s not highly accurate, but that’s, I guess, feasible solution for the farmers. That does not cause any data capture, because we know that data capture and logistic is an issue.

AB
…but to have 4K video processed every day requires hard drives, internet, battery density to recharge versus petrol, internal combustion engine, tractor, phone or GoPro strapped to it last years or a secondhand phone like that and taking a static picture, 10, 12 megapixels every X meters, you know, you could email those around much, much easier.

So yeah, the scale of well, data, IOT, photos, but then videos 4K, that makes perfect sense. Absolutely. Yeah. Not quite low fidelity, but right fidelity for the, for the problem at hand. Yes. All right.

Mehdi 17:20
And it’s quite almost online because you can collect the data, you can process in a couple of hours because the images are not too big, and you see that because the resolution is so high, it’s so close to the object, you see the detail.

You can process it, you can create the event, you cannot cover the entire farm, but from what you can extract, it’s really accurate. So you can extrapolate to the entire farm and you can put it in the tractors and you’re ready to go.

I think this is one of the great solutions, especially in Australia, we’ve got connectivity issues and I noticed the same issues in Vietnam. One of the things that the company came up with an idea that using a terrestrial station and using edge computing, essentially estimating a number of parameters on the farm like moisture and this sort of thing, but pest detection and insect detection was one of the solutions without uploading imagery,

they do it actually through edge computing, which is fantastic because although that’s not about the precision agriculture or smart farming, because for this one, you need to capture the entire farm and find the exact location of issues.

But still, I think for farmers to have some idea of what’s happening on the farm, this is great at significantly lower cost.

AB
No matter where you travel, even if it’s only on, you know, going to the top paddock, the side paddock, you’re right, you don’t get the entire acreage, but you’re getting data points that you wouldn’t get unless you, yeah, did it, you know, for 10 -fold, 100 -fold more robust way.

Any data is better than no data, nothing worse than bad data! but to get samples that give you that confidence in ads to story, you’re almost doing Google Street View for a farmer’s own premises.

Mehdi 19:25
Google should be also good. I guess any existing data sources that you can get access to, I think that’s great. That certainly add values to what you develop. And this is, I think, has been used in not only in agriculture, we also used it for the insurance as well.

So getting some estimation of the situation of the building. So for example, finding the number of floors or the situation in around properties, this is also very useful. So this is a great resource.

The only challenge with Google Studio is that it’s a bit outdated. But as you mentioned, when you’ve got some data, you can use it and you can enrich the dataset.

https://mapizy.com

AB
Now, can I ask a loaded question as CEO and founder of Mapizy? Is this the kind of tool and techniques that you’re turning into products? out another way: what’s the focus of Mapizy?

Mehdi
Our mission is to make the mapping an easy task, because mapping is still a complex task. The mapping organization, they have a large department and group just in charge of making sure that the remote sensing observation correlates to the ground data, and the quality check, and all this sort of thing.

We know that it is still a challenge, but things have been a lot easier these days. Essentially, we in Mapizy, we started with forestry, counting trees through collaboration with a local drone company, and then we gradually explored agriculture and fisheries and other areas.

But after a couple of pivots, we found a bigger opportunity in insurance. There was a hackathon organized by Deloitte a couple of years ago, and Mapizy was one of the participants. One of the challenges in that hackathon was how remote sensing data can be used to assess properties for insurance industries through IAG and RAA.

These are two major companies. RAA is based outside of Australia, and IAG is a global insurance company. Mapizy was awarded by both companies to solve two challenges at the same time. We found opportunities.

We had a conversation with these companies. We said that if we develop such things, would you buy from us? They said, of course, this solution is not available if you develop something like this. Of course, a lot of things happened.

They didn’t buy exactly from us, but we had collaboration with them, and they helped us to develop this. Essentially, to be able to use various data sources, that’s Google Studio or remote sensing from satellite radar and aerial imagery.

Using this data and providing insurance companies with accurate condition of buildings, the risks from the surrounding environment, and the way that this has been designed is suitable for insurance because it’s a fully automated process.

Mehdi 22:45
Let’s say if you’ve got millions of property addresses in Excel file, you can upload into our platform. Automatically, this will be processed. Excel file will be populated with a large number of attributes, at least 30 plus attribute changes to buildings and properties.

Then they will be notified when this data is ready. But in the traditional sense, normally, some big insurance companies, like the GIS department and large team, they are in charge of doing this manually, which is a time -consuming process.

We did this in a way that is fully automatic. It uses kinds of machine learning on -demand processing. If you’ve got clients, we process it so the clients actually pay for the processing time. If not, everything actually will be in the cloud with no cost for us.

It’s a kind of sustainable business in this space because, as you know, GIS special space is one of those spaces that the GIS special data means different things to different businesses. Making a product that everyone uses GIS special data is really hard.

AB
It is, isn’t it?!

Mehdi 24:01
So, you have to be very specific and in this case, we just targeted the insurance market because although the agriculture is big, the forestry is big, the fish is still, I guess, compared to other people performing.

AB
High priority, yes, no, it makes perfect sense.

So what were the sort of things that you had? What were the sort of things you could add to that data set per property address? What were some of the low, mid and high level sort of inferred attributes you could add?

Mehdi 24:39
Actually, the information that we can extract from existing data in different categories. For example, the building itself. For example, what’s the quality of the rooftop? We can estimate the quality of rooftop at the tile level.

If there is a broken tile, you can’t find that. So that’s what the shape of the roof and the shape of the roof is really important when there is, for example, a strong wing or a tornado or things like this, the shapes matter.

And that’s one thing. And also the information about the land parcel, whether that’s, for example, a swimming pool or, for example, information about shoulder panel, whether there is a tree, a risk of tree to the building, we call it tree overhang, whether there is a shed in there.

Mehdi 25:36
So any types of things that could be liability and can create issues for the building and risk, we identify them, we quantify them. So that’s inside the land parcel. Should be started from building land parcel and out of land parcel is this size to water bodies distance to the bushland.

And the size of the bushland based on insurance definition should be over one hectare. And these are a bit of detail, but for all of this, we develop an algorithm that, for example, if there is a bushland, we identify the soil should be over than this.

We estimate the size of the building to all these objects that pose a risk to properties. And so this information, just one step. Another step is change detection of these attributes over time.

AB
I was about to ask, doing it once is spectacular, but it’s becoming back in a regular cadence and far more regularly, there you go, people wouldn’t be driving past or doing a light plane overhead inspection, so having it in the system and just go back to what asked the same question over time and if the value changes, red flag or start to warn, gotcha.

And then what sort of time series data can that give you per property? And also does that upscale to trends per suburb per region?

Mehdi 27:03
Actually, this is a great question because we know that in change detection, this frequency matters. For different applications, especially in the case of natural disaster or flooding, the frequency should be on a day or week basis because we see a lot of changes.

Insurance companies need to be able to very rapidly assess the damage and making sure that they’ve got enough funding in the bank to assess claims. This is critical in this difficult time. Having said that, when the business is normal, as usual, so there is no natural disaster, normally, this yearly update is necessary.

Every year, they update the policy. This is really critical for user experience because still there is lack of confidence and trust in insurance industries, not in Australia, in some other countries.

Mehdi 28:00
People, they say that the insurance companies, they update the policy, increase the fee, without no apparent reason. If you use this change detection and change analytics and say that, look, for this reason, for example, you added a shed or pole, or for example, you add a solar panel, you don’t have protection, you need to update your policy.

I think this is a great user experience that overall good for the industry. It’s essentially a data -driven decision and the update of policies based on the data. That’s change detection. Also, for claim assessment, this is also another important aspect that during natural disaster or for whatever reason, people lost claim.

But you know that although most of the people provide accurate information for insurance industry to be able to validate claims, they need the independent data set. We can provide them the current condition based on accessing the latest remote testing data.

You mentioned, for example, for regional Australia. Regional Australia, of course, is the challenge. I guess remote testing these days, very high resolution satellite imagery at around, I guess, better than 50 centimeter, these days are available.

You can get access to it a bit pricey because normally for regional, you have to order the satellite and pay a bit more. But the good news is that it’s available. If there is an emergency situation, you can purchase imagery, you can order in, let’s say, one or two days, you can get your imagery.

For major metropolitan areas and densely populated areas, we have no shortage of data. There are lots of data. Now, as you mentioned, there are some leading companies like Aerometrex, Nearmap, they are quite well known.

They capture data regularly. And so, this data can be used. Satellite companies regularly capture large cities all around the world, at least once per week. In the urban areas, always there is, I guess, abundance of data set.

AB 30:19
If we look on Google Maps, probably our houses haven’t … I mean, I still can look at the Google Map of my house here, and there’s a car in our driveway that we haven’t owned for seven, eight, 10 years.

You can sort of date … I think that was if there was “the black car”, then it was 2000 and large swathes of Google Maps, which is one of the predominant daily tools for us. Again, Ask Me Immortals, they don’t change too often.

It’s really heartening to know that there are companies out there that, if you do hear a light plane flying overhead, chances are it’s either, in my part of the world, down the surf coast, either a shark spotting, fire spotting, or it’s one of the commercial light planes doing those oblique, downward facing intermediate altitude, but high resolution, high frequency updates.

It really is mind boggling to know that there’s that much starter floating around, if only someone had a big credit card and knew how to get it, but that is phenomenal that there are companies who just specialize in doing, say, low altitude, high definition imagery at a high frequency.

AB 31:26
Have you also had, or how do you see the move to low-earth orbit satellites? Starlink, Mr. Musk, of course, doesn’t generally have cameras. They are internet comms, but planet.com is another constellation of low-earth orbit meant to be freely available, insert credit card here, satellite imagery, and visible bands.

Is that a new technique that’s coming through, a new data set that’s possible, or what are the advantages or disadvantages of low -earth orbit versus traditional plane -based imagery?

Mehdi 32:04
I guess these days there is a good source of high definition, high resolution satellite imagery. I say very, very high resolution because there is two categories, high resolution or very high resolution.

So, high resolution normally… I feel like those labels might have to move in.

AB
10, 20 years time to ultra, ultra high and no, no, no, really ultra high.

Mehdi
it’s yeah

AB
Absolutely, but sadly correct for today. Gotcha. What are those definitions? What do they have?

Mehdi
Yeah, high resolution from 1 to 5m, but anything below 1 meter was very high resolution and it started in 2010. In 2010, GEO, AI, IKONOS, QuickBear, these are satellite imagery that launched mainly from the digital globe.

Now they changed the name to Maxar and I was lucky that I was involved in calibration of some of these sensors with Professor Clark Fraser at CRC for geospatial information. And then Airbus launched playouts and other companies planted then came to existence and they have their own constellation.

Now we see that recently there is a good source of radar imagery, especially for flooding mapping for insurance, that’s very important. And especially for different types of, I say, complex applications that normally aerial imagery is a bit of a struggle to identify some of things.

You can use a combination with radar, especially over, for example, the marine environment and this application is also very good things. This is a new trend that you can see high quality radar imagery in combination with optical and so now, for example, at the current state, there are companies that they convert very high resolution satellite imagery.

They add more detail through AI. They add more better geometric definition to the detail, which we in my PC, we already had this technology. We were using this for a long time. These are new developments in this space that you can see that because, as you know, US companies, they are not allowed to go above certain threshold for security reasons. So they can’t really produce raw imagery at, let’s say, 20 centimeter resolution. They are not allowed. Although maybe in military applications, they use it, but for civilian applications, for commercial use, they are not allowed.

Mehdi 34:43
Having said that, I think this is still fantastic for many applications. For example, for insurance, we noticed that the same level of detail you can get from aerial imagery. You may not be able to get from satellite imagery, but still, the main features are there.

For example, trees, buildings, if there is a pool in there, you can see it. For solar panels, I think it’s still, I guess, algorithms are struggling to find solar panels on the rooftop.

AB 35:14
You’re looking for edges that may not be there very much. Yeah, absolutely. Can we talk about, we did before we pressed the record button here, we talked about super resolution. And I guess this is one of the, sorry to put a pun in or a dad joke, but this is one of the superpowers of remote sensing or GeoAI that are probably most categorised by me eye rolling whenever watch movies for the last 20, 30 years of, you know, ‘enhance, enhance’ and magically mission impossible ‘insert spy movie, sci-fi movie here’.

You know, you can’t read someone’s newspaper from over their shoulder from a satellite. I don’t think so anyway, but tricks of Hollywood notwithstanding, I’ve been flawed in recent months and in the last year of the advances in super resolution, things that you would not have thought possible.

It’s talk about a dark art. It’s also on the cusp of, I’ll say it carefully, keep it family friendly, making stuff up. There are techniques using AI to (I’ll ask the question in a second, apologies!) but to look at obviously with high resolution source, if you had it first, you could down-res it and then ask AI how you get from one to the other. So how you would up-res it in the future, but it still feels like it’s not real. It still feels like we’re seeing images that are, have a resolution that is, yeah, tree level, can kind of figure out the ages of buildings vaguely if you squint.

But then these techniques are looking at edge detection, minute subtle changes in well -calibrated digital imagery and back inferring or down inferring, shadows, doors, things on roofs, things that if you really squint it, a human might be able to say, oh yeah, there could be a car parked behind that house, but I wouldn’t want to bet my life on it.

And these tools can, to different levels of competence, detect another doubling, another four times, another zoom level in that is just unbelievable.

Can you tell us about, am I joking? Is super resolution a thing and how much have you been able to embrace that kind of technology?

Mehdi 37:25
Yes, I think that’s that’s real. And one of the I guess the development behind this is very accurate and recent image matching techniques that and technology is you know, to create, to capture a lot of imagery through a combination of lenses through camera system that just with the one single shot, you can create a depth model, right depth model.

And so essentially, the geometry because traditional photogrammetry is overlapping imagery. When you use drone, you capture imagery in a way that should be 60% overlap. And this one is completely different.

So essentially, the way that lens has been organized within the, the camera, you can capture a lot of images in different lens distances. And then through this, you can create a depth map, a single shot.

Mehdi
So perhaps from that…

AB
I see – so from the one viewpoint, but different zoom levels, that gives you enough of the perspective shift to reconcile what it was in 3D?

Mehdi 38:33
So that’s a new technology that created a lot of opportunities in various applications, not only mapping, but in the, for example, in food processing and monitoring. But having said that this technology by Tesla has been used, just single camera has been used to create a depth map.

We use the same technology in another project for fish measurement and the quantification of fish population, especially in the, this area is called closed range photogrammetry because you are close to the options.

You are not that far. And in the underwater environment, we use normally stereo cameras. We use just a single cameras. And when we capture the picture of fish, we use an AI techniques to create the depth map.

And we use that through, of course, it requires a lot of training data. We create training data through photogrammetry, through traditional photogrammetry, but train the AI models. This is exactly the way that Tesla is doing.

So the way that Tesla creating 3D model online is through the same technique. They use it to serial laser scanning data with streets. And then they use a single image to train the model to understand the depth.

AB 39:53
That’s phenomenal. I must say I was in a separate field two years ago with robotics and the trend was happening there too of say tractor wheeled vehicles. And at the start of that journey, those robotic platforms had multiple cameras.

And yes, they could do depth perception on the fly if needed. But the computational power to do that on the fly was pretty harsh. And if they lost one, ironically, having lots of eyes and lose one, that really hurts.

Versus if you have one eye or probably two, but both of them doing the same task in a monocular single fashion, then using AI to help you with that algorithm means that you’ve got redundancy still, but any one camera can just figure things out mostly by itself.

So that reduces the risk of overthinking and having spurious data throw your model into a spin. Less is more. And to quote Mr Musk calling the Tesla out, he famously in one of the interviews regards SpaceX said “the best part is no part”. And to have AI let us do that only two years later is just phenomenal. The depth perception, grayscale mapping from single images, even on dodgy cameras.

Now, it is just unreal to be able to see it’s been fantastic. So you have been involved in the calibration of underwater versions of that. How did you get involved with

Mehdi 41:28
Yeah, because as I said, I was lucky to work with another world leader in photogrammetry at RMIT University, Professor Mark Shortis. And he’s one of the leaders. And actually, we work three years on the research project for fish measurement calibration.

We use pool environment for calibration of the camera. So that’s, I think, one of the important steps in any remote sensing photogrammetry, you need to have a test fish. Of course. But then that’s great, quadrupole and pool environment.

So you need to calibrate things to ensure that the measurement you get is actually correct. Well, if I…

AB
If I can ask about the fish measurement, if you’re looking from a satellite down, you probably have things like a house or a car or a road. There’s probably times when you can grab out the virtual ruler and sort of maybe self calibrate or at least bring us off back.

But surely with fish, murky water, how do you know whether it’s a, was it a little fish close to the camera or a big fish far away? How do you get any sense of scale if there is not a lot else apart from murky, muddy water?

Mehdi
Yeah, I think the thing is that if you believe in the physics, I think that’s the physics principle. I believe in the physics level. Because we use underwater, we use two cameras, and it’s like the principle we use in photogrammetry.

So you see the same object from two different perspectives. Yep. So therefore it must be a depth x. Yes. And you can estimate the distance, so that’s one thing. But the most important in all the distance, we need the distance just to get fish length.

Gotcha. So essentially, length measurement is the most important thing. So if you know the depth, this depth will be used for measurement of distances.

You know, in reality we normally use DTM or depth to rectify the, I guess, positional discrepancy through caused by depth or height. So this is for creating, let’s say, orthophoto, you need to create this data first.

And for fish measurement, of course, we don’t have topography because this is underwater. But yeah, and the view is like an oblique view or a vertical view. Sorry, horizontal view rather than vertical.

But you need to be able to create some sort of depth map that gives you some scale for measurement. Normally, for the grammetry, I think this is one of the fidelity of the scale. You need to solve the scale issue.

So the scale issue is solved by the depth map.

AB
Gotcha, can I ask what were the applications of FishCam? What was the name of the project and what were the uses for it? Fish ladders or fish farms? What was the?

Mehdi
Yes, the application actually was supported by Australian Marine Research Institute, AIMS, and the grant was supported by the Australian government, 250 applications received, and four has been funded.

Mapizy was one of them, because we had track record and fish measurement accounting. And then the purpose of this was that AIMS, Australian Marine Institute, they collected videos all over Australian coastline.

They captured, let’s say, one hour videos in 100 -meter depth, close to the shore. And then the challenge they had, they didn’t have enough resources to manually go through the videos, count, and measure fish and identify species.

AB
This is a really time consuming process.

Mehdi 46:03
Yeah, but I’ve been, as you know, this has been a research topic in many universities, but the challenge is that most of this work was not suitable for industry and was not suitable for the Australian environment.

Mehdi 46:17
Because fish species, the quality of the water, and also the hardware, so everything was different. So we used the existing system that Ames was more comfortable with, and they used it in their operation.

We didn’t want to change the existing processing workflow for them, because they already used some of the tools and techniques. We just automate the procedure, especially to be able to make it more accurate.

And hopefully, in the future, we replace humans for this test. And one of the interesting things about the length measurement, which I noticed during this project, was that when fish are moving, there is a rotation in the body of the fish.

And then you need to, with AI, you need to identify whether this fish is vertical to the camera, or there is any movement in the body. When they are in both cameras, when they are exactly straight, then it’s a good time for measurement.

Otherwise, the measurement is not accurate. Now, to be able to find those important moments, you need to try.

AB
Boundaries of fish, yeah, indeed.

Yeah. And tracking for fish is different to cars, you know, because, you know, a large number of them come together. So they, they change direction suddenly.

AB
Such work in self-driving car and car safety technology, which is brilliant. Love it to death. But it is all from the point of view of four or five feet off the ground, forward-facing down a road. Love it, because it means we can try and solve that problem as, you know, to as nine -nines as possible.

But there’s not a lot of other data sets that are cars around. There aren’t a lot of long-running drones. And so to have sterescopic imagery from around the coastline and have it just supplied with, I guess, the metrics known of the camera distance, the focal length, the cameras themselves to get rid of distortion, that’s just brilliant to be able to harvest that.

AB
Can I ask what the accuracy classification was? Was it comparable to humans? Not as good, but more massive, therefore reduced error by volume and by more measurement points? Or was it starting to surpass a human’s capabilities?

Mehdi 48:45
So the accuracy was around 1cm, that was the phase one of the project, we achieved accuracy around 95. But the standard was a bit higher. So we developed a process that tools that for AIMs to be able to feedback the result to the training loops.

For example, if you notice that the fish has been identified wrongly, you can remove it for the entire video because we track every fish for the entire video. So this is the whole procedure because we know that in an AI project, this is really critical to be able to develop a feedback loop and improve the quality of the AI models.

Mehdi 49:35
We developed that for AIMS. And having said that, the most innovative part for us was the measurement. So this measurement was the most time -consuming part because you need to track fish and then you need to measure into different frames and you need to measure all of the fish.

So imagine that for each of the fish, you need to track them for a long time. When you see that it’s a good time, in that perfect frame where they are just silhouetted.

AB
So I can see that the AI is doing multiple tasks at the one time. I’m also, you did probably didn’t notice, but my jaw just about dropped to the floor where you said you are not only tracking fish, I can’t believe we’re having a 10 -minute conversation about fish, but I’m actually literally astounded.

Not only are you tracking a fish, but you just said you’re actually tracking the individual fish. That technology for humans was pretty well impossible science fiction a few years ago to bring it back to the domain of cars.

It was very common for a forward-facing camera on a vehicle to say, that’s a human, no worries. I can identify that, the silhouette’s known, even give it an ID, but then a few frames later if they happen to go out of shot is me and come back.

Before recent times, that would be rocket science, it would just be classified as a new human giving a new ID and would be consistent throughout the frames, but not consistent as I’ve seen that one before.

You’re able to do that in a brand new domain is just shocking technology and phenomenal that you’ve been able to adapt. Getting into the weeds and asking the big, nerdy question now, were you starting from scratch?

AB
Were you able to harness large, open source models that had some of the domain and latent domain knowledge or did you literally start line one, position one with training data and here’s what we need to do?

Mehdi 51:29
Yeah, I think AIMS was great in providing, they actually created a large volume of training data of the fish species, and they made it public for everyone. Gotcha. So if you are a researcher, you can use those open source data sets.

If you are a company, you can use it. And this is great. I’ve never seen such things in other countries. It’s a bit hard. So it requires a really dedicated team to create this, especially fish population.

There are a variety of species, and unless you’re an expert, it’s really hard to recognize them. At least I must have learned a lot. Well, at the end of your project.

AB
Can you now recognize fish at a glance – your superpower?

Mehdi 52:14
Yes, at least in a broad category, I can recognize that it’s still a challenge, not recognizing the different types of things. But yeah, I guess this is a good thing about geospatial industry that although it’s kinds of not these days, merged with computer science, computer vision, machine learning, this sort of thing, there are lots of overlap.

But you learn a lot working with different industries and end users and you learn the various things from agriculture, forestry, fishery, to insurance, oil and gas mining, urban planning, and various different applications that I think the beauty of surveying engineering and geospatial these days is one of these things that you will not find in other disciplines because other disciplines are just focusing on one thing and one thing.

AB
Almost anything, we are looking at the patterns to be able to transfer that from top -down imagery and what can I infer to the worst of murky waters around the coastline of this country is phenomenal, but I can hear that there’s the same patterns, the same challenges thrown at you, not the same tool sets, but the same way of tackling the problem in a logical and masterful way.

Can I draw back through your recent, and I’m going to say very recent LinkedIn history, can I ask what you’re doing in Vietnam and Indonesia in the last couple of weeks? You’re obviously linked to some Australian trade, attache, what’s it say, holidays, mission, thank you kindly, I was about to say holiday, there probably wasn’t a lot of time sipping a pina colada on the beach.

Can you give us a bit of a rundown on what you weren’t doing on the beach, but your kinds of meetings and the kinds of firms and the kind of teams you’re meeting with?

Mehdi 54:08
Actually, I’ve been very active in last year visiting different countries, especially in Asia and the Middle East, exploring opportunities and see how they can benefit from our technology. One particular category of clients are national mapping organizations, and especially they are struggling for creating foundational databases, let’s say, for example, national ortho photo.

And as we know that this data is critical for the economy. Once you’ve got a good quality foundation data set, you can distribute this to businesses and they can benefit from this. For example, orthophoto or photo map is one of them, a grand database of grand control point or secondary grand control point is another one.

Mehdi 54:58
For example, the national elevation data is another one. That was one of the projects that we worked with the national mapping agencies in Thailand, Saudi Arabia, and Indonesia, helped them to reduce the requirement for grand control point.

It’s a bit technical topic, but essentially, in order to be able to reduce the requirement for collecting grand control point by sending people to the field with the GPS, collecting the data, bring it back to the office during the geo-referencing of satellite or aerial imagery, this is a very time -consuming.

We developed a technology that can reduce the requirement of the grand control point, and this work was awarded by the American Society of Geometry and Remote Sensing, a couple of years ago. This technology is very unique, and we know that in the world, a few people can do such things.

We helped this space and mapping organization with this technology. That’s one kind of line of work. Another was in different markets. For example, in Thailand, Indonesia, agriculture is a big thing.

Forestry and crop performance, farm insurance is a big thing. Unlike Australia, which is property insurance is equally important. In Asian countries, we noticed that they are significantly different, and we need to pivot to agriculture.

That’s because Australia properties fall apart from each other a couple of meters, and you can see a nice backyard with a range of things. In Asia, you cannot find this. Often, you can see that you don’t see the border between properties.

It’s really hard to differentiate them unless you use the cadastral boundaries. Again, these cadastral boundaries in Australia, we are lucky because this is open to the public. You can get it, but in other Asian countries, this data is not available, and you need to be able to bear through some connection to get it, or you need to justify it.

AB 57:09
So in both cases, you’re talking of a national foundation database that is open, free data and but that the quality is all for SIGMA, it’s all the nines and it is constantly being tended to, there’s data governance, there’s data conservation and tending to it.

So there’s a common repository that people can trust and expand on versus competing regional and sub-regional and I’ve got a patch of data here that’s kind of okay, but it doesn’t match up with the national set.

Yeah. Can I actually just walk back a couple of terms we just talked about there? If people aren’t exactly au fait with them, I’ll give a lay definition and when I’m wrong, please correct me. Ortho, so ‘orthophoto’, orthographic, essentially means top down what we would normally associate with a paper-based map.

So it’s the, not the flattening, but it’s the view from the satellite view from space, the literal top down, there’s no 3D per se, it is just the, if it was squashed to a 2D, what would it look like?

AB 58:13
Another topic there which I’d love to talk about is a ground control point. Historically, survey points are the ones that we’d use if we’re doing surveying large sways of land and you can still run across them in many countries of the world, concrete blocks, there’s almost one on your local mountaintop that would be used and a brass plaque with a dot in the middle that your surveyors would lay there.

Ooh, got a light, that tool, but now we’re talking digital versions of those, so known nodes, known places, but rather than putting them physically in the land, what you’re saying is you can then find a way where if you find a really solid known point in your imagery, in your data, you can then go out or you can have someone go out with a GPS with multiple satellite signals and get such an accuracy on this one point, you can almost stretch your map to fit where it should be, is that like a retrofitting of the data set? So rather than starting with known data points and working forwards, you can work from a data set and infer where it needs to be to be 100% correct to that land terrain.

AB
What’s the advantage that a nation state would have in maintaining its own geospatial data?

Mehdi 59:57
I think one aspect is currency and the level of detail, because in the elevation data, we know that there are data from NASA that is available worldwide, but at the national scale, to speed up the construction project and different types of projects, you need better quality data.

And that means no more detail and more up -to -date, and then you can provide businesses with this data so they don’t need to create this data for themselves. So that’s one thing. But coming back to your question about Asia, one interesting use case that I noticed this time, especially in Vietnam, was illegal building detection through drone imagery, which because not previously, I thought illegal buildings happened only in Australia.

But in Asian countries, they got buildings and like in Vietnam, especially to be able to use their own imagery and find, especially people, it’s amazing. Sometimes they set up buildings overnight and it’s not a real building, but they know they just set up in this way and they start building.

AB
And then after a year passes, there’s statute of limitations and hey, I know that’s not real, but it’s been there for so long. It is now. So what you’re saying is it’s vital for every nation to own that data because of cultural norms of differences and also the fact that you do need people on the ground to validate that a global data set just doesn’t have the resolution.

The classic elevation data is from NASA from the space shuttle. They basically, as they were doing laps and laps and laps, they would have a single laser rangefinder would give single elevation as it was doing all its hundreds of laps.

That data is, well, 80s and 90s and I don’t think it was ran in the 2000s, but we’ve had lots of inference to upgrade that data. We’ve had a lot of calibration to make that data better.

AB
But that still probably is the baseline for the world’s elevation data. I know I was using it for a certain Australian client who shall remain nameless. That’s OK. And I was being challenged of what’s the accuracy.

And I’ll embarrass the team who I was talking to saying, look, it doesn’t matter where I live. Elevation is 12 metres or 12.0572 metres. And elevation is a guess. If you move two metres left, right, north, south, east or west, it’s going to change.

It’s nice to be close. But being correct to seven decimal places is probably overbating the cake slightly. So but that’s what the satellite was able to give us. It was sorry, the spatial was able to give us the first pass, but it’s an estimate based on thin data points..

Mehdi 01:03:09
Yeah, I think as you mentioned, elevation, I guess one of the biggest challenges in mapping community because still, AI still isn’t able to give us very, I guess, acceptable accuracy elevation data, and still we have to go through the regular processes or use the ranging technology that is a bit expensive.

And we know that if we use the traditional techniques in the mapping community, elevation is something that you get indirectly through a lot of computation. So just getting this data, you have to go through a lot of processing.

And so as a result, it’s a costly data set. Yeah. So it’s a bit of challenge, I guess, still in the mapping community, although they’re picking the laser scanning, ranging technologies like radar, LiDAR, but this technology is still, still expensive and they need to

AB 01:04:07
Can I tell my pub trivia point? I hope you’ve heard it. Probably all our listeners and viewers won’t have, so I’ll go through 10 seconds of it and I’ll see if — please, again, correct me when I’m wrong!

Apparently in the 1920s, maybe 1910s, England sent out a very expensive team of surveyors to get the height of Mount Everest. They knew it was the tallest mountain in the world, and thay they could do early estimates, but this time they said, “right, we’re going to do this properly scientifically”.

They started from one of the Indian coastlines and they worked their way methodically over many, many months to get the angle from their last known good to finally get the angle and they could cite Mount Everest.

I’m going to change tabs here for a second. Yep, we now know that Mount Everest is 29,031 feet plus or minus eight and a half inches — or bit under 9,000 meters, but the feet was the important one. They had error in their data, but they came back.

Well, when they finally took the final measurement of Mount Everest, they calculated it to, drum roll, 29,000 feet, and they went, oh no, we can’t possibly after a many multi-month, multi-million pound endeavor, go back home and say that Mount Everest is 29,000 feet on the dot. It would be as though we just said, “oh yeah, it’s 29 ,000 feet”. In a history of significant digits, they added, I think, six feet to it. They said 29,006 because then it sounded like they knew what they were talking about.

Of course, the number has been updated since then, but that just shows that the importance of elevation and how close do you get depends on the year, depends on the day, depends on everything. The elevation question is still something that, yeah, is a tough call.

Not only does snow and make things rise or fall or water and precipitation makes everything, the valleys get deeper and the hills slide down more, trees and foliage. There is almost no good answer for elevation here into that joke story.

Was I kind of right in that story? Had you heard that story before?

Mehdi 01:06:17
Actually, not at this level of detail, but it was fantastic and fascinating to see that people are interested to get the measurement very accurately. This is so critical to be able to get this. I think this is, of course, to get the attention of people worldwide, but in other applications, elevation is so important.

We noticed that in various applications from now at the age of climate change, estimating the height of the tree. I didn’t know that it’s so important because the height of the tree has something to do with shadow and shadows links to the consumption of water through air cones and this sort of thing.

So for local councils, it’s so important if you just give them the coverage of the tree, that’s not enough. They would like to know the height of the tree as well.

AB
I’ve noticed that there is almost no human superpower ever for estimating heights of things. Even estimating a telephone or a power-pole close to you, and you’ll be off by a massive factor.

There’s been times when I’ve flown the drone, kept the camera level and went excellent, it’s 14 meters high. It’s vital when you’re flying a drone to know that the highest thing in your vicinity is 15 meters high, therefore I’m gonna stay at least, you know, 10 meters above that.

But humans don’t have any multi -thousand -year history of estimating the height of things, it’s just not in our toolkit. So, glad to hear that.

Now, Mehdi, can we turn from what you’re doing last week in the last few weeks, you’re just sitting around the world, can we get you to put forward your vision of the future?

What is, well, first of all, what’s on the dance card for Mapizy, but also what are you most excited about in the next two or three years, even five years? Don’t go so far out that we have to put in, you know, 20 years on flying cars and that kind of thing, but what are the things that are really for you, your things that get you out of bed every morning, the things that you’re looking forward to seeing come to fruition soon?

Mehdi 01:08:25
Yeah, absolutely. I guess this is really important to know about the latest technology. And for us, to be able to help businesses with better quality data, with their new application, new use cases. Recently, we worked with mining companies for carbon measurement through satellite.

And this is something really great for mining companies, because it’s exactly fit their criteria. Because with satellite, we can access remote areas, we can get extra information they need for the carbon measurement.

The same concept for, for example, the farms. It’s really important to be able to have farmers to make sure that they are compliant, they reduce their carbon footprint. So these are the new things for insurance companies to help them get better quality data faster, more detail.

And this is something that we are working on. And there is no limit in this because more sensors and data becoming available cheaper, on a more frequent basis, we would like to be able to access this imagery, process faster, create the insight that industry needs to have.

And working with different industries, not only just insurance. This is actually my passion working from national space and the mapping organization through to local drone companies, companies like Toyota, Airbus, we’ve worked in the past for a time and we’re still having road mapping.

So doing various things, although we have a platform and product, but we are not limited to just one product. We are essentially a product and service companies. And this is actually my passion to learn from other industries doing different things.

And because that’s one of the reasons I quit my job in academia, because I did just building detection for 10 years. And I said, I don’t want to do building detection for the rest of my career. Very cool.

So that’s the great things about us. And you know, the future of geospatial is really bright. It’s one of the interesting science and technologies that is so critical for a nation because every nation has a geospatial agency.

That’s so important to be able to support various other organizations that they rely on this data. And the future is more sensors, better quality sensor. And so that’s a great future, I guess, for geospatial industry.

AB
Absolutely. Like the last few chats I’ve had, I support you a thousand percent. That’s the convergence of these fields that I’m loving to follow. The fact that the geospatial world is finally gone from famine to feast of data and more data is always better.

So this day and age in 2024, data is plentiful and we can store better, we can process it better. I’m looking forward to when that’s even more of a just a hand wave than it is today. It’s the days I think in the 90s when you’re playing with couples of dozens of digital images to try and rectify them for photogrammetry to now I can go into a museum with my phone and just casually scan a picture of a 3D mesh of a sculpture while I’m passing it by.

I’m looking forward to when the grandkids, I don’t know, the hollow lenses on the headsets and I’ll say it when these glasses that we’re wearing now are everything that we need to overlay data with the real world.

That’s the sort of the spatial AI that would be epic in my books, but you’ve actually been on the forefront of this field for such a wonderful period and it’s heartening to better get some of the sense of the data journey you’ve had, the business journey, but also that learning journey.

I’m wrapt to hear that the lessons that you’re playing with right now are transferable to other nations and you’re able to use that wealth of experience to assist other countries to get their own data sets into healthy states and really fast track.

AB 01:12:35
Absolute joy to have this chat with you. We’ll put thousands of links in the show notes. These tabs I’ve got over here which I’ve been casting my eye at the last couple of minutes and goodness me there’s so much I would love to cover with you and dive deeper.

Mehdi, can we get you back in a little while? Can we also anytime you’ve got something that is of great interest to this team, can you give us a call? We’d love to catch up and talk about another one of these topics that we’ve barely scratched the surface on.

Mehdi
It was an absolute pleasure to have a chat with you, discussing the interesting things in our geospatial industry. As you know, we are a world wide, we are a small community, but we have the biggest impact on various industries.

So it would be great to, as you mentioned, to be back in your show in the future and give you an update about the latest development in this space.

AB
Oh, that’d be lovely. And I, and we shall all live vicariously through your worldwide travels. I dare say your passport is well stamped. So my congratulations to you. Again, thank you for your time. Absolute pleasure.

Thank you. Goodness me. We’ve, we’ve had a long interview. We shall leave it here, but from everyone at Spatial, we’ll catch you on the next episode from us. Farewell.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

To absent friends.

The post Episode 022 – Medhi Ravanbakhsh: the OG of Geo AI & Remote Sensing appeared first on SPAITIAL.

This week, AB talks Alison Kealy, the inaugural Director of the Innovative Planet Institute at Swinburne University in Victoria Australia. The discussion covers the institute’s goals, focus areas, and Alison’s background in geodesy, positioning, navigation, and timing (PNT). Alison’s expertise lies in satellite positioning, quantum sensors for navigation, and the fusion of various technologies for accurate and trustworthy PNT solutions.

We touch on the institute’s focus on achieving the UN Sustainable Development Goals, harnessing innovation for real-world impact, and building partnerships within Swinburne and externally. AB also *finally* learns the right way to say the word “geodesists”.

https://www.linkedin.com/in/allison-kealy-7419804

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://www.youtube.com/watch?v=-fBsbLGdN7M

AB
Well, g’day and welcome to SPAITIAL. This is Episode 21. We are back after a minor hiatus. Yes, winter colds and perils did come our way, that kind of thing. That’s what happens in this part of the world in the Southern Hemisphere.

I have with me though, a beaming smile. Alison Kealy, welcome to SPAITIAL – so glad to have you with us!

Allison
Thank you so much, Andrew. Lovely to be here.

AB
Cheers. I’m going to say I had to look back – it was two months ago – I saw your LinkedIn profile saying, I have just been, you know, I’ve changed roles and I’m now the… I’ll give it a go, but you need to correct me: the Director for the…

AB
And here we go. Yes. The Institute. No, I got it wrong. The Innovative Planet Institute. Goodness me. Can we start there? Can you tell us what the Innovative Planet Institute is, what your role is, and what’s been on your plate the last little bit?

Allison
I’m two months into the role at Swinburne University, where they’ve established a new institute called the Innovative Planet Institute, and I’m the inaugural director.

And so it gives you the chance to build itself into what I’m calling sustainability at Swinburne. This is really where Swinburne wants to provide its people and technology solutions to addressing a wide range of sustainability challenges that the world is facing right now and framing it in the context of the UN SDGs: sustainable development goals to do that.

AB
Wow, that’s phenomenal. It’s so you say it’s a brand new Institute within Swinburne University. As you know I do read LinkedIn quite a lot. Swinburne just got a lovely press release last week. They’ve climbed the worldwide in Australia and Victorian in Australia University ranks. They’re definitely top 50. Top three in Victoria, I think. So numbers are going up, which is a glorious sign. Joining the ranks of other institutes within Swinburne, is it a brand new facet/silo or is going to be cross-collaboration across the Swinburne ecosystem?

Allison
There’s three other institutes at Swinburne, and obviously to develop solutions that address sustainability problems, you can’t do that in one institute. So there’s obvious connections into things like the space and health and defense and all of these platforms, data, AI, all of these things coming together.

And so it’s a real opportunity for Swinburne to leave Roof, which is what is really powerful capability for a small university into really making impact.

AB
Batting above its weight, love it. So what are the sort of the, uh, pillars within the Institute that you’re going to be driving? What are the, what are the focuses the foci?

Allison
I’ve kind of set centered it around for four key pillars and those four key pillars are energy transitions. So looking at how we achieve zero emissions, Supply chain, decarbonization, Our Future Living, How do we build cities of the future so that they’re not just smart, but they’re regenerating as well.

A subject matter called geo-resilience. I love the word, but that’s because I love the word geo and I love the word resilience. And I put them together. You know, it’s about people and communities and building resilience as we face more and more of the effects of climate action and cuts through climate change.

And then the other one is to do with transport and looking at how we would make transport smarter, which obviously fits with my role previously in the Department of Transport and Planning.

AB
Very cool. I also note with interest, we were delayed from when I first spoke to you when you launched in this new role by a couple of European conferences. Well done for having to do that work to travel to that part of the world and spend time. Well done. What’s your background, your fields? How did you (not be in this role) – but what are the sort of science journies – the career path – that’s led you to these things being front of mind?

Allison
Yeah, look, you know, I started with, I’m originally from Trinidad. So I started with a land surveying degree in Trinidad. And then I did a PhD in something called Geodesy over in the UK. And then I moved to Australia.

So my background is really in satellite positioning. So looking at things like positioning navigation timing solutions, which has now taken me into the world of quantum. So that’s, yeah, so very exciting.

And the potential of quantum sensors to play a role in navigation and positioning in areas where GPS doesn’t work. And so that’s my science journey. That’s my passion. It is in Geodosy and positioning.

And so currently, I get a chance to use a whole range of brand new technologies and sensors in order to do that as well, but then leverage that into the applications where GPS would normally be working.

But if it doesn’t, then what do we do to secure our banking infrastructure, our trains, everything?

AB 05:31
We had a chat with Phil Delaney in our last episode and one of the nuggets he left was how the Australian GPS network is being built up to be even more exacting and also the coverage to be made better.

And the number he put out there was billions of savings proposed in the future, A billion here, a billion there – pretty soon you’ve got some serious money on the table. Is this the brave new world of expanding the positioning around the world or is it, as you say, what are the kind of places where positioning is not covered by those constellations of satellites flying just above our heads?

Allison
So it’s two things, Andrew. It’s one – it’s anywhere that you can’t see the satellite signals. So in tunnels, underground, indoors, those sorts of environments. But then there’s the other thing of when you have applications that are safety critical, like our trains, and we want to make sure that we have a solution that is really, really one we can trust, then you want something else there that helps you do that.

And these alternative technologies help you to provide that trust in positioning that is required for this application.

AB
So complete secondary technologies or supporting technologies around the GPS framework?

Allison
I think you need ‘bolting’. I think you need an augmentation like the space-based augmentation system that Australia has just invested in, SouthPan. And then you need alternatives that aren’t subject to the same errors as the satellite signals are.

So you have an independent system as well.

AB
Gotcha. And I can reveal I was also in a European country a little while ago that is a known hotspot for GPS spoofing. So I guess say these technologies are also to combat trust in said signals and “am I really where I am or should I just fall into the ground?”

The security aspect is significant. I take it this conversation does have implications in that kind of space.

Allison
Yeah, so it’s, you know, the Quantum PNT problem is one that defence is very interested in. Sovereign PNT capabilities is one that Australia is very interested in, as are other countries around the world.

And we’re seeing lots and lots more the impact of spoofing in certain areas, which, you know, creates that unreliability and mistrust.

AB
Yeah, yeah, and a million mistrusts a second does tend to make a signal processor confused. And if it’s confused, that’s not a state you ought to be in.

Allison
Yeah, I think I saw a number somewhere in the US where if they lose GPS for a day, it costs about a billion dollars.

AB
Right. Yes. Yeah, we are talking what was military only technology in the 80s, I would say. And then was first released commercially, but with bricks, I guess, bigger than mobile phones of that era.

My father did have a mobile phone in that era, you know, carried around the suitcase of the battery with it. The GPS, the fact that it’s everywhere and augmented and we just rely on it. And it’s also, you know, the number of constellations that we can tap into is now increased, although perhaps there might be few less in the future.

That’s a conversation for something else. That’s, that’s all right. Maybe getting back to the Innovative Planet Institute. What are the first few things you’re doing? What’s on your, what’s on the dance card the rest of this year?

AB 09:21
What’s the, is it you’re looking for partners, you’re looking for projects, you’re looking for other students and building up teams? What’s the first 100 days look like?

Allison
Yeah, the first 100 days is actually, which is rapidly coming to me. I know, I’d say it’s already been two months. So what’s the next 100 days? It’s really about, you know, I’m new to Swinburne, building those partnerships within Swinburne.

You’d be amazed at the fact that that is a thing to do. It’s own challenge. So it’s doing a lot of matchmaking. A lot of blind dates.

So I’m going to try and just kind of get the right people into the right room, into the room with the right questions. So I think one of the things I’d like to focus on is what does impact mean for an institute like this?

And impact for me is about how do we get the innovation into the hands of the people who need it? So the decision makers, is it society? It’s unlocking it from where it kind of sits. So, you know, it’s getting technology and it’s getting people and it’s putting them together and saying what is the best way to provide a solution to this challenge?

And you know, that’s language. You know, everybody’s speaking a slightly different language, the vocabulary, it’s really kind of a collaborative endeavor that requires a little bit of finessing.

AB 11:01
Everyone has their own lens. If you come from a certain field, everything, you know, hammer looks like a nail, but also to communicate the values that ethos is always a hard thing, even to figure out what that is, but then to, you know, say it clearly plainly is a great thing.

So it is the goal of the, can we be calling it the IPI, the Institute of Planetary Institute – is it going to be a short name, a long name and acronym forming here, or we keep it correct first?

Allison
I know Australians like to shorten their names. I’m going to keep it formal. But within it we will have a lot of very, very informal conversations and relationships developing in order to achieve the kinds of output that I think really presents itself as an opportunity for Swinburne as well.

AB
No, fantastic. Do you have a premises, a level, a room, a building carved out? Are you operating virtually? What’s the future land grab of the Institute?

Allison
Yeah, Land grab is a good 90 day plan, I have to say.

AB
Have you sighted exactly which place you’d like to work at?

Allison
They’ve given me a lovely office in the engineering building, which houses me, which is great. It is a virtual institute, and that’s been brilliant given everything we’ve learned during COVID. I don’t know what it would have been like before that, but given the experiences of COVID, it’s a really lovely platform for engaging with much more people, increasing the diversity of participation, but purely making this a virtual interaction as well.

And so in terms of space, we have a lot of space that there was on campus that works really well. I’ve stumbled across a beautiful visualization hub full of big monitors and screens in which you can run a lot of spatial problems, digital twins, AI on digital twins, and invite people in.

So if I could tell people using this podcast, if you want to come in and play in our spaces, we’ve got some beautiful spaces for co-designing solutions.

AB
I absolutely would love to second that. I haven’t yet played in that area. I do operate around the Swinburne area every now and then, so I will take you up on that offer to go and play. But it is such a vital part.

There’s a lot of talking about ways to communicate and show deep nested 3D dense gnarly problems, but there’s nothing quite like experiencing it. I don’t know what the ratio of, well, if a picture tells a thousand words, I don’t know what the ratio of spatial intelligence is to words, but I really don’t want to write a 20 page thesis versus “come and put this headset on” or “come and play with me and we’ll find out in five minutes”.

Obviously, the speed of knowledge transfer is a much different for those types of problem sets.

Allison 14:16
Yeah, and you know, a spatial person trying to show a planner, the impact of something happening or building being built or one decision making process, it’s much more interactive to see what the effect of that.

And if you think about, you know, the future where you could layer onto that AI, where you don’t even have to play around with computers, you can just tell it to show us certain things, you know, then you have a really powerful narrative environment for, you know, multidisciplinary, transdisciplinary interactions.

AB
I’ve been in a lot of digital spaces for a very long time. I would say how long, but back in the early days of – gee – even Photoshop and imagery, I many times had someone say, excellent, can you move that image to the left? “Yep. 10 seconds done.” Now can you move it up? “Yep. Come back tomorrow.” It’s that there’s some things that are easily done in some places and others are “that’s a ground up rewrite – that’s a grand up rewrite”.

And it’s really lovely to separate the technical and let the technical happen, but have people just come in the front end and ask the questions unfettered and obviously these are spaces being, you know, multi-screen multi-collaboration and in a world with multiple headsets all working together that mode of I can tell you I can talk you through it or I can just show you and you know, let’s experience it.

That’s exciting.

Allison
Yeah, and experiencing it’s something like that together. I think we’ll open up more dialogue. It will encourage more conversation about what is possible, you know, so intent and context and all of these things become conversations when you move beyond technology.

AB
Yeah, indeed. The check is changing so fast. It’s nice to keep the right handle on it. But as you’ve been saying in your first 100 days, it’s the talking. It’s the trying to piece together what’s the aim, what’s the plan.

And this is easily doable in X. This is all let’s put that on the long path. And the other end of the stick is often that thing. Yeah, that’s a button on your phone now. It’s easily done. So the pace of change is, I’m glad to see you’re harnessing it and using it as best you possibly can.

Allison
Oh, absolutely. I’m so excited. It changes so fast. I tell people, I remember when I started using GPS and there were only a few satellites in the sky and you had to wait until it.

AB
“And look, there’s one coming!”

Allison
Yeah, yeah, that’s 3 AM. Yeah, indeed.

AB
Now, when I fly a DJI drone, it connects to two or three satellite networks and oh, it’s got 14 or more. It kind of should know where it is reasonably well. That’s untenable. Yes, as you say, 20 years ago, that was like “back in my day”. Okay, there we go, kids. We’ve said it now.

So, I can see five research programs you’ve got here on the website.

https://www.swinburne.edu.au/research/institutes/innovative-planet

But future places for living, urban infrastructure, urban decision making. Wow. Can you tell me about those sort of problems that you’re looking to solve? What size scale? Micro, I mean, what are the definitions of the problems that you’re trying to look at?

Allison
Yeah, I should also add that in my 90 days I would be doing a website update.

AB
Okay, gotcha. Asterisk: this information may change.

Allison
I mean, I guess a previous incarnation of the Institute was to do with smart cities.

AB
So, many of these things have collapsed into one of those pills that you were talking about before?

Allison
…to plan it and allow this to increase the scale of what we do. So we will still keep the smart cities focus and pivot to something along the lines of future living, you know, create an affordable housing, create in green housing that is affordable.

Those kinds of challenges will form part of what future living is going to be. But we will extend that into, well, how do we build transport networks that support new communities? And how do we manage energy and supply energy to cities of the future?

And what does that mean for us as a society in terms of skills and workforce planning for the future as well?

AB 18:52
That’s exciting. That’s a massive ring bit. That’s not a small task. So obviously you’re jumping in with two feet. As I say, I’ll come back to the question of what’s the rest of this year to formulate and plan and prepare and then what’s your goals for 2025?

What would you like to be doing if we had this year to get in trouble and stuff?

Allison
So for the rest of the year, I’m looking at building up a key set of programs, one of which is obviously aligned with quantum. Looking at where Swinburne can position itself in delivering to the different roadmaps that are being created in Victoria nationally and internationally.

That is one thing. We also have the French-Australian energy transition being initiated across universities, across industry sectors between France and Australia. And so looking at how we would build that into really harnessing international collaboration, as well as local intellect to deliver that and what those programs might look like.

And so those two plus actually getting the Institute bedded down and working is going to be a really important thing. And for next year, I think, you know, I’d really like to position ourselves to where Swinburne is seen as a key accelerator of the SDG goals.

And so, you know, when we’re talking about this next year, Andrew, which I hope we are. What you’ll be seeing is that we’ve created the pathways and the partnerships that are actually allowing us to do more rapid, not just new innovation, but actually getting innovation out of the university and into the practitioners’ hands as well.

And that’s more about policy and standards and all the things that have to happen.

AB
Lovely. Can I twist around to talk about the buzzword bingo that we just got our viewers and listeners to tick off their bingo cards? Can we go back to quantum? It definitely is, yeah, it’s a big word.

If you can’t say “AI” and “quantum” in any marketing brochure or website, you’re sort of, you know, what are you doing?

Can you tie together both what you’re doing with the Institute and quantum, Swinburne, but also your studies and the conferences I’ve been watching you do with living vicariously through your LinkedIn photo reel.

Quantum PNT: can you explain to us what that word means in that context? What it’s distilled down to or what that one word, if you can unpack it, is it just marketing? What’s the core of it?

Allison 21:56
Okay, so the core of quantum, and I’m not a quantum physicist, right? So I, that’s my disclaimer.

AB
Is there a quantum physicist in the room? No, that’s okay.

Allison
I’m oddly married to one, so there’s sometimes is. Okay.

But in this case, I think what I’m interested in is, is just the sensors that allow us to do more accurate measurements. So when we talk about quantum, we’re talking about sensors that are typically able to perform measurements with very, very little error in it.

And so what that means is we have the ability to do things like measure acceleration very, very precisely. And doing that very precisely means that we’re able to measure things like the magnetic field.

We can measure things like gravitational effects. We can measure a lot of sensing parameters. So, so just to be clear, when people talk about quantum, they sometimes talk about quantum computers, or they talk about quantum communications, or more secure communications.

When I talk about quantum for things like positioning and navigation, I’m talking about sensors.

AB 23:10
Gotcha. And you’re using quantum in the sense of the word of the smallest increment possible. Quantum being beyond micro, being the smallest measurements, the most precise. Gotcha. So error is reduced.

And then that is added up, averaged to let us humans know. But on the smaller scale, you’re doing the lowest level math, science, physics to make that happen.

Allison
Yeah, and so what it allows us to do is measure things like understand how the gravity field of the earth is changing, which is really important for mineral exploration and for some of the geodesic work that we do.

But the accelerometers that are being built using quantum physics, they’re actually allowing us to do things like navigation underwater, for example, very, very accurately. Because GPS isn’t going to work underwater.

It’s a great alternative too. And so that’s where, and you’re right, you can’t get away. If you’re not saying quantum in every conversation, are you even doing science? That’s it.

AB
How do you possibly get funded if you don’t sprinkle one of those three, four, five words into any year? No. And that just it. It’s tricky when that is the actually correct word and it’s meaningful and pushing the boundaries of science correctly.

First is the startup in the garage that says we’re doing quantum AI, synthetic cyber yadda, yadda, or you’re just laden in every word sadly is loaded and has multiple lenses on it. So it’s great to unpack that.

Thank you for that. Also, thank you for telling me the precise pronunciation of ‘geodesy’. As someone who sees these words more than more than says them hits was it it’s always hard with the emphasis on the right syllable and geodesy or geodesic.

Obviously the pronunciation changes depending on the word, but geodesy and yeah – I read more than I talk. So it’s it’s often hard – thank you for helping with that. That’s really good.

Could you define geodesy as a field and and how it relates and how it’s permeated throughout the things that you’re playing with them?

Allison 25:24
Yes, so geodesy is actually the study of the size and shape of the earth and the positions of things on the earth. So it’s the fundamental property that we use for determining things like where are our property boundaries?

You know, how do we know where things are when we give it coordinate values? It sets up what is called the datum against which we’re able to do things on a global scale. And so it’s the foundation of maps, it’s the foundation of navigation systems.

It’s the core piece of information that allows us to understand where things are on the surface of the earth. So, you know, when we hear things like Australia is moving seven centimeters north-east, yeah, the reason we’re able to say these things is because we’ve defined a reference frame that allows us to detect those kinds of movements.

And so geodesy is that is that study. And we have a worldwide skill shortage of geodesy. So this is my pitch for anybody who is interested in this. This is the place to think about it.

AB
Now this wasn’t a course in my career is counselling in high school. Was it for you and for others? Is it a science first then specialise? Is it a specialised university that teaches geoscience, geodesy or is it…

AB
What is the career path? This is not a field that I’m… There aren’t many ge… Hang on. There aren’t many geodists out there?!

Allison 27:13
Yeah, there’s not many geodesists there even less they normally came through a land surveying program, which is why I did geodesy, but even the land surveying programs we’re seeing a lot of them dwindling, not just in Australia, but nationally, internationally, it’s a global problem.

This this shortage and it is recognized. So yeah, you’d have to, you know, there is no geodesy course that you signed up for. You do a bit of it in physics, you do bits of it in space science, you do bits of it in in geomatics or in surveying.

But, you know, there is no course called Geodesy in Geoscience Australia. There’s a lovely internship program where people can go and learn about Geodesy as well. But yeah, it is it is one of one of the things that I do like to tell people, you know, we’ve actually see people, you know, when you take up your phone and you’re using a map on your phone with GPS, you’re using Geodesy, right?

It’s mass market Geodesy we’re talking about now. And so we need to get more more people interested.

AB
I hear you. That’s a lovely ad. And it is true that, you know, just as you think you’re in, I don’t know, the start of high school and primary school, I’m going to be a doctor,a lawyer and your horizons open.

I’ve watched my kids go through university with good and firm ideas, but still no one even knows what’s at the end of those, you know, choices three and four years later, and it’s great to see that these are specializations within deep fields.

I did a startup a decade ago called My Perfect. Basically, it was the opposite of Google with rather than starting with a search engine to have millions of results, you’d start with lots of results and play 20 questions till you got down to your perfect thing. The best one of all was My Perfect Career. It was actually neat that it asked questions that split the field really quite wildly and came up with a thousand different choices of what you should do. And it was kind of fun to play my own game of what should I be in or what should I’ve done because if you don’t know, you can’t make a choice.

So it’s always good to find out that there are more options out there than what you could possibly think of until you’re halfway through. So brilliant.

Allison
Yeah, I’ve got two kids. One just started uni and one’s in the Year 11. And yeah, it’s a really interesting conversation to have with them, given all the choices that they do have now and trying to inspire them into STEM related areas. It’s a challenge still.

AB 29:59
Can we even walk back a little bit more? You talked about another well, an acronym which you explain, but it is all over your LinkedIn field and that is PNT, position, navigation, timing.

Obviously, that’s an acronym that is so frequently said, it’s turned into an acronym. What is the international focus on it? Or is Geodey the underlying maths and PNT is the daily application? What’s the linkage between the two?

Allison
Yeah, Geodesy is the underlying, you know, the sort of theoretical infrastructure under which positioning, navigation and timing is, it sets up all those kinds of reference frames within which we make measurements of positioning, navigation and timing.

The PNT aspect of it refers to the applications, the technologies and the systems that are used to provide position, navigation and timing. So, satellites, for example, would be part of that PNT architecture, where quantum clocks are now part of the timing and architecture as well. But underpinning all of these would be the infrastructures of Geodesy.

AB
Got you. With more satellites up there, are you able to make use of the low-earth-orbit constellations? Starlink is the classic one, but there’s a lot of, there’s many frameworks. Well, it’s a large place up there, but certainly it’s getting much more crowded. Is that adding influence? And is that adding its own level of accuracy to the conversations? Or is it the ground-based systems like we’ve talked about before with Phil, but also, is it the actual underlying hardware and the techniques, software based on the ground that are actually helping to really bridge these gaps between where we want to be and where we are today?

Allison
Oh, funny, it should lead me down this pathway, Andrew, because I’m going to say that the thing that I fundamentally adhere to is the idea of fusion, right? So I don’t think there’s going to be any one one algorithm, any one technology hardware technology, it’s not going to be any one thing that provides the answer to the increasing number of applications we’re trying to service with positioning and timing.

So I think for me, anything that that comes up – low earth orbiting satellites, quantum, whatever it is – it has strengths and it has weaknesses. And the way we maximize and optimize over this is we combine them together.

And the mechanism we use to combine them comes down to what the application needs. So if the application needs our high accuracy and high trust, then we should be able to sample from our little toy box of things – and grab everything we need to give us that solution. Whereas if we’re not so interested in accuracy, it’s on our mobile phones, we just want to get to a restaurant. You know, that’s a different different type of class of application that I think we’re talking about.

So for me, the low orbit in satellites offer a lot of opportunity for us. It comes with its own complexity as well. You know, you need a lot of them as well to cover areas. And so, yeah, it’s it’s all part of a hierarchical solution.

So for me, you have things on the ground, you have things in the middle of the air, so drones and things like that, you have things near space, high altitude balloons, and then you start to have things in space.

And these are the layers of sensing and capability that we have to help build our solutions for things here on Earth. And they all have to work together synergistically.

AB 34:06
Very cool. That’s great to hear because that means you’re not putting all eggs in one basket. You’re just hoping up to whatever comes along. And I guess a follow up is what sort of year on year benefits are you hoping for?

Well, what are the metrics you’re seeking or seeing? Are you saying we’re getting better tolerances? Are you saying we’re getting better trust? What’s a fireball team? Any of you?

Allison
I think a 5 or 10 year view starts off with something as simple as sovereign. You know, I think the 5 or 10 year view has to be what is Australia’s sovereign capability in this area, but also what is Australia’s contribution to the global you know, toy box.

AB
Gotcha.

Allison
Yeah, yeah, you know, what are we doing? And then, you know, there is also this idea of, you know, where can Australia position itself to benefit from some of the things that are happening internationally?

So one of the things I’ve been working on is this space and spatial roadmap, where spatial, which has always been a user space technology, can now leverage the greater investment in space sensing to deliver a greater downstream benefits for things like bushfires and emergency response and climate change and things like that.

So I’m hoping that we’ll see intelligence and AI become some of the offerings that Australia and Victoria in particular, I’m really trying to get Victoria’s the smart state to catch on. You know, so that we build intelligence solutions that are almost technology and the sector at no stick.

AB
Look, as a fellow Victorian, I support that. That’s all good. If I have a conversation with someone in Sydney, I will have to change my tune. That’s okay. This is just the classic Australian debate of who’s got the best footy team, it’s generally the fact that it works.

You;ll be dealing with partnerships and projects – are you looking to on-board students and directly have active projects or is it more outward facing partnerships and projects? Is it outside Swinburne? Is it bringing up people through Swinburne and releasing them or is it to grow the institute and be a powerhouse of R&D?

Allison
I think Swinburne has a real brand in providing technology solutions outward facing. It’s one of their moonshots. You know, every industry partner gets a technology solution. And so I think that’s where their brand is really strong.

And so I’m really keen to really do that, that partnership building with goals and impact in mine. But I don’t think we can do that without building up the workforce that can support that technology rich future as well.

So it has to be done both ways. But the idea would be that we have students who are working very closely with industry as well so that they’re building up not just the theoretical and academic skills, but they’re also getting that on the job training that helps them to build out the support that the industry will need.

AB 37:52
Alison, thank you so much. Awesome to chat to you. I’m gonna say it’s opened my eyes and my ears to the kinds of things that are on the riser. Obviously drawing together such, I mean, the field of view of the Innovative Planet Institute is obviously so much bigger than yes, first website look, but also that vision for the future is just glorious to hear.

Where do you, we asked where the field is for PNT, but where do you think the institute will be in five years time? What’s the, not the 50 year time, that’s, you know, we’ve all got matrix style jacks in the back of our head and we’re all wired into the central thing.

Where’s the sensible first horizon that you’d just love to be?

Allison 38:38
I think I would like to be able to point to a few things that has made society better. I would like to see some of the work that we’ve done actually being used in creating better resilience, meeting the needs of the most vulnerable people in society today.

And I think if we were able to even do a little bit of that, that’s what I would think we’re on the right track.

AB
Oh, that’s heartwarming to hear. And it’s perfect to finish with Alison. Thank you so much for your time. I appreciate it. We’ll leave contact points you LinkedIn profile will leave a link to the website.

Looking forward to seeing a version 1.1 or 2 .0 release. That’s all great. But it’s really happening to hear that vision and the passion come through. We all fully support it. Even if we’re not in Victoria, Australia.

Obviously, there’s ways to get in touch and get involved. So thank you for your time. Congratulations and welcome to the directorship of the University of Innovative Planet Research Institute.

AB
Alison Kealy, thank you so much.

Allison
Thank you, Andrew.

AB
That’s it for now. Catch us all next time. Next time more interviews coming through. This is perhaps going to be the format going forward. So looking forward to having some more of these kinds of chats. Colds notwithstanding. Yes, it is cold in this part of the world. Heaters on. Cats are hibernating. But from us at SPAITIAL, we’ll catch you next time. Bye bye.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

To absent friends.

The post Episode 021 – Allison Kealy & The Innovative Planet Research Institute appeared first on SPAITIAL.

This week, AB talks with Phil Delaney, CEO of MapAI. This new company aims to provide a conversational interface for querying spatial data using large language models fine-tuned for spatial analysis. Phil discusses the motivation behind MapAI, its current state of development, and plans for future deployment and integration with existing platforms. He also shares insights from other projects he has been involved with at Frontier SI, a social enterprise focused on spatial technologies, including Farm Map 4D for monitoring land cover and Value Australia for automated property valuation.

We also touch upon the Australian Satellite-Based Augmentation Service (SBAS) called SouthPan, which provides precise satellite positioning corrections, and its potential $6 billion in benefits across various sectors in Australia.

https://www.linkedin.com/in/phillipdelaney

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://www.youtube.com/watch?v=2lMGd1Bygec

Phil discusses the motivation behind MapAI, which arose from a collaboration with the University of New South Wales to explore the spatial capabilities of large language models like ChatGPT. The goal was to create a fine-tuning layer that could translate natural language questions into spatial queries and provide appropriate responses, including maps and visualizations. MapAI aims to integrate with existing platforms and data sources, acting as a conversational interface for spatial data analysis. Phil shares details about the current state of development, including early proof-of-concept deployments with a major data company and local councils.

Phil provides insights into other projects he has been involved with at Frontier SI, including Farm Map 4D, which used satellite data to monitor land cover and assist land managers in sustainable practices, and Value Australia, an automated property valuation model that incorporated a wide range of spatial data and allowed users to simulate the impact of infrastructure changes on property values.

Phil discusses the Australian SBAS called SouthPan, which broadcasts correction signals from space to improve the accuracy of satellite positioning systems like GPS. He explains the motivation behind SouthPan, its potential benefits across various sectors, and the economic analysis that led to its operational funding. Phil also touches upon the GNAN program, which distributes positioning corrections over mobile networks, achieving even higher accuracy on compatible devices.

AB
Well, g’day and welcome to SPAITIAL. This is Episode 20, and we are joined by a special guest this week. I have Phil Delaney with me. Phil is – goodness me – wearing many hats. We’ll find out exactly which hats he’s wearing today.

I was intrigued by Phil’s LinkedIn post a couple of weeks ago. He is CEO of the newly formed organization called Map AI. And yes, dear listeners and watchers, you can tell that I was intrigued instantly.

https://mapai.net

AB
Phil, welcome. Great to have you with us. Look, your history is wild and varied. And I say ‘hats’ is probably not even the correct terminology for it. Is there a plethora of many hats phrase we can possibly put on you?

But by all means, can you introduce yourself a little bit? Tell us a bit about some of the hats you wear right now. We’ll get up to MapAI soon.

Phil
I wear two main hats at the moment. One is as Deputy CEO and Chief Growth Officer of Frontier SI. Frontier SI is a social enterprise that’s been around for about 20 years. And we really sit between research, government, and industry to help bring interesting ideas through the real-world products and services in the area of space and spatial technologies.

And then through that, one of our recent innovations with our partners at University of New South Wales has been to combine generative AI and spatial analysis. And so through that, we’ve created a subsidiary company called MapAI, and I’m the CEO of that as well.

So I spend, at the moment, about half of my time with each of those two roles.

https://frontiersi.com.au

AB
Wow. So, we’ll tell you what, we’ll leave MapAI for a few seconds because that’s going to be a large tangent. Frontier SI. So, 20 years of a social cooperative. Can you explain what the brief has been there, some of the projects you’ve been involved with?

Phil
So, we were formed in about 2003 and originally funded under the Federal Government’s Cooperative Research Centre program. So that’s a program that’s been around for a really long time that looks at funding collaborative innovation in a whole lot of different industry sectors, everything from hearing to pork production to mental health.

And so we were funded as a centre to focus on spatial information. We had two rounds of funding, one that went for seven years, one for eight years. So that covered our first kind of 15 years of operation.

So we were funded through a government program, but that covered some of our funding. More than half of it came from contributions from industry, university, and particularly government agencies. We were a CRC for about as long as you can be under legislation.

And so at the end of that, the thing that our partners valued the most, even more than a lot of the solutions that we’d created, was the model of collaborative problem solving that we made available.

So we’re now coming up six years at the end of June, which will be six years since leaving that program, kind of standing on our own. I’m still looking at a similar mode of operation, but we do a lot more of the direct problem solving ourselves with some internal technical capability and specialists, particularly in the area of precise satellite positioning and in satellite image analysis and other image analysis.

But we still very much worked through that collaborative approach where almost all of our projects have some combination of government industry and academia working together to solve major problems that use space and spatial technology.

AB
And look, I need to mention, you’re also a local lad. We’re both on this side of the Southern Hemisphere, here in sunny Australia, granted it’s sunny as of this day, but it’s cooling down. That’ll give us some ‘place’ when we’re talking federal government – we’re talking about the Australian Federal Government.

I’d love to come back to that concept if we had time of the global positioning satellite in Australia. The acronym is definitely new for me, so it has escaped me. But I know that that has a lot of headway also around this part of the world.

Frontier SI had yet 15 years of fully funded CRC, and then the last many years it’s evolved into perhaps more traditional commercial kind of mode?.

Phil
Yeah, I mean, so we’re a not-for-profit organisation, but we like to describe ourselves as a social enterprise. So that really means that we use very commercially grounded strategies in order to make a substantial difference in social, environmental and economic outcomes.

So as a not-for-profit organisation, we don’t have shareholders. We do run a lot of projects that cover a really wide variety of work from independent advice that can happen anywhere in the world through to that technology development kind of projects and any surpluses that we make along the way through the projects that we run, then get reinvested into creating new technology like MapAI.

AB 05:57
Lovely segue. I’d like to thank the Honorable Minister for his question. That’s a sitting duck for us.

MapAI, so the big announcement was three weeks ago. Can you tell us about how MapAI was formed? By all means, we have a teaser of what it is, a website, which we’ll link in the show notes.

https://mapai.net

AB
If need be, we can do some screen sharing and I can talk to it for those listening in with headphones. But can you tell us about MapAI, its purpose, its future and where it sits right now?

Phil
Yeah, absolutely. So, like I said, we’ve had a long history working with research teams to try and take ideas through to real world products and services. One of the universities that we have a long relationship with the University of New South Wales.

And last year as part of just the regular discussions that we have around finding ideas that mentioned that they were kind of noodling around with chat GPT and trying to investigate its spatial capabilities and had seen certainly some good ability to translate someone’s question into code that we could execute.

But a lot of the understanding of the nuances of spatial information and why it maybe requires a little bit more special consideration than some other types of data meant that we saw a good opportunity to create some fine tuning on top of what at that time just ChatGPT was doing in that conversational AI large language model type work to see whether or not we could get a working prototype that would answer questions in a bit of a different way.

AB
The problem normally within the spatial world is that people don’t have the vocabulary to know what they can ask. So was this the layer that lets people ask deep, dark questions and have sensible prompting?

Phil
That’s exactly right. I mean, I think that what we were really trying to do then once we made that first connection that let people interrogate a single data set was to say, OK, great. One of the biggest things, as you say, that inhibits people better using the insights that they can get out of location information is that they don’t know what questions to ask.

They don’t know what data might be required. They don’t know how to translate between the raw question that they just need answered and what that might mean from a process and data perspective. Yeah.

And we wanted to try, and absolutely not for every application, but for a lot of simpler data sets, well -structured, well -understood geospatial data where people have regular questions. We wanted to be able to create a chat -based interface where someone could just type a question, and then the work would happen in the back end to create a query automatically that could be executed and would either respond to that user with a text response, some kind of table showing the information, or, of course, a map.

AB
Brilliant. That is, I think, where the future is heading. I know we had a chat last episode where we said the tools for the future will be less ‘fingers, keyboards, mouses’. It’ll be a good microphone and a glass of water.

I think our voice boxes are going to get slightly hurt during the next couple of years. I’m always reminded by the Hollywood scenes of, you know, when someone is talking to their AI chatbot – it kind of makes perfect sense when it’s one-to-one, but I can’t foresee an office full of a hundred people all talking to their computer at the same time?

I think that would be like, I guess, a phone call center, but without any end customers.

Phil
So, that’s right. I think the other complication there is that we can often type faster than we can talk. And the other great thing is that we can edit before pressing go, which is still something that we haven’t managed that real time editing of the question that I’m trying to ask via voice hasn’t really got there yet.

There’s no such thing as “backspace, backspace, backspace” in vocal terms yet. I’m not sure if we’d want voice dictation to be “no, no, dammit, hang on, wait, stop that, no, no” – you know, the “no, yeah, stop that, pause, please”?!

So, MapAI, is it cloud-based, or is it going to be local-based. How does it interrogate data sets? Do you have to introduce it specifically, or you can marry them together?

Phil
So the way that we’ve planned MapAI to work, we figured that there were enough platforms out there, whether they call themselves geospatial or not, there are enough platforms out there that have location -based information that we didn’t want to introduce yet another platform to the sector.

The way in which we want to work is to be a white-labelled solution that integrates into existing offerings that really creates a chat layer between a user and the data that a platform is already managing.

At MapAI, we don’t want to be a UI company, unless there’s a big gap. We certainly don’t want to be a data management company. The idea behind what we’ve got is that the data stays where it is at the moment.

The platform stays where it is at the moment. We just add in two new layers. One is a context layer that helps us to describe what data we can interrogate. And the other one is a question and answer API that feeds the questions from the user to us in our cloud environment.

And then we feed back the code that’s required to then answer that question for the user.

AB 11:45
That’s very smart. No one needs to reinvent every single wheel – and interfaces alone are perilous and large… and expensive! And if you can get away with, again, having the microphone at no interface, that’s a great option.

But also, same with last week’s chat, an interface-less interface: how do you tell someone what they can ask? How do you prompt the user in those early stages to do some other training, learning or exploration mode?

Phil
Yeah, that’s a good question. And the way in which we’re doing that at the moment is by focusing our early proofs of concept on quite defined applications. So rather than a free for all, come and ask any question of all this data that we’ve got.

We’ve got a defined application, which is, you know, for example, a customer service team inside a council or an asset owner trying to forecast where there might be demand for new types of their assets in the future and having a very defined use case, which is only about asking questions along that particular problem.

AB
So the users, they may not be completely across all what they can do, but they at least the data set is home to for them. They are already across what its purpose is, its intent is, this just introduced them to new ways of actually finding out what’s

Phil
Yeah, that’s right. I mean, I think that it’s easy to get carried away with the potential about where this might go in the future, but this is all still very new. So our focus has been at the very first instance, just trying to test whether or not the large language model actually answers questions in a useful way.

And so to do that, we want to make sure that the scope and focus of where our first deployments happen are really clear.

AB
Yeah, it makes perfect sense. So you say you’ve got a fine-tuned layer on top of ChatGPT. That by itself is reasonably groundbreaking. It’s pretty darn hard to do any fine-tuning on large language models.

How did the team come to that conclusion and how did the team tackle that? Was that a lot of seriously supervised training or just exploratory work to actually be able to find that extra layer to have that command set active?

Phil 14:09
It was a bit of a combination of both. And the other thing that I’ll say is that we started with ChatGPT, but the way in which we are deploying our fine-tuning model is as a middle layer that can work in between a user and any large language model, be they, you know, the ones coming out of Amazon or Meta or some of the open models that are out there at the moment as well.

Our aim was to provide a fine-tuning solution that could be a large language model independent as much as possible. That way, we could integrate with any solutions that people were already using from a large language model perspective, but would also mean that we had the flexibility not to be locked into a particular large language model solution, as we’ve seen happens with many large corporations as they get kind of more share, they can then do what they want with kind of pricing and usage and that kind of thing, and that can cripple a business that has a sole reliance on one model.

In terms of the way in which we go about creating that fine-tuning, I don’t want to say too much – mostly because I’m not the technical person here – but what I can say is that we have a whole range of question-and-answer types covering a wide variety of potential spatial concepts and spatial questions, and then start to look at how we create pairs or multiple matches between what a question might be trying to ask and what the spatially appropriate method for answering that question is.

AB 15:43
Perfect. A separation of powers of large language models of choice. By all means, if you started this concept late last year, and here we are in, well, we’ll call it May, but actually it’s the start of June by the time this episode comes out.

It’s been another few epochs even this year of iterations of not only the large language models, but even the fast following the HuggingFace open source world models, student-teacher, decimated local models that have, sure, they are a few percentage points down in terms of efficacy and brilliance, but they are local – and mere mortals like us can run them on pretty beefy units.

So this does open the door now to not having to even be reliant on the big four/the big five in the cloud. Is there a local mode in the future? For local versions of this kind of tool?

Phil 16:40
Yeah, definitely. I think there’s two approaches to that. One is the co-pilot type approach. It’s still all based on a cloud solution, but one that’s ring-fenced to an organisations cloud tendency, so nothing needs to enter or exit the fence that’s around what an organisation owns on the cloud.

And the other one, certainly a little further in the future, I wouldn’t want to put a date on it, but we absolutely talk about it, is the ability to have a model that can be deployed, you know, cloud-based, server-based, even, you know, powerful local machine-based, as that pair between our fine-tuning and a locally deployable large-language model.

AB
That’s spot on perfect. I think the pendulum is going to swing from “let’s go cloud and let’s find the value and the power of these tools” to okay, sure it’s great, but we’re now racking up not only queries, but we’re racking up internet traffic in and out to get our data in and out.

This was the question of “where is my data?” That’s right. And what are people doing with it? I imagine you’ve met the kind of people in your past 20 years who still want to see their spinning hard drives and LED lights flashing in a server room somewhere…? I dare say those kind of people are not multiplying, but there are still many fields where I want to know where my data is.

Can you give us like a use case of the kind of special queries that MapAI can help with? Pick any of your virtual customer bases, but what are the kind of unknown unknowns that MapAI can reveal that you can do that mere mortals would not really know how to do unless given this kind of tool?

So I want to start by saying that nothing that MapAI does is something that can’t be done by a mere mortal, as you phrase it. But what I would say is that a lot of the work that it does is trying to alleviate a lot of the base work that, say, our very busy GIS teams might be doing.

Responding to simple questions, like trying to get a filter of a subset of data from within one data layer, or looking to find the addresses that are around a point, or just some relatively simple one, two, three combinations of datasets and features within that dataset to understand questions around assets that might be a risk of flooding or locations to target for a particular type of demographic profile.

All those questions that are relatively simple, but in that feedback of, I need to find an analyst, I need to wait for that analyst to be free, they will provide me an answer, I need to iterate that a little bit and then get back that can take hours, days or weeks, depending on how busy a team is.

The idea here is that for a lot of those more simple applications, we can just type in a question, look at the answer, then refine the way that we asked the question.

AB
So in the bulk of digital queries, the dead easy ones are dead easy. The interface probably gives it to you. The other end of the spectrum is the gnarliest, biggest, deepest analytical questions, probably reserve that still for the analysts to do the big gnarly multi-week. “I want to find every single house that has, starts with a letter Q where there’s one dog and one cat”. That’s a query for someone who has more time on their hands, but that leaves the middle wide section of “find me all the, how many primary schools are there within an X radius at this point” or “refine this”…?

Phil 20:41
That’s right. So the common, I guess, starts to a lot of the queries that go into MapAI at the moment are show me, or show me where is, find me, summarize, or display, something like that.

AB
Yeah, gotcha. Powerful tools that are probably by definition the highest volume kind of things you’d like to know.

What state is MapAI in? The website… it’s a little bit light. So only about three pages?

Phil
We’re only… So we’re about six months in to MapAI development in terms of real time. Especially when you take out the Christmas break and all that kind of stuff in terms of the effort that’s gone in so far.

And anybody who works with the research world knows that often it’s hard to get a lot of useful work out in a short timeline for that, particularly as the problems that we often tackle with our universities are pretty big and long-term.

Saying that, the team in the City Futures Research Centre at UNSW have done a really great job in continuing to release and improve a version that can be kind of deployed and demonstrated. And so it was only kind of four months in that we had a version that we could ask questions of.

And while we don’t want to be a UI company and we don’t want to be a data management company, obviously to show that it works, we had to take a bunch of data. So a couple of months ago, we had a dump of a range of data out of the New South Wales government in Australia’s open data website, things like planning zones and properties and addresses and roads and those kinds of things.

And we described that and we put it into a simple user interface and then started asking questions of it and used that as a way to kind of refine how the model worked, the kinds of responses that it gave.

And pretty quickly, we saw that even the version as it was then could provide a lot of really useful work, say for people that only wanted to ask questions that needed, you know, three or four or five different data sets to be combined in order to be able to answer them.

Phil 23:02
So we started to have a look at identifying a few organisations that we could then prove this concept with, even with this really early work out in the real world. So we have now… And I can’t talk about it publicly quite yet in terms of specifics, but hopefully I will be even by the time that your podcast comes out, ideally.

But we’re working with a major data company as a way to integrate with a lot of the work that they create in terms of unique national -scale data products and a way to query those in a brand -new way.

And then we’re also working with two local councils, one for more of an external type application in helping to answer questions that come in from the community. Things like, you know, when is my bin night?

Or what are my planning zones? You know, can I open a hairdresser at my address? These kinds of questions. And then the other one is more of an internal use case in supporting field staff to better understand the sites that they’re going to visit.

So even with this early large language model, you can see that with each of those applications, there’s probably five to 10 questions that are pretty… Predominant. …repetitive, but site independent, so that you need to answer them in a slightly different way, depending on where you’re going.

And we can answer them using kind of a handful of datasets. So it’s with those that we’re now kind of rolling forward to demonstrate this in the real world. And having those agreed was our kind of prompt to then let the rest of the world know that MapAI existed.

AB
So we’ll put in show notes or a extra update if the podcast and the video beats this extra news. But by all means, the show notes will hold the source of truth for the latest release of MapAI.

Cheers for that. Look, just changing tack a bit. MapAI by all means is gonna be the right kind of tool for – we’re talking local councils, we’re talking service based organisations, those who already have spatial or a location bent.

I roll back to some of the other things that are on your bio, which are quite fascinating. I’m looking at FarmMap 4D, are there more sorts of tools in a similar vein? I guess we’re getting deeply into agriculture, farming and the permaculture world. Can you give us a brief about some of the spin-off companies and side projects you’ve had from Frontier SI?

Phil
Yeah, sure. So there’s probably two good examples. I can talk about Farm Map 4D and then I can talk about another more recent commercialisation, which was for a company called Value Australia.

Phil 25:56
So Farm Map 4D was actually – nd its original name was the Natural Resource Management Spatial Hub – this was the reason why I was employed by Frontier SI in the first place almost 10 years ago. And what it was was a collaboration between a range of natural resource management organisations, you know, government-funded organisations that look after the environmental management of a particular region.

Focused on the middle bit of Australia, the really kind of dry and arid bit, where we have a lot of cattle grazing. And the idea behind that was to take remote sensing data, so images from satellites where we have derived insights about the quality of somebody’s land, and take that away from being just a tool used from government to monitor the performance of land managers into something that we could put into the hands of those land managers to help them better make decisions, right?

That was the whole idea behind that. So it used a derivative satellite data product called Fractional Cover or Fractional Ground Cover that this was. What that actually means is that it simply takes a whole lot of satellite information, calibrated to a whole lot of ground sample sites, to tell you for each little square of your land over the last 30 years, how much of it is a live vegetation, dead or non-photosynthetic vegetation or bare ground, right?

So, and when you track that over 30 years, you can start to have a look at how sustainable is the level of cattle that I have in a particular paddock. So are we getting to the same level of green cover every year, or is that slowly declining over time?

We were going to have a look at how the practices then impact on things like erosion. So as we’re overgrazing land, that leads to a lot more runoff of the good soil on top, which goes into the streams, and then Queensland washes out into the reef and causes a lot of problems there.

So again, this is about the long -term trends and helping people understand things like sustainable carrying capacity, erosion risk and things like that.

AB
Up 30 years worth of data on this site>

Phil 28:10
I was working on NASA’s Landsat data at the start and then using some of the European Space Agency’s data from Sentinel to then enrich that for more recent time at a higher resolution. And it was run as a collaboratively funded project between a range of different organisations.

But as all projects do, they come to an end and people didn’t want to see the end to the solution when project funding runs out. And so sometimes having that operate as a government service is the best way for that to happen.

That’s what we do with our positioning work. Sometimes releasing something open source is the best way to get it maintained. And sometimes putting it into a company is the best way to have a sustainably managed solution.

So that was the case for the Spatial Hub, which rebranded then to Farm Map 4D.

AB
That is of immense value to know, over such a long term. I take it at the start of the 30 years, your data points were fewer and further between, but to have that such a long term smoothing graph of bare earth is slowly climbing?

Phil 29:34
And it’s really a testament to the work that NASA has done in keeping the Landsat satellites consistent and regular over the last 30 years to allow that kind of long-term history. And in fact, while we were running that project, we were highlighted on the NASA news page as a great example of using their data.

AB
Sounds like you were “employee of the month” there for a while.

To your other example – this one was a bit more recent the ‘large acronym’, yeah.’…?

Phil 30:08
The stuff with what became Farm Map really went out into the private world in about 2018, I think, and is still going and serving customers now. A bit more recently than that, we did work, again, with the team at University of New South Wales in automated and interactive property valuation.

So we ran a workshop a number of years ago with a whole range of, again, industry, academic and government people to try and find interesting problems where we thought that spatial data could play a new type of problem-solving role that it hadn’t necessarily been doing so far.

And one of the use cases that was raised by government and validated as a good industry problem as well was trying to understand when we do something new, major, like building infrastructure, like a train station, what impact does that have on property values?

And if we can get a way at the moment, that would cost you hundreds of thousands of dollars in consulting time to try and do something, answer that kind of question. Is there a way that we can quickly and easily model what it’s likely to be?

Because that then plays a critical role in a business case for infrastructure. Because as property values go up, that obviously benefits the community, but it also increases things like land tax to help to then fund that infrastructure.

So we did some work to try and work out what’s required to make that happen. And the biggest thing that was required to make that happen was an automated land and property valuation model.

We need to understand what the value is, but even more than what it is, why is it that value?

AB 32:03
Gotcha. So, the breakdown of the components of, you know, the closeness to the primary school adds this much. Was it that discrete or was it fine-grained?

Phil
We get a lot of really high-quality data assigned to each individual residential property across Australia’s capital cities, something around 10 million properties, I think. And it was a whole lot of data that was just kind of straight out of, you know, government databases or high -quality private databases, and then a whole lot of things that we kind of derived as well using, you know, geoscapes building products to look at, you know, the heights of buildings, all sorts of different things.

That then let us work out for each individual house, not just generically what might the value be, but for this house, what is its value and why is it that value? Because then when we understand why it’s that value, that’s when we can start to go about playing with future scenarios.

So either for an individual landowner, maybe looking to say, what happens if I, you know, should I renovate my kitchen or out of bedroom? What’s going to create more value for my house? Through to the government, looking to build new train stations, revitalise the quality of the local school, and the quality of the local school is a big driver of value.

Access to open space, all sorts of different things. We then built a system that let people interactively, you know, like SimCity, create that new infrastructure and then immediately see what the impact on property value was.

That was a great use and as well the actual valuations model itself was not just useful for the governments but also to banks who helped to make our risk decisions about mortgage lending and things like that.

AB 33:56
This is the holy grail, actually, of spatial intelligence. Sadly, it’s not quite the ‘hello world’ – I think it’s the first thing that everyone jumps to, to see if they can figure out how much their house is worth.

And then the way that you’re doing it is the way to crack it, of doing it large and doing it, throwing, I guess it’s, classically, the data science way. You throw all the data at the problem – you just throw the bucket at it, and you find…

Phil
…a billion data points in the model that we were running.

AB
Ouch! And then what falls out what actually was relevant, vs what was irrelevant. The number of cafes in 10 square kilometres didn’t really matter as much as hospitals and primary schools and police and those sort of things.

We tested well over 50 different modelling approaches as well to work out which ones were both accurate and explainable. Because having not just an accurate model but one that you could talk about with somebody was also really important.

AB
Yes, when computer says X, that’s nice. But in this vital field, you need much more than that.

Phil 35:02
Yeah, that’s right. So we put that out into a company two years ago. Andt we needed some people with great data about land – what we’d really built was the engine – we needed someone with fuel.

And so we ended up commercialising the company with PEXA, who are a digital conveyancing company in Australia. For those not from Australia, every time you buy and sell a property, the transaction will go through PEXA’s infrastructure.

And so it really complemented a lot of the work that PEXA were doing, trying to diversify what else they understand about land. And so that Value Australia company has seen a great life growing as a commercial entity over the last couple of years as one of PEXA’s family.

AB
The digital marketplace is certainly a beautiful theory and when one organisation can not only be the transaction hub but also be the value add, it’s a bit of a slam dunk that makes a perfect marriage there.

Can we change tack one more time? One last time from acronyms and long project names.

Now, I said earlier, I do not have in my head what this acronym or wording is but the “Australian uplift of GPS”. Could you give us a briefing on that? Your involvement and where you see the future going?

Phil 36:28
Sure, so the generic name of the system is SBAS, which is Satellite-Based Augmentation Service, and Australia’s SBAS is called SouthPan.

Southpan is is a way for us to broadcast from space a correction signal to satellite positioning like GPS. There’s a number of different constellations up there operated by different countries or consortia, like the European Space Agency.

And every signal about positioning has an error that happens because the atmosphere changes over time. And so, when we position ourselves on our phones or on our cars’ GPS or whatever that might be, we generally are only accurate to 3 to 5 metres, which is great to know approximately where we are in the world and what road we’re on for navigation purposes.

AB
…just enough to be able to get you what lane you’re in, but that’s actually about the limit, yeah?

Phil
Not even – sometimes only just the road, because three to five metres is enough to put you on the other side of the road.

AB
Which would be bad.

Phil
And so what we needed to do was to work out a way to correct each of those signals.

So what we have is a bunch of really well-known locations around Australia and New Zealand, where we know really precisely where they are. And what we do is we collect all of the signals from the space -based positioning and say, well, for this, you know, for the little region around where this precise thing is, we know that the position from satellite X was out by about this much.

And so when you collect all those things together, you can start to greatly bring down the accuracy of the satellite position. But then you’ve got to go get that to the user.

AB 38:26
To find a way to tell someone that they’re off?

Phil
That’s right, and that’s where an SBAS solution comes in. So there are other SBASs around the world, and they’ve primarily been brought in for aviation purposes, so help with safety of life of autonomous flying in planes coming in to land more safely in airports and things like that.

But in Australia, we didn’t have enough airports for the use case to stack up just on flights.

AB
We talked about the ‘big hole in the middle of the continent’. Here in Australia, the population of people not on the coast in this continent be measure in hundreds of thousands.

Phil
So what we did a few years ago was work in partnership with Geoscience Australia and with Land Information New Zealand to get a pilot of this signal going from space, so where we got a communication satellite, we’re able to broadcast that correction signal, get some devices down here that could read and understand that, and then we deployed that across 27 different projects, I think on nine different application sectors across the economy in Australia and New Zealand, and then did a rigorous approach to look at what was the economic benefit in having this position, this kind of 10 centimetres, versus having three to five metres in agriculture, in resources, in navigation applications for the blind, in drones, in augmented reality, all sorts of different applications.

And essentially we came out to a figure that I think was around about $6 billion of benefit over the next kind of 10 or 15 years that was all driven just by having a much more precise location. So the government then used that to get operational funding for that, and that’s now a service that’s available to anyone really across the Asia Pacific region to access, if they’ve got an Airspace enabled device.

AB
Gotcha. So first off, gold star for you. That’s phenomenal. Secondly, what kind of device is needed to receive this correction signal as well? Is it beyond consumer products or is it coming down to that level soon?

Phil
And most consumer products at the moment, although we’ve got a range of open source kind of examples as to how people can build something, it’ll cost kind of $1 ,000 to $2 ,000, as opposed to maybe $50 ,000, which is what it was beforehand to get that kind of accuracy.

I’m saying that we also have another program that we’ve been working on with Geoscience Australia, which is called GNAN, which essentially distributes those same corrections but over the mobile network, so via internet connection.

And for that, the accuracy is actually better. It’s down to around two to three centimetres. And in theory, we’ve got that working on Android devices. So we haven’t kind of mass marketed that on Android yet, but Android lets you have access to the raw positioning data so we can correct it here.

And we’ve had that kind of accuracy on Android phones and tablets. Hopefully, Apple might do the same kind of access to data at some stage, but… opens APIs. Yeah, but then you have the gaps in the marine areas, in the areas that don’t have internet coverage.

You aren’t able to get that correction, but anywhere that has mobile reception, we can start to then incorporate that in our daily applications.

AB
Is this a worldwide phenomena? Is this a movement that is going to be coming in the next two or three years that it’s just a matter of time and pressure and priority?

Phil
Ideally, yeah. I mean, there’s nothing that stops it working anywhere in the globe except for communications. And most of those accuracy correction services exist in most places around the world. So it’s certainly coming more and more.

As with any major new capability on, say, a mobile device, you’ve then got to balance things like battery performance. Because as you can imagine the compute required to go from what road am I on to where in the lane am I is substantially is a substantial increase.

AB
So it’s about spheres of influence and how good do I need to be right now? So if you need to be, yeah, choosing the right lane on the highway, that’s kind of neat. And I imagine most GPS’s are just guessing.

They assume you’re on the correct side of the road until you’re not and then politely inform you to come back. Gotcha. So might there be a future where just as we see world time service to, you know, go and check with the world clock or world clocks, would there be world time GPS grand station correction services as a…

Phil
Yeah, absolutely. And like I said, you can commercially buy them in many places in the world at the moment as well. A lot of the driver behind this was to make those even more accurate, but to also make them take away that cost barrier so that the benefits of that kind of accuracy could be available to anybody.

AB
I was just overseas in a place (that I didn’t tell my wife till afterwards) was the worldwide center of GPS spoofing.

Phil
That’s another topic entirely that we probably don’t have time for!

AB
We don’t, indeed. Phil, thank you so much. Absolute pleasure to have a chat.

What a seriously wide range of topics and projects you’ve been involved with. Not only through frontier SI, but deeper and darker history, your bio on LinkedIn is phenomenal.

https://www.linkedin.com/in/phillipdelaney

If you’re interested in MapAI, please reach out to Phil. We’ve got as a links to MapAI and there are, I believe contact details on for you on that website.

So just final thoughts: do you think that this year is going to be a groundwater year for, language being the tool of choice – is that where that AI is trying to strive for its future customers?

Phil 44:37
I hope so. I think if we go back, you know, 15, 20 years, Google Earth and Google Maps was a major explosion in the use of location-based information by everybody. And, you know, at the time, I think a lot of people were wondering whether or not it would be a threat or an opportunity, but we’ve got a lot more demand for spatial applications now that people think about spatial as part of their daily life.

Really, I hope that that combination of simple kind of chat and large language accessibility combined with location will be the next, you know, major inflection point for how useful location -based information can be around the world.

AB
Hear hear – I fully support that idea that I guess that’s why we’re here having a having a long chat.

Phil we all really appreciate it. We’ll let you get back to your day and for those listening and watching along, everything’s in the show notes. But for now we’ll sign off on this episode of SPAITIAL.

Phil
Thank you for your time.

AB – Andrew Ballard
Spatial AI Specialist at Leidos.
Robotics & AI defence research.
Creator of SPAITIAL

To absent friends.

The post Episode 020 – Phil Delaney & MapAI appeared first on SPAITIAL.

Mirek and AB discuss the rise of fast large language models and their impact on various industries and workflows. We cover the recent announcements and capabilities of models like Meta’s Llama 3, Google’s Gemini, OpenAI’s GPT-4, and Microsoft’s Co-pilot Plus. We explore the models’ speed, multimodal capabilities, context windows, and conversational interfaces – and discuss the implications of these advancements, such as improved human-AI interaction, potential privacy concerns, and the challenges of integrating AI into existing workflows. And because we can’t help it, we speculate on Apple’s upcoming AI developments and the future of AI assistants in coding and robotics.

We’re also publishing this episode on YouTube, if you’d like to watch along in full living colour: https://youtu.be/ROBjyhYZKgs

We begin the episode on Meta’s Llama 3 and Google’s Gemini models. Llama 3 is praised for its speed, with responses being generated almost instantly, making it difficult for users to keep up. Gemini, on the other hand, is highlighted for its multimodal capabilities, allowing it to identify objects in live video and engage in conversations about them. We also discuss Gemini’s large context window, enabling it to build conceptual knowledge during conversations.

We analyze OpenAI’s GPT-4 and Microsoft’s Co-pilot Plus models. GPT-4 is praised for its conversational interface, including synthesized voices with human-like qualities. However, concerns are raised about the potential privacy implications of Microsoft’s Co-pilot Plus, which can record users’ screens to provide context-aware assistance. The discussion also touches on the challenges of integrating these models into existing workflows and the potential for AI hallucinations or inaccuracies.

We speculate on Apple’s upcoming AI developments, given the company’s control over both hardware and software stacks. Anticipation is that Apple may announce tighter AI integration across its devices and operating systems during the upcoming Worldwide Developers Conference (WWDC). The discussion also touches on the potential for Apple to improve its virtual assistant, Siri, and integrate AI capabilities more deeply into its products.

Finally, we shift to the potential impact of large language models on coding and robotics workflows by exploring the benefits of using AI assistants for tasks like documentation, code generation, and problem-solving. However, we also acknowledge the limitations of these models and the need for human expertise in integrating and understanding complex systems.

AB
Well, G’day, welcome to SPAITIAL. This is Episode 19 coming to you the last week of May after a gap of one week. So it’s been a fortnight since we had Merrick, who’s here online. G’day, Mirek.

Mirek
Hello, how’s everybody doing?

AB
Good, good. Yeah, we had a great reaction to your All about ROS, which has been good. A lot of podcast downloads. The YouTube channel is starting to bite a little bit, which is really good. We are definitely in the playpen of load numbers.

So, hey, if you haven’t checked out the YouTube series, it is what you are listening to now, but you can watch it. So tell you what, we’ll wave – something that you can’t see on the podcast. There you go.

You also can’t see Hubert in the background doing laps. He’s the turtle. And you can’t tell what Mirek is drinking at this moment. Look, welcome. This is almost a “news of the month” kind of thing.

AB
There has absolutely been a trend that we’ve been watching. In fact, about three weeks ago, we were about to do an episode on this topic. We’ve got pulled away. But in the last three, four weeks, this topic has just blown out of proportion.

What is it? Well, you probably read the title. This is the rise of really fast, large language models or really fast models. A question we’ll ask sometime in this episode is, are they just language models?

Are they also spatially-aware general world models? Casting the clock back about four weeks, I’m actually looking back at Meta’s announcement. So Meta, i .e. Facebook, late April, April 18th, they announced the release of Llama 3.

[https://ai.meta.com/blog/meta-llama-3/]

Llama 3 is an open source language model – and language only. It doesn’t do any multimodal: no images, no uploads, simply text. But the big thing there was that it was fast. If you played with any of the GPTs before, you’re sort of used to typing in a request and that tiny pause, the pregnant pause, the little pause where it goes and has a bit of a think and comes back with, yeah, a wall of text and often the wall of text, you can kind of scroll and catch up with it.

Well, literally four weeks ago with Llama 3, you couldn’t do either of those things. The moment you hit enter, you pretty much got the wall of text and you couldn’t scroll fast enough to keep up with it.

What that meant was it suddenly opened up doors to, you could normally outthink what you were doing with a large language model, with the chat modes. Llama 3 was the first one really that made it quite astounding that the moment you asked it a large, deep question, it would be bang, there’s the answer.

Mirek, had you tried Llama 3 at all? It is available on a website called Groq.com, which is easy to get to and nice to find. It’s also been hosted.

Mirek 03:12
I don’t think I did. I just follow the news and you know, I’m I’m playing with open a eyes like this and greatest every now And then but I’m I’m what’s happening. Is it just faster or is it better? There were a bunch of news from Google also About their new Gemini and I remember the the previous Gemini wasn’t this glamorous, right?

There was some there were some funny moments there

AB
Yes, it was hallucinating a little bit and it wasn’t quite taken off the board, but certainly it was put up in great lights and then maybe deprecated slightly. Yeah, look, next on the list is the Gemini, and I’m glad to hear it’s Gemini, not Gemini, so the old NASA rocket platform with multiple ways of saying it.

But Google came out with that just at the same time as OpenAI. We’ll talk about Google first, it sort of makes more sense. They were also the number one thing, the most amazing thing was it was fast and multimodal, so the demos they were showing were not only text -based, long queries, and bang, wall of text, which was brilliant, but grabbing the live camera out and actually walking around the offices looking at things, sort of doing what YOLO would do, you only look once, which ironically just did a new version just yesterday, I think called version 10, which is faster, so that’s the threat of this episode.

But the Gemini model was able to take live video and not just identify objects where needed, but have conversations about them pointing to a pot plant while you walk around an office and asking, what is it?

[https://gemini.google.com/app]

The fun one was looking at someone’s computer screen at a wall of code and asking to decrypt and have look and explain that code, and it was happening fast. Oh my God, and we’re going to have to pull out the phrase again, it’s one of my favourite phrases to drop in a meeting and basically scare people.

It’s nice when you can paraphrase Stalin, I’m just going to look at your eyes now. No, okay, not scary. His quote was, quantity has a quality all of its own, talking about takes and soldiers in World War Two, but to badly paraphrase it is, does speed have a quality all of its own?

Does going fast get you out of trouble so much that it can do more things than if you were doing something well but slower?

Mirek 05:43
It’s hard to benchmark these models right – or compare them with one another so it’s really not just the speed it’s it’s what you’re getting yeah how useful it is and I mean you you mentioning that multimodality that Google demonstrated why I think there’s something different happening than than YOLO. YOLO is is a model that you train to recognize classes of objects and so fundamentally it can only recognize what you trained it on so unless that is happening in those demos which is possible you could pre-train models to recognize a certain person there’s probably something else when you you’re able to you know detect from the image some some some more more information than just you know bounding boxes and respond to that I hope yeah

AB
Yeah, that’s what the Gemini showed of at least it can do live video object identification, which is not new. It was really just the amazing thing was how panning around an office, it would just find things.

Sorry, it wouldn’t highlight everything because that would make your screen quite cluttered. But if you pointed it at something and asked a question in context, it kind of figured out what you were asking about and it could have a serious conversation.

The reason I’m asking about that bad quote about Stalin is does having the speed of reaction go from, I guess, in human interface, HCI, H, anyway, human interface guidelines sort of Higgs, isn’t it, go from being if something has a weight, then humans can also react and run with those pauses.

But if something is instant or near instant that crosses the gap to changing the mode of interfacing to being conversational and I hate to say getting a little bit like the humanistic bot chat bot that we all think they are.

Mirek 07:47
Well, it’s definitely interesting for like, you know, human interaction and getting as much as possibly done in a little time at Google, the Gemini announcement also was about something else. Why they have this this very large context window now, I think they said million tokens or 10 million.

Yeah, can correct me. So that’s, I think, more interesting than the sheer response time, because with large context window like that, you can remember things and you can sort of build up context on top of the training during the conversation you have with the human.

And that can be much more interesting, I believe than just than just pure speed. But I really can’t tell the difference between, you know, those benchmarks and our people who are doing great work actually doing all that and comparing what they’re getting from these models.

I can’t do that. It’s I have other things to do.

AB 08:49
There’s two more models to talk about, and then I do have a tab, a resource, there’ll be a link in the show notes, to a lovely, you know, one of those population graphs that moves over time, but this one has to do with the chat response.

So using ELO scores of the major models, essentially humanistic and responsiveness. So we’ll tackle that as the end of, you know, trying to look at the overall comparison between our models over time.

But hot on the heels of Google, of course, came OpenAI’s 4 .0, which of course inspired the title of this episode, Oh My, for a few reasons. The similarities were not, you know, were quite the same as the Gemini, multimodal images, video, being able to pan around, but the real killer feature and all the last week of news, which we may cover later on, is regard the live chat with the synthesised voices that had some, well, had ums, nars and giggles and little bits of humanistic input to make it more than just a friend’s Siri and Alexa.

Mirek
Yeah, some sound suspiciously close to certain humans that we know, right? We’ll go straight there, yes. We can recognize by their voice.

AB
Oh, yes, Sky. I must say I was able to use it for the first few days And my wife was able to use it while we’re at and about but it has vanished and there is some Good drama and some good back and forth over that Let’s leave Scarlett for a second But by all means it is a absolutely new way of working If you can text quickly or use the microphone to do your own dictation into a text field, that’s fine That speeds things up a lot But the open AI is mode of just having an ongoing open microphone conversation with your AI That had some character I’ve got to say In actual fact, I would have loved to have seen like we’ve been used to buttons on models that have a heat control if it’s hot, it’s a bit noisy and Hallucinating if it’s cold, it’s a bit mechanical and a bit straightforward We’d love to see a heat sort of dial on 4 -0 Essentially while it’s lovely to have your AI sort of Yes, well, okay Put in those bits and bobs to make them sound human.

It gets a little bit tiring after a while And I’d almost like to take the gaps out of Some of that and have an option where I’m in a hurry. I just want the answer. I don’t need the I Don’t need the icing on the cake.

I just need the cake Did you get a chance to try the chat mode of GPT 4 .0?

Mirek 11:48
I didn’t spend much time with it, but I believe you can donate and you could do that for a while. I remember my girlfriend having formal, like all the GPT version draft something for her. And then it was like too pushy, too salesy.

So just, she just said less hyperbole. Like it’s as simple as that. And it produced something that was like human language, you know, not too pushy, but you can, I think you can customize the output a little bit like that already.

So that should probably work with synthesized voices also.

AB
Nice I didn’t do my free prompt engineering to say I want this voice but I also wanted terse and less of the stumbles and giggles and stuff like that

Mirek
But I believe the context window how would that like to with that customization because I believe what you get with like every new prompt and GPT these days on open AI’s website is a new instance that doesn’t know anything about anything else So the customization you can just inject a prompt right before everything But that only gets you so far.

Yeah in the way of like making the model kind of more In general behave in you know the terms of what you actually want from it and how you want to get it

AB
Yeah and I’ve got to say that interface of live chat was pretty phenomenal. It was kind of the interface that I think people had been forecasting for a very long time and again with that bad quote from Stalin it does change the game a little bit to have something that responsive and lets you stay in the flow as the human as opposed to I am driving a cursor and I’m waiting for the screen to scroll.

In this case having a conversation with a chatbot, an agent is enlightening. It starts a new way of working. It means that you can on the fly refine it. I’ve just been loading up a few tabs here which might be on the show notes and in the video for some same queries across all these four different large models that we’re talking about today.

I’ve got to say copying and pasting and typing was probably the hardest bit. I did the same typo on two out of four frames. So that’s my fat fingers or my muscle memory not working well. If I could have just said oi try this spoken out my prompt across all four models that would have been brilliant.

I think the new tools are going to be a good microphone, good set of headphones, quiet room and large screens. I’m not saying keyboard and mouse is going away but this actually does start that conversational extended conversational mode which up until the moment like now there’s always been a microphone button on almost every text input option or every keyboard has got a microphone option so you can dictate but this changes from dictation to conversational and yeah pretty wild.

I must say I should have

Mirek 14:59
Mean yeah, I see it coming. I don’t think we’re there quite yet And I think there’s like maybe not all sorts of work that you want to apply this to I was just listening to this conversation a podcast by Neuroscientist his name is David Eagleman and he wrote a book with his friend who’s a musician on – you know how all these things kind of come together artificial intelligence human intelligence how we’re creative and how that happens and So they argue and I agree with that these AIs are already very creative But what you can do like say you’re writing a piece of music.

[Spotify Link]

So you work on the whole piece all the time you come back and make changes and you know You you see the whole thing in its entirety and you figure out how this doesn’t work so well and you do this in software also In other areas of you know intellectual work We don’t work like that with AIs. We just feed it something and we expect the perfect result but like evaluating that and coming yeah, you know making Fundamental changes.

Yeah, or little changes as we go little tweaks is really difficult Yeah, it’s like it’s a one -shot as opposed to a week -long endeavor. Exactly. Exactly. It’s this one shot Kind of like give me a good brief for a graphics designer about the logo That you know is for the company that and you might spend that then it was something that’s Refining it averaged because of this whole data compression that is doing.

But then does that and I mean we’ll see how far we can get with these kind of systems But it’s interesting to sort of even like it shines light on various things that we do and don’t think about that much and you know things about how we learn and how we create and What does that actually?

Anyway, I get to philosophic This is what I see. I see what you mean, but at the same time I don’t think it’s gonna like replace Intellectual work just yet.

AB
Gotcha. This is the version of the machine can’t sleep on it or come back with a good idea on day three or four at this stage.

Mirek
Also, the reality will be quite different than what you see in these tech demos. I wanted to just reiterate back to this Rabbit R1 orange gadget that we mentioned on this podcast. I was just watching this YouTube video by a pretty big channel called Coffeezilla.

We can link that in the description. And he looks into that product that seems to do nothing of what it advertises, and they based all these promises of how it can do everything for you. You just talk to it and it will order your food, order your rides, handle your day -to -day mundane things.

It does nothing of that. They based all this on saying they’re developing this large action model, which is supposed to be this new smart thing that makes you… …makes the rabbit click. That apparently doesn’t exist.

And what it is, in fact, actually it is a GPT that’s instructed to not tell you. It is a GPT. Plus some UI scripting on various websites that order your food and do other things. And so that means that when the interface of the website changes, this whole line of functionality stops working.

So this guy on YouTube tries all that was advertised in the promotional videos, and none of that works. And he can’t even tell his position from GPS correctly. So that’s like that level of concerning.

Mirek 19:00
So don’t spend your money on that. I think we should not advise on buying that. This is entertaining to watch so we can link the video and have people decide for themselves. But even the way the company answers reacts to the criticism is a huge red flag.

They pretty much gaslight the guy and say that he’s not an AI expert and they will only take criticism from AI experts and with a certificate or something like that, they say. So that’s… It’s very entertaining, but don’t spend your money on it.

AB
Makes sense. The Rabbit has been put in the same bucket by some large reviewers as the Humane Pin. I’ve got to say I love the concept of both of them. The Humane Pin, I think news of the last few days is that they’re appetizing that they’re happy to be bought out looking for a buyer, looking to get absorbed into one of the big four.

It kind of alludes to a conversation that is about these large -language models. We’ve had Meta, Google, OpenAI, and the last one is Microsoft, and then the hanging asterisk in the air is Apple. With those five, those are the largest companies out there by market share, by thought share.

How possibly – I’m going to ask the ETH app, but please, if you can have a think about it – how possibly can a company that’s coming in as one hundredth at the size have both a product and a software focus, hardware, software interface?

It’s almost no doubt that it’s going to be impossible for the small fry to break into these large markets now. The dawn of the web, 25, 30 years ago, everyone could do their own thing and there was some amazing one -person, two -person, five -person bands did some epic work that became the big players, but now we’ve got to the point of the big players are sadly probably 89% of the market.

It’s going to be almost impossible to break in.

Mirek
i think there’s still room for lots of creativity and we’re just discovering what these models can do and i wouldn’t underestimate the creativity of startups but you have to put some actual work into it to create products that you know do something interesting and this seems like too good to be true you know many of these products that will apparently magically do everything for you it just doesn’t seem like we’re there quite yet and it seems like yeah the product’s actually over -promised and under -delivered quite often so

AB
No, it feels like it feels like it’s a bit too easy to do epic marketing, nice PR.

Mirek
I don’t want to sound just negative, I started noticing language models for software packages, like I think 3JS that I’m using for something, they have documentation and next to it is a language model, you can talk to it about the documentation and examples and I think this is enormously useful.

But it doesn’t do everything for me, right? It’s still work that remains to be done and things need to be… I think I just talked to somebody that there’s a lot of work that remains to be done in integrating all these services together into products that actually make sense and it’s not just the language model, there’s so much to be done and if you can sort out different kind of area of problems or pain for the users and just combine that with this, there’s a product right there.

AB 22:46
Yeah, so having a large document library like 3JS and all the comments there is great to have that as a searchable, talkable documentation index, but that’s then a silo, a pillar, which is itself smart, but it’s separate from the others, surely.

So there needs to be a time when either there is a standard interface between silos or something that can come across all of the documentation and have topic specific querying. So right now we probably couldn’t ask any one of these large models, okay, within the bounds of just 3JS now, can I answer this question?

They are general models. They’ve either read half the internet or the whole internet or not. They either don’t know what they is behind a closed door, but you also can’t say just the next five minutes.

I just want you to focus on this code base and only answer within that sphere. They don’t have that level of filtering just yet. I think they would hallucinate and get out of jail pretty quickly and start to answer things in a general sense far too fast.

Mirek
Yeah, that would be fantastic to have, but I think that’s fundamentally not how these models work. Yeah, yeah. Yeah, maybe with that, I’m really interested in what’s going to happen with this larger context window, because I think that kind of allows you to build more of conceptual knowledge, you know, during, around that area or task or subject matter.

AB 24:24
Great segue, because the fourth of the large models that was announced in the last three or four weeks was Microsoft with a co-pilot plus. It has an asterisk here as well, because co-pilot’s been around for a little bit.

[https://www.microsoft.com/en-au/microsoft-365/business/copilot-for-microsoft-365]

The two things they announced, or the two sides of the world that just climbed into the middle. I’m looking down. Yes, I’m on a PC. Sorry to all my Mac friends from around the world. Microsoft, I now have a co -pilot button in the bottom right corner of my screen, which is useful and okay.

It’s not fast like the others. It does have that big pregnant pause and the I’m thinking, I’m thinking, and you can scroll faster than the output comes. The announcement from Microsoft is that their next round of hardware slash software tools are going to be co -pilot plus PCs.

That is, having co -pilot more directly tied into the operating system experience. With the first bullet that everyone just opened their eyes at and had a bit of a half a fit about is that it’s able to almost always record your screen to have that context window of what you’re doing and what you’re working on.

If you were able to type or grab the microphone out and say, hey, I’m having problems with what I’m doing now, we already know what you’re doing with context, context window and context in the language sense, and be able to assist you with what you’re working on.

“Hey, I’m having problems with this document. Can you reach in and help me?” It already has knowledge based on not screen scraping, but direct connection to GPU and the drawer of the screen to know what that is.

That makes sense in that regard, rather than doing a bolt -on over the top of pointing a video camera at your screen. Makes sense to go right to the ones and zeros before they became pixels and have that knowledge.

But that co -pilot plus PC has been pretty, both welcomed and divisive. I think everyone with a privacy bent has just taken a double take or a triple take. It’s been quite a astounding revelation that that’s probably something which we’re going to be well used to in two years, but I think announcing it right now is definitely a few people pretty hard.

Thoughts on that? If an app was able to natively operating system assist you with your tasks, whatever that was without having to interpret pixels, but know before you even draw it on screen.

Mirek 26:55
Yeah, that definitely makes sense technically to integrate something like that as close to the metal as possible, right? But this I agree with those with the concerns that you’re raising because there’s yes it’s yeah, it’s a big big brother that you’re learning into your computer and giving it all the keys and uh Yes, I mean

AB
Yeah, it’s it’s not almost certainly with the way that these last language models are working. It’s not using your pixels to train on It’s been trained. It’s just using it to interpret that request at that second but at some point in time, it’s going to Assimilate your culture into our own as the book would say

Mirek
there will be concerns, at the same time Microsoft is really going all in on AI, on all fronts, and I can’t even mention all the names of the companies they invested in, but it’s not just OpenAI, they’re even investing into OpenAI’s competition and just covering as much of the market as possible.

And that tight integration with user products is, I believe, where the value is. Yeah. So, but yeah, it’s kind of concerning and interesting to watch and kind of like you want the computers to be smarter and faster and do everything for you, but at the same time.

You don’t want that.

AB
Look your segues are working really well – I don’t know whether you knew we’re going to be hitting here but of the four models talked about again: Facebook’s llama 3, Google’s Gemini, OpenAI and Microsoft – but the gorilla in the room (the elephant in the corner?) is Apple.

You know the pundits say they are lagging behind but their WWDC is happening in the next couple of weeks and I would probably bet the farm that they could have well let me put it this way Apple is the only company who has both the hardware and software stack completely under their control every Apple phone and Apple device in the last well the M series has just as many TPUs basically AI neural chips as they do graphics or CPU chips there’s a lot that happens when you take a photo on your iPhone that is more AI than anything if it’s a picture of the sky it’ll probably make it a bit clearer and a bit bluer if it’s someone’s face they’ll probably try and figure out that let’s not make them look like they were you know pale and about to keel over there’s a lot of computing resources tied directly to an operating system in that whole platform and they have announced well chat GPT also at the same time that they announced for Omni also put out a first native asterisk ish ish app for Marcus for Mac OS kind of neat kind of weird that that was the case you can access it quite perfectly fine on browser or device relatively natively to have it as a first party application is a sign yeah I’d say I’d bet the farm that in the next month that the worldwide Apple conference that will probably have some tighter AI integration everywhere let’s face it Siri has devices didn’t go on excellent

Mirek 30:22
Siri has been able to set my timer perfectly almost perfectly Siri for years now. Yep We’re we’re thrilled to see what’s next

AB
There it goes, one, two, three devices just woke up. It’s okay, Siri, you can stay.

Mirek
pretty much all I was ever able to use Siri for. So if they’re just make this work as they advertised years ago, that would be fantastic.

AB
Well that actually is probably one of the fundamental human interface sort of hassles of the last when a Siri had been out. I’d almost say a seven year Siri has been around. The problem with an interfaceless interface is that for all of us mere mortals who are used to screens full of icons, if we don’t recall the keyboard combination or we can’t recall what the icon looks like, hunting the menus is a thing and within a half a second, you found what you’re looking for.

It’s not too hard. It’s there somewhere is normally the answer when you have a rich interface. When you have an interfaceless interface, it’s very binary. Either it does a brilliant job and you love it or it sucks.

The early days of Siri were leading more to the second than the first of if you would ask it, how do I get from A to B? Or if you ask it a question that literally can’t answer, I think in the beginning, it couldn’t even go out to the web and find a half -assed answer for you.

It would simply, I can’t answer that. You could do things like control your timer, set alarms, integration with maps, but a few core phone apps, nothing more. In the last few years though, and this is the peril of an interfaceless interface and a lesson now for, chat to you for Omni with this chat interface, that balance between an AI hallucinating and telling me why the sky’s purple in convincing 12 ,000 page essay.

Great, but you’re wrong. Versus Siri and other agents being helpful and being able to solve more problems than solve just the right number of problems and then quietly give up when really they shouldn’t answer a common query.

Have you used Siri for more stuff in recent times or more the point?

Mirek 32:38
would love to, but it simply didn’t work ever. And I don’t think it’s just my funky accent. It’s just like that for everybody. So timer or wake up alarm that I barely use and that’s it like, you know, you can’t ask it what song is playing when you’re driving because it just pops out Google search.

AB
My number one use of Siri I must say is around the family dinner table when someone in my family says, oh, yeah, I must remember to do that and invariably forget maybe, you know, we sort of go grab your phone, hey, doofus, remind me tomorrow to do the thing.

And more often than not, that’s perfect. That is an example of the human being kind of trained to use keywords, get the order vaguely right and, you know, do commands.

Mirek
at the same time the human is trained to not trust Siri because it just doesn’t do anything else well. So I can’t rely on the fact that it’s gonna put anything in my calendar if I just tell it and then check on it.

And if I go and check if it’s really there, like I could have done it. You could have done it yourself, yeah, yeah. It’s really like you figure out what kind of tasks you can use it for and then you use it just for that because you know that nothing else works.

And it’s really hard to convince you that oh, we updated it and now it’s fine. It can do like all these other things. That’s it.

AB
There is no manual for it and if they do five years later release a slew of features or a slew of API You know to check the baseball scores in your in your region Unless you are made aware of that if you try twice to ask that question and you get stupid answers You will not try a third time and even if they sell you later on that that feature has been put in Chance of you using it’s low So that that is the classic example of building up trust over a period of years with an interface lists no button No icon interface that you either trust or you use for small things My question to you is comparing that to open AI’s chat interface with audio with yes Not Scarlett Johansson, but Sky and the five other four other voices now that one’s being removed What would you prefer?

Would you prefer something that is conversational and Ironically might make stuff up over time or would you prefer something that’s a bit more mechanical and concrete? But you have to learn how to trust it.

How do you certain features? Where’s your level of preference there?

Mirek 35:09
I think it really depends on what you’re doing, right? First, it’s a personal preference, but then sometimes you want the hallucination, sometimes you want some inspiration for something creative. And I think that’s what this works really well for, although, you know, it’s not the most creative, it’s kind of like mediocre creative.

Oh, gee, wow, you’re in trouble. But it’s there. And so the one task is to do that, the other is, you know, yes, no answer to something, and I want to read five polygraphs of fluff to, yes, no, it would be fine.

But it really depends on the situation, I guess.

AB
I’ll put forward two scenarios that our family has used ChatGPT for in the last couple of weeks that have been good use cases that we wouldn’t have thought. My daughter’s doing university, yeah, okay, astrophysics, so not exactly rocket science, but the next thing over.

She was, now I’ll say this carefully, she was having to do an essay on a really deep gnarly topic, all good. She couldn’t get started because the one phrase that she had in her mind of the topic that she was trying to do was the wrong phrase.

It was the one that she had in her mind of what she was trying to focus on. And guess what? There were no references in scientific papers for it. It was just the wrong word. And Google could not help when she searched for that phrase.

She found millions of answers, but none with that context. Well, guess what? ChatGPT was able to assist by basically being the world’s biggest thesaurus for synonyms, basically putting in that key phrase saying, if I’ve got this term, what else should I be looking for?

And bang, it came out with a long word list bullet points of, if you meant this, here’s other topics that are highly related to it. And suddenly that opened the door to her too. I had my facts wrong, my keywords wrong, now the doors are open.

But I could find everything I need to research my essay. That was really good because all of the online tools were just being quite mechanical in that search phrase equals this, but no creativity and little context apart from, if you meant this, did you mean this?

AB 37:23
So that was a win.

The other one was showing my wife, yes, ChatGPT-4o and having a conversation about geology. Now, we know with ChatGPT three and four plain, you could have GPTs, basically other GPT instances that people had made where they’d done the pre-prompt engineering to say, you’re a geologist, answer this in either a lay person’s terms or a professor of a certain educational level to make it work.

But they’re basically pre-faced. Most questions need to be answered with this context in mind. Those were okay, not exactly awesome. They were fair, but they didn’t really have a free range of options.

They basically said, here’s five texts or five corpuses, use that as your foundation. Four zero was able to have a conversation with my wife about not only the ground underneath which she was standing, but, and here’s the spatial part of it.

If I’m traveling from A to B, what shall I be looking for while I’m on this route? And it did a pretty bang up job to actually say, well, as you transfer from A through B, make sure you’re watching out for this kind of feature in this first half.

And if you look across here, you’ll find it didn’t say left or right. It didn’t know which way she was facing. But it had that context, both of topic and arcane topic, like a deep topic, but it also had that in terms of relevance of place.

And that was quite phenomenal. There, we were tourists at that point in time. We didn’t have our encyclopedia or our geologist’s handbook to Estonia and Finland. There you go. So they were dropping for you where it was last week.

AB 39:18
So we couldn’t in fact check it. But tell you what, we turned over every rock that we saw and my wife was able to validate most of what she was talking through. So pretty phenomenal. Obviously, this is a tool that as we’re at the internet, we all kind of know that.

So the body of knowledge is huge. But able to have a deep, dark conversation on a pretty specific topic with spatially aware, I’m here, I’m going to there, tell me everything I need to know as I travel.

Yeah, I must say, we were reasonably flawed. And then to have it icing on the cake. Here we go. To have that in the Scarlett Johansson voice. That was absolutely, it was a delight to see and to hear, but it was definitely a little bit freaky.

The controversy there is only escalated in the last few days. Apparently, which is not rocket science now. Apparently, OpenAI did not use her voice. They did hire an unnamed actress who was still remaining unnamed.

That’s all fine. But without fail, eerily similar. And I think everybody in the world kind of thinks that voice number one sounds a lot like a person.

Mirek
Well, it doesn’t matter if it’s not. It’s meant to be replicating that person makes sense. had it for some weird times it’s only gonna get funnier yeah yeah and weirder

AB
It is, it is. I know the the Siri lady I believe is from Australia and has been having an awesome time in the last many years extending her vocal repertoire to in -person events and the rest of it. I believe she spent a couple of many, many sessions talking all sorts of things.

It wasn’t with Apple at that point, it was with a third party who she was hired to be the voice of. Obviously it went further. I dare say she’s been paid quite well but that voice -off is certainly pretty infamous now.

I must say what voice do you use on your devices or your Apple devices? Do you use the default Siri? By that I’m meaning female voice or do you use

Mirek
I think I do. I think I use the the British theory. Yeah, I know it.

AB
I didn’t realise it was such a regional thing, but it does have a psychology to it of do you want your assistant to be something that you’re used to, something that you’re not used to, it almost defines whether, it’s the same with interface -less, it’s almost hard to figure out what other voices there are.