XR for Business

By Alan Smithson from MetaVRseBusiness Technology Arts

XR for Business episodes

October 02, 2019 Visualizing the Future of AR, with Visualix’s CEOs Michael Bucko & Darius Pajouh
All the world’s a
stage, but in AR, that’s a stage we’re still building. Visualix is
hard at work building that stage with their street mapping
technology, which will one day help make everything from digital maps
to Pokémon Go a whole lot better. Co-CEOs Michael Bucko and Darius
Pajouh drop in to discuss their technology with Alan.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today’s episode is
with two amazing people from a company called Visualix. Michael Bucko
and Darius Pajouh are really, really passionate about analytics and
teleinformatics. Michael is a CTO and co-CEO of Visualix and has a
computer science/teleinformatics background, he worked as a data and
software engineer and founded many companies before. At Visualix he
does packaging, partnerships, and technology, as well as make sure
that Visualix has the best tech team in the world. Darius, on the
other hand, is the co-founder and also co-CEO. He studied physics
with a specialty in non-linear optics. He did not stay in research
for long, because he founded a startup and then raised $200,000 and
it fails — we’ll get into that — but he worked at a company called
Innogy, the largest energy utility company in Europe. And the venture
developer program that allowed employees to start companies, funding
from the mother company. And so that’s how in 2017, Visualix was
born. To learn more about Visualix, you can visit visualix.com.
Welcome to the show, Michael and
Darius.
Darius: Thanks. Thanks for
having us.
Michael: Thank you very much.
Welcome.
Alan: We’ve been talking for so
long and now we finally get to have a conversation on the record.
Michael: Amazing. It’s been
awhile.
Alan: It’s been a minute. It’s
funny, because one of my interviews today was with Dr.
Walter Greenleaf. He’s been working in VR for 33 years.
Darius & Michael: Wow.
Alan: So when you think you’ve
been pushing hard for a long time, think about Dr. Greenleaf. So,
Michael, tell us what is Visualix and how does it work? Why somebody
would want to use it?
Michael: Ok, so Visualix is a
mapping and positioning platform. We allow the largest scale, most
reliable augmented reality in the world. It’s very simple. You take a
mobile phone and you map a space, for instance, your apartment or a
warehouse. And then in this map, in this digital twin that you’ve
created, you can place augmented reality content. And then people —
viewers — can see this content in real time extremely accurately.
And it works at scale. So it’s very, very reliable, works at scale.
And we have an SDK for that.
Darius: And if I may add
something, so the USP that we created is that we do all the
computation on the backend. So we use the mobile only as a sensor to
really get the data in terms of data we get on the phone. But the
real computation, the heavy lifting for mapping, as well as
localization, is in the backend side. So this allows us to basically
escape the limitations of mobile devices such as phones or let’s say,
AR glasses that only allow shared experiences on an area of about 20
square meters or 50 square meters. So basically a small room. Where
we extend this from 20 square meter to 20,000 square meters. So about
a thousand fold. And this is something that you only can do if you
have a powerful backend. And this basically makes us the only company
that can create one spatial map that is together. You can basically
create one spatial map onto, let’s say, a whole factory floor where
you can localize with centimeter position.
Alan: That’s amazing. So some of
the use cases that I can think of just off the top of my head would
be directions for heads-up displays for people driving a forklift and
they need to get around to pick up precise things around the
warehouse. That would be one thing. What are some of the use cases
for this technology?
Michael: Okay, so the use case
that you mentioned is one of the very good use cases you can imagine.
There could be a person on the forklift, but you can also think that
that can be — the forklift itself — equipped with ARCore or ARKit
ready device. You can also think of telecoms and OEMs, they want to
have a large scale augmented reality infotainment, for instance, in
malls. And you can think about how automotive companies, so the
automotive sector, they might want to have in-car gaming experiences,
or outside of the car gaming experiences. They might be willing to
optimize their warehouses. They might be willing to map the world and
then augment the world the way they want.
Darius: Yes. We focus in the
kind of three areas of production manufacturing where basically, for
example, manufacturing companies and also automotive companies want
to create use cases around autonomous maintenance, basically having
all the digital twins that are currently running in silos, for
example — the digital twin of a specific machine, the digital twin
of the whole factory layout. They combine it basically in our spatial
map. So they overlay these digital twins with our point cloud. We
create and they’re basically– we create a standard of visualization
of all the digital twins that can be accessed at any point. So this
allows a worker to just hold up the phone and see basically the right
AR content on every machine, on every pathway, seeing the warning
signs that are particularly tailored to the specific status of this
worker, for example. So this kind of factory production setting is
very strong with us. The second one is this logistics type of thing,
where we scan warehouses only for the sake of mapping and
localization. And with this, we hope to either substitute a large
part of the beacon deployments, or of course enrich beacon
deployments by creating continuous digital twins that can be used for
documentation as well as other things.
Michael: So you can think of so
many use cases. I mean, we launched the SDK, and we have people
trying to do visual prototyping. We have people trying to do
construction. We have people who try to do maintenance. We have a
very good friend of ours who wants to be the large scale game. We
have OEMs who want to integrate our software into their hardware. We
have, as I mentioned, telecoms that might want to have our software
in the edge. Plenty and plenty of opportunities. And sometimes it’s
very, very difficult to imagine those use cases like, for instance,
in the construction space, somebody who’s going to assemble buildings
and you need to find innovative ways of assembling buildings, or
somebody who wants to do maintenance in, say, parking lots. Like
very, very unexpected use cases sometimes.
Alan: That’s incredible. I think
people are only starting to scratch the surface with the use cases,
which is why I asked that question first. And things like Microsoft
introducing– or announcing the world of Minecraft and augmented
reality, meaning anywhere in the world, you can start building
Minecraft blocks and it will stay there, positionally tracked, so the
next person that comes along and– basically what you’re doing is
that level, but made for industry and enterprise.
Michael: Yes. Yes, exactly. You
mentioned Minecraft, that is very, very thoughtful of you. So
Microsoft, as you know, we support ARKit- and ARCore-enabled devices.
But you can think Visualix could support, we have a number of
requests from AR glasses company, we could also support AR glasses.
Microsoft launched their Azure Spatial Anchors, we could support
Azure Spatial Anchors as well, and take their augmented reality to
the next level.
Darius: Yeah. So the cool thing
about our technology is that we are platform-independent,
essentially. So what we do is we take this local tracking capability
of, let’s say, ARKit, ARCore, Microsoft Spatial Anchors, or any type
of equivalent that exists in there. And basically with that, create
our own point cloud in the background, that enables the localization
and mapping. So this allows us that all types of devices can visually
localize themselves, and therefore, also display the same content in
the same map.
Michael: And now imagine, think
about Minecraft. What we can do with Minecraft is that we can create
the most realistic AR game ever created. So we can take it, we can
place this Minecraft content in the building or in a warehouse, and
you could actually live in the Minecraft world.
Alan: That’s just ridiculous.
I’m really excited about that. I want that in my life.
Michael: I want that as well.
Alan: I think if you took this
technology that you guys have built for creating this mapping and you
combine it with something like 6D.ai, which is doing real time point
cloud mapping, I think you could combine the two and really make
robust augmented reality, that is world sensing. If you wanted to
have, for example, a scavenger hunt and the scavenger hunt would then
take you in different places and you’d have to go find things, but
they could be aware of your surroundings and they could hide behind
pillars or people or– one of the things that’s coming out this
summer is the new Harry Potter game in AR. And I think they’ve got a
lot of things that they’re working on, to make it know that the world
around them is there. And so being able to use real augmented reality
in the world. I think we’re only scratching the surface of what’s
possible on this. Let’s talk about some other use cases, because I
think at the end of the day, you make a technology platform in which
people can create literally anything they want and by having this
very detailed mapping. What are some of the use cases? One that comes
to mind would be retail. If you wanted to have very robust augmented
reality experiences within your store without having to bring it
under beacons in your store, you could have centimeter accuracy
within a store and give customers the ability to go round and find
Easter eggs throughout the store. Is that something that people are
working on?
Darius: Yeah. We partnered with
Deutsche Telekom to basically enable to start. So the developers are
building applications that are tailored for this market. And we are–
we can’t share the news right now, but it goes exactly in this
direction. An exciting thing about us is that our technology is very
robust to changing environments. And that’s why retail specifically
is a very important thing for us, because in retail, such as a normal
shop, you can imagine that if you move products around, if you have
changing light conditions, solutions that do mapping and localization
that, say, only with ARKit or ARCore are bound to the limitations of
ARKit and ARCore. And this is very heavily correlated with robustness
and changing environments. If you change, let’s say, a small thing,
these technologies usually don’t work in localizing, but our
technology can. Yeah, it’s extremely robust to changing environments.
You can change up to 40 percent, around 50 percent of the scene and
still localize with high accuracy, still centimeter precision. And
then also we fill the point cloud and always keep it updated. And we
actually made this happen now.
Michael: By the way, we’re
actually discussing with one of the largest retailers in the world,
we’re actually discussing the rollout. So this actually happened.
Alan: One of my predictions from
a few years ago was that Apple and Google will probably create some
sort of game or experience that you’ll find yourself alone in your
house, chasing Pokémon or something around your house and you’ll be
scanning up and down and left and right. And really, what they’re
doing is creating a persistent cloud map of the inside of your space,
because if you think about it, Google has a map of the entire outside
world of this planet, barring a few other local places. But they’ve
got a beautiful map of the whole world that you can go in VR and
travel around in it, and Google Earth VR. But they have no real data
about the inside world. And that’s going to be very important as we
move to a world where we have a digital twin of every space.
Michael: Well, that’s one point.
They definitely have a very amazing map and they have done a lot in
this space. We actually know some of the guys that you mentioned
working exactly. He used to work on Google Earth. So he’s a good
friend of ours. And then my point was they definitely have a lot of
amazing technology and they have maps. But now the question is, will
they be able to– using their maps, would they be able to enable this
kind of accuracy, this kind of robustness? That’s an open question.
Also, another question is, we focused mostly on the indoors, exactly
what you said. And indoors, we can optimize our algorithms for
indoors as well.
Darius: And especially, what I
think it’s important to mention is that from a business strategy, we
went this direction of going indoors, especially in an industrial
warehouse, production facilities, retail. And this is where, let’s
say, a Google or an Apple can drive around with their cars, which
they have the power to do so very quickly. So we thought that the
space that I just mentioned before is a very good space to enter,
lucrative, we can really dominate the space without, let’s say, big
competitors quickly going in there.
Michael: Nonetheless, we’ve been
asked by a couple of top American companies, Fortune 500s, whether
that B-to-C direction would also be interesting to us. I mean, we’ve
focused on the industry. Nonetheless, we have a number of requests
from companies in the home appliances space, from OEMs who want to
have our software integrated, from companies that want to build
games. So our technology could potentially be used to– like for
instance, imagine that you take something that Google has, or one of
the other companies that use satellite imagery to create outdoor
augmented reality experiences. Now, if you have a three meter
accuracy, it’s not always going to be– or five meter accuracy is not
always going to be good enough for all of your games. So what you
could do is you could use it together with Visualix to create better,
higher quality augmented reality experiences in at least some places,
some parts of your game.
Alan: Interesting. All the
processing is running on the cloud. So this is probably lends itself
nicely to your Deutsche Telekom client. What are the implications of
5G with this technology?
Michael: So 5G is gonna bring a
completely new world, a whole new world. So there is gonna be– the
latency problems are not gonna be this dramatic anymore. We will be
able to introduce multiplayer. Say, imagine we for instance can–
Berlin, we’re based out of Berlin, we scan Berlin and we create a
game, and there will be Brandenburg Gate and a couple of other
places, say, one hundred places in Berlin, they will be spatially
mapped. So we’ve got to spend some additional time — like 2, 5
minutes each — and were going to to map all of them. Then what’s
gonna happen is you’re gonna have a very, very high quality augmented
reality, especially in those places. Now, if you want to have a
multiplayer and you want to be able to re-localize very often, then
this is where you see 5G, because this is gonna get heavier and 5G is
going to be very, very important for that.
Darius: Also very important
point is that, so as Michael mentioned, the throughput is one
important fact. The data throughput, as well as latency. The
throughput is going to be very important for the resolution of images
that we stream to the server in order to much more accurately and
robustly localize. This relates to, for example, the distance of
localization. This is something that current OEMs have a big problem
with, because you can only localize if you are, let’s say, two, three
meters, four meters from a wall. But with our technology, for
example, where we already stream high resolution images, we can
localize at distances up to 20 to 30 meters. This is a bigger SP that
on a street level is very important to localize with centimetre
precision rather than, let’s say, four metre precision.
Michael: Also, we optimize for
robustness. So imagine you have 50 players, very close to each other.
We can still deal with all of them. So it really works. And this is
very, very helpful. And 5G in this case would be necessary.
Alan: Yeah, absolutely. So if a
company is using this, how is it built in? You’ve got an SDK they
can–? Are they building it into their apps, or can they use web
based AR? How does it technically work?
Michael: Currently, there are
two major directions. So the first direction is that we work with
multiplayers who work directly with end customers, with large
companies and they do products together with those large customers.
The second use case is we approach large customers directly. They
have their own developers and they are building their products,
products that they want to build. In all cases, it can be either
something integrated into their existing apps, but in the very
beginning it’s most often a new app. If you want to start with the
Visualix SDK, it’s essentially five minutes, you download it, compile
the demo app, and it works immediately. You can map your space and
you can create a game. So if you are our customer, you will just take
the SDK. You will have an application in one hour or something. Very,
very simple. And then if we want to integrate it, you’re going to
integrate it. Some people want to integrate it into their existing
apps, like, for instance, royalty apps or user– like, apps for the
malls. Some companies just want to create some new experiences in the
very beginning.
Alan: It’s interesting, malls
was something that I was thinking about as well, for navigation
around malls, but also creating some rich experiences for customers.
And I think a lot of these companies, it’s hard to wrap our head
around the fact that we’re using smartphones now, but in five to 10
years — and I don’t know when — but we’ll wear glasses, and those
glasses will be our world of computing. And so what you guys have
done with the visual mapping and positioning is going to be vital to
the success of that, because right now the best thing we have is
Bluetooth beacons, and stuff like that.
Michael: Well, this is the
direction that we very much love. So we very much believe in this
direction. We very much believe in this world where everybody is
going to be wearing AR glasses. And what this means to us. Well, what
this means to the world is that the Visualix can enable pretty much
every AR glasses. We’ve had over 20 requests from different AR
glasses companies. So a number of them. And some of them just wanted
special modules, that they could use for AR eyeglasses. Some of them
want some sort of integration. Some of them just want to do some sort
of partnership. Some of them just want to learn. But that’s
definitely well, our patented technology is crucial to developing
glasses, because we work on top of ARKit, ARCore, potentially Azure
Spatial Anchors. We could support Hololens. We could support– well,
today I actually learned a couple of amazing things, and seems like
we actually pretty much off-the-shelf would be supporting a number of
them. But let us tell you later, because this is not yet public
information, but there’s gonna be a lot of amazing developments in
exactly this space.
Alan: Pretty exciting.
Darius: Also, another
interesting fact is that our solution can be already used, especially
in the industrial sector. Many, for example, camera-based systems
don’t have ARCore/ARKit included. So many companies are asking us if
we can make it a go-and-work with six-degree-of-freedom sensors that
they already have off the back, or include with the camera system.
And these things are about to come, that we will engage in further
tests with this, where these companies are interested to basically do
visual mapping and positioning, let’s say, with forklifts. So
forklifts are a big topic with our company and they are fine with
testing ARCore and ARKit enabled devices initially. But later they
want to go into the sector where they only have those
six-degree-of-freedom sensor plus camera systems. And this will, of
course, drive the cost even further down, down to, let’s say, $20 or
$30 a pop. And then if you compare this with a potential beacon
rollout that you need to have, with a thousand beacons this costs up
to 200, 300,000 dollars, plus wiring, plus maintenance. And if you
could substitute that amount of investment with only, let’s say, 20
dollars times — I don’t know — 20 forklifts, that’s like 500 bucks.
And then it works off the shelf. So that’s a very exciting future we
see, especially when it comes to warehousing.
Alan: Wow. It’s really
incredible. There’s so much to unpack here. I don’t think people
really understand, being able to visually map a space and apply
graphics and computer graphics to it. It’s actually not even
augmented reality anymore at that point. It’s really mixed reality,
because it’s world sensing and I think you guys are really onto
something with this. While you were just talking, I downloaded Harry
Potter’s Wizards Unite. It’s finally out.
Michael: Yeah. That said,
definitely very exciting. I mean, we know a couple of people at
Niantic. And they’ve been definitely working very hard on this. So
I’m very much looking forward to checking it out.
Alan: Yeah, it’s very exciting.
I can’t wait to go home, I’m going to go run around the neighborhood
and play Wizards Unite. [laughs] It’s like Pokémon Go meets Harry
Potter.
Darius: We have to try it out,
Michael. We have to try it out tomorrow.
Alan: Yeah, you do. Are there
any of the use cases that are using your software, are any of them
out in the wild right now, that people could try?
Michael: Well, definitely. So we
have– well, not exactly people. So the way it currently works, we’re
mostly industry. So what happens is- for instance, we have an
application, we do maintenance in the factory. And then people can
test this maintenance application in the factory. We also have some
forklifts and some devices running from other warehouses. And then
view in real time, so some other people, some warehouse workers can
access this information about the trajectories of the forklifts. And
then we have some deployments in the edge where, for instance, in the
US or in Asia there are developers working on building experiences.
Those deployments are in production and they are ready to use and
they are being tested.
Darius: One very important point
that we have to mention is that most of our clients for now, in the
beginning — this will perhaps change later — but especially in
Germany, most of the industrial clients are very privacy focused. So
we have a solution ready to go on premise, that just works
immediately. We can either install our service directly on their
server, we can also basically send them our package with a physical
server, that they just integrate quickly into their network, and they
can immediately develop apps within their network. So this really
alleviates a lot of effort. The installation is super simple. It
takes a half an hour to hook it up into your wi-fi and developers can
right away build applications that live in the AR space of their
factories. And I think this might be a very exciting direction for
the future, because if you have this physical server on premise that
does all the mapping and localization, then you can use, of course,
the images that you gather from the mapping process and the
localization process to get more analytics out of it. Semantic
segmentation on top of that. So we believe we really can be this hub
onto which other applications and other companies can just hook into
to create even more value beyond the mapping and localization alone.
Alan: Wow, you guys realize
you’re just gonna get bought by Google or something.
Darius: Bought?
Michael: [laughs] It’s always
very difficult to say. So, you are saying we’re going to buy
Google? Yes?
Alan: [laughs]
Michael: Not the question, are
we going to buy Google or…?
The question is definitely not whether
you’re going to buy Google, whether they buy you.
Darius: They have many smart
people, we know that. Probably have enough already. They don’t need
more.
Michael: Yeah. You always need
better and better and better. You can always get better.
Alan: Well, you guys seem to
have an amazing team. How many people are you guys now?
Michael: We’re currently 12 full
time people, around 15 overall. And you’re right, we have a very
amazing team. We would love to thank them from here now, because they
have helped us so far achieve everything we’ve achieved so far. We
have won a number of awards together. We won the Deep Tech Award in
Germany. We have demoed our application to a number of directors,
senior directors, everybody’s laughing always because we demo those
to a number of important people in this world. They really see that
we’ve made tremendous progress. And this is thanks to our team. And
we’ve been working very hard together. We’ve had very many moments
most of the time, currently beautiful moments. But also there are
challenges. We still are launching our Unity SDK, which is, again, a
challenge. And there will be challenges in the future, and maybe
cloud deployment, and a rollout. So a number of huge challenges in
front of us. But we can make it happen together. I mean, we work with
the ex-head of robotics at Google. We work with the ex-senior
director of AR enterprise at Adobe. And we have a number of amazing
friends who gave us advice. We work with around a number of people
from Metaio — ex-Metaio people. You know, Metaio got acquired by
Apple some time ago. So we really do focus on the team and we very
much believe that the value we create is thanks to the team and
thanks to our focus on the product, on the customer. And the biggest
focus on the team.
Alan: You’re on the right path
for sure. So is there anything else you want to leave people with?
Maybe some– a couple more ways they can use this, because I think
the best thing that we can do for the listeners is really give them
real life world use cases of this technology so that they can start
thinking, how else can I use 3D mapping and scanning in my space? I
would think mining would be another one, being able to have really
accurate maps of mines using people’s devices that they have.
Darius: So, for example, in
construction side, I believe Michael mentioned already, amusement
parks are a huge thing. They have a– I don’t know how many square
kilometres of space. And here it is very important that you have this
large scale mapping and localization enabled, that anybody who
basically picks up their phone, any one of these amusement parks,
they can just immediately localize with high precision. So everywhere
where we have a space from, let’s say, I don’t know, a thousand
square meter to infinity and you manage either things or people, then
our technology is kind of a must.
Michael: If your company is
going into the IoT space and if you are using, for instance, if your
insurance company. You might be interested to be in IoT space in the
future because you want to be more informed. And there are many, many
companies like these that you sometimes don’t think that it could be
relevant to them, but it is. If you want to be in the IoT space, if
you want to improve your IoT capabilities, then Visualix is perfect
for that.
Darius: Another very important
point is the ability to just do planning, factory planning, for
example. Many of our customers just use the technology to scan their
entire space — or the entire space of their customers — in order to
then better understand “How can I move this big machine into the
right place? Is there maybe an air duct that has to be removed? Are
the electricity outlets positioned at the right place?” So there
are many use cases like this where, for example, a large German
automotive company, they want to — with the Hololens — be able to
fuse together the future 3D model of the entire factory, and drop it
into the empty factory. And they are the challenge that it’s very
hard to localize and map an empty factory. But we actually managed to
do it because our technology is very sensitive. So they were very
astonished that we could actually fuse together this digital twin of
the factory with the empty factory floor. And anybody could then
localize and see exactly where the machines will be placed, and could
then better understand and say, “Wow, we really have to change
these electricity outlets. We have to maybe broaden the space so two
people can fit in at the same time.” These stuff are very
costly, if you don’t think about this early enough. And like this, it
can really prevent these mistakes from happening.
Michael: And there a bunch of
other use cases like for instance: currently if you want to map
spaces and create digital twins, you very often use large robots. In
our case, you don’t need a robot. In our case, you can just use a
mobile phone, so you can reuse your existing resources and very
cheaply create a digital twin of this space. We’re not a
visualization company, so it’s not going to be beautiful, maybe. But
what’s gonna happen is you’re going to have a very affordable digital
twin and it’s gonna be actionable, meaning that it’s gonna be an
actual IOP setup where you’re going to know the positions and
orientations of all the devices. For instance, you can use it for
documenting things. You can use for remote work. You can also use
those digital twins for virtual prototyping, like you want a new
forklift and you don’t know if this forklift is gonna fit in, and you
don’t want to destroy something. Also, it might be a very heavy
device, you don’t want to carry it. And you could try it in the
digital twin mode. So all of those use cases are possible. And the
SDK, that’s the crucial part, it’s very easy to use. It’s
frictionless. So you just take it in your hand, you give it to your
software engineers. They can immediately start working on these in a
5, 10 minutes. They can actually have the demo app ready in two
hours. They can have a simple app ready. This is how it works.
Alan: Wow. It’s really
incredible. You guys are providing a service that I don’t think
there’s anybody else doing this.
Michael: We don’t know about
anybody doing something like this. We don’t know about anybody. We’ve
been doing some research, and to the best of our knowledge, we are
pretty unique. And, you know, the technology’s patent pending. So
everything that we’re talking about, or a number of things we’re
talking about are patented.
Darius: There are some companies
we know of — I think like one or two — who do outdoor mapping with,
of course, help of LiDAR and other methods. And then do localization
in a perhaps similar way. But there’s no company that gives the owner
the power of mapping and localization at the same time with any
device. So there we are, very unique.
Michael: The very amazing thing
about our technology is that we reuse what’s the best in ARCore and
ARKit, potentially Azure Spatial Anchors. This means we could support
pretty much every device of the future. So imagine there’s gonna be
AR glasses. The AR glasses are probably going to use Visualix
technology. That’s how it’s going to look like.
Alan: Wow, that’s pretty
impressive. So if I had to put some money on this. My guess is Apple
or Google should acquire you. And this is my prediction. So I think
Apple or Google should acquire you now, because if they don’t, then
all of their competitors have the advantage that they will come from
your service.
Michael: You know, we are
starting to make money. So what’s happening to Visualix? We have
achieved a reasonable amount of respect in our space and work with
amazing companies, amazing customers in Germany, in the US and in
Asia. And we are starting to make money, meaning that SDK sales is
significantly easier than sales of technology because it’s an actual
packaged product that you can use pretty much off the shelf. So we
hope to start making more money, and take over more and more of the
industrial spaces in the world and create even more value for our
customers.
Darius: In terms of your comment
that you mentioned before, regarding acquisition: of course OEMs are
one thing, but if I was a, say, a large either industrial client, for
example, who wants a monopoly on this technology has a huge
competitive advantage in contrast to other industrial players.
That’s, of course, interesting as well. There are, of course, a
number of OEMs and software platforms, that are also active in the AR
industrial space. And that’s, of course, also potentially interesting
that they can defend themselves or have the unique value proposition
of our technology, to basically not let this go into other platforms.
So there we believe there are many avenues where we can go to,
partnerships, acquisition. But I think our goal is to really grow the
company and really create a lot of value. I think that’s something we
very much agree on, to continue this journey.
Alan: Finally, a startup that
wants to actually make money.
Michael: [laughs]
Alan: Yay! Yay!
Michael: We definitely make
money. We definitely want to create a lot of value. We work very hard
on this. We work pretty much all the time. And we are very motivated.
And we speak to a number of tier o1 investors in the US and some of
them really express– some of them approach us, and they say, “OK,
guys, you’ve made a tremendous amount of progress. So this is really
happening.” And, you know, you’ve seen Charlie [Fink] writing
about us. And, you know, he’s one of the most respected figures in
this space. And we are huge fans of Charlie. And when he wrote about
us and then again, we gave him a quick demo and at AWE, Santa Clara.
Those are very, very beautiful moments. And then we see that we
actually bring this value, we push the world further. And now, since
Apple and Google are pushing ARKit and ARCore and Microsoft is
pushing Azure Spatial Anchors, this is an amazing use for Visualix.
This is the best thing that could be happening for Visualix, ever.
Alan: I love it. Well, guys,
it’s been a real pleasure to have you on the show. We’ve got to wrap
it up. But thank you again, Michael Bucko and Darius Pajouh from
Visualix. It’s been a really amazing, enlightening podcast. And just
kind of close off, Charlie Fink is actually one of our mentors on the
XR Ignite program. So really excited and honored to have him as part
of that as well.
38 min
October 02, 2019 Visualizing the Future of AR, with Visualix's CEOs Michael Bucko & Darius Pajouh
All the world's a
stage, but in AR, that's a stage we're still building. Visualix is
hard at work building that stage with their street mapping
technology, which will one day help make everything from digital maps
to Pokémon Go a whole lot better. Co-CEOs Michael Bucko and Darius
Pajouh drop in to discuss their technology with Alan.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today's episode is
with two amazing people from a company called Visualix. Michael Bucko
and Darius Pajouh are really, really passionate about analytics and
teleinformatics. Michael is a CTO and co-CEO of Visualix and has a
computer science/teleinformatics background, he worked as a data and
software engineer and founded many companies before. At Visualix he
does packaging, partnerships, and technology, as well as make sure
that Visualix has the best tech team in the world. Darius, on the
other hand, is the co-founder and also co-CEO. He studied physics
with a specialty in non-linear optics. He did not stay in research
for long, because he founded a startup and then raised $200,000 and
it fails -- we'll get into that -- but he worked at a company called
Innogy, the largest energy utility company in Europe. And the venture
developer program that allowed employees to start companies, funding
from the mother company. And so that's how in 2017, Visualix was
born. To learn more about Visualix, you can visit visualix.com.
Welcome to the show, Michael and
Darius.
Darius: Thanks. Thanks for
having us.
Michael: Thank you very much.
Welcome.
Alan: We've been talking for so
long and now we finally get to have a conversation on the record.
Michael: Amazing. It's been
awhile.
Alan: It's been a minute. It's
funny, because one of my interviews today was with Dr.
Walter Greenleaf. He's been working in VR for 33 years.
Darius & Michael: Wow.
Alan: So when you think you've
been pushing hard for a long time, think about Dr. Greenleaf. So,
Michael, tell us what is Visualix and how does it work? Why somebody
would want to use it?
Michael: Ok, so Visualix is a
mapping and positioning platform. We allow the largest scale, most
reliable augmented reality in the world. It's very simple. You take a
mobile phone and you map a space, for instance, your apartment or a
warehouse. And then in this map, in this digital twin that you've
created, you can place augmented reality content. And then people --
viewers -- can see this content in real time extremely accurately.
And it works at scale. So it's very, very reliable, works at scale.
And we have an SDK for that.
Darius: And if I may add
something, so the USP that we created is that we do all the
computation on the backend. So we use the mobile only as a sensor to
really get the data in terms of data we get on the phone. But the
real computation, the heavy lifting for mapping, as well as
localization, is in the backend side. So this allows us to basically
escape the limitations of mobile devices such as phones or let's say,
AR glasses that only allow shared experiences on an area of about 20
square meters or 50 square meters. So basically a small room. Where
we extend this from 20 square meter to 20,000 square meters. So about
a thousand fold. And this is something that you only can do if you
have a powerful backend. And this basically makes us the only company
that can create one spatial map that is together. You can basically
create one spatial map onto, let's say, a whole factory floor where
you can localize with centimeter position.
Alan
0 min
September 30, 2019 Three Decades of Medical VR, with Stanford’s Dr. Walter Greenleaf
XR has come a long way, baby – and we have one of the technology’s earliest pioneer’s on today’s episode. Dr. Walter Greenleaf has been working in the field for 33 years, since the days when VR was little more than a twinkle in research scientists’ eyes. Now, he and Alan chat about how far the technology has come, and how far it still has to go.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today’s guest is Dr.
Walter Greenleaf, a behavioral neuroscientist and medical technology
developer working at Stanford University. With over three decades of
research and development experience in the field of digital medicine
and medical virtual reality technology, Walter is considered the
leading authority in the field of working in this industry, and he’s
been doing this for 33 years. Unbelievable experience. Dr. Greenleaf
has designed and developed numerous clinical systems over the last 33
years, including products in the fields of surgical simulation, 3D
medical visualization, telerehabilitation, clinical informatics,
clinical decision support, point of care, clinical data collection,
ergonomic evaluation technology, automatic sleep staging systems,
psycho-physiological assessment and simulation assisted
rehabilitation technologies, as well as products for behavioral
medicine. Dr. Greenleaf’s focus has always been on computer supported
clinical products, with a specific focus on virtual reality and
digital health technologies to treat post-traumatic stress disorder,
anxiety disorders, traumatic brain injury, stroke, addictions, autism
and other difficult problems, and behavioral and physical medicine.
He’s currently a distinguished visiting scholar at Stanford
University’s Media X program at Stanford University’s Virtual Human
Interaction Lab and the Director of Technology Strategy at the
University of Colorado National Mental Health Institute Center. To
learn more about the work that Dr. Greenleaf and his team are doing,
you can visit the Human Interaction Lab at Stanford at
vhil.stanford.edu and a new
organization that he’s formed called the International VR Health
Association at ivrha.org.
Welcome to the show, Dr. Walter
Greenleaf. So great to have you.
Walter: Thanks, Alan. I’m
pleased to be here with you.
Alan: That’s an honor. You are
considered one of the godfathers of this technology. You’ve been
working in it your whole life. And I want to personally say thank you
for laying the groundwork that allows people like myself — the new
people getting involved here — to really pick up where you left off,
and where you’ve driven this whole industry forward to, and let us
really build upon your lifetimes of knowledge. So thank you very much
for paving the way for us.
Walter: Well, thank you, Alan.
And really, without everybody else’s and my colleagues work and your
work and other people who are helping the trends position from
something that for a long time was a research lab curiosity and
something that really hadn’t escaped the confines of academia. Now we
have it out there in the world. And I’m particularly excited about
all the progress has been made in applying VR and AR technology to
difficult problems in healthcare. For me, it’s a very exciting time.
Alan: I’ve been keeping track of
all of the different things that come up in my news feed and I have a
health and medical folder. And it’s interesting, because last year I
actually had to break it apart into a mental health folder, in
addition to the traditional health and medical. So there is an
enormous amount of, not just research, but real practical
applications being created for this. You know, one of them, one that
stands out the most to me is being able to use virtual reality to
treat lazy eye or strabismus. I thought that was just amazing. Within
a few sessions, people are seeing complete reduction and elimination
of their lazy eye using virtual reality. And that’s just one of a
thousand use cases in this technology. So maybe walk us through the
work you’re doing at the Virtual Human Interaction Lab, and what are
the some of the great use cases that you’re seeing now?
Walter: OK, well, sure. Jeremy
Bailenson is the director of the Stanford Virtual Human Interaction
Lab, and I help out there as the medical advisor and expert. The
Stanford Virtual Human Interaction Lab has a focus on exploring and
studying how VR technology can promote pro-social behavior, such as
helping us better understand how our behavior affects the world’s
ecology, how our behavior affects other people, and studying how we
can shift attitudes and shift behavior using virtual reality
technology. The lab is really one of the pioneering research groups
in the field of VR and behavior change. And I encourage you and your
listeners to check out the website. There’s a lot of really amazing
research material that’s been accumulated there. But my role is sort
of translating that research and the research of other groups from
the academic arena out into the medical product arena. So in addition
to my work at Stanford and at the University of Colorado National
Mental Health Innovation Center, I’m doing a lot of work advising
some of the early stage and some later stage medical VR startups,
those that have received investment funding to build and bring to
market products. Then also helping those groups connect with the
healthcare ecosystem, the pharmaceutical industry, the medical device
industry, the health services industry, and the insurance providers,
making sure that everyone in the healthcare ecosystem is aware of the
power of VR and AR technology and how it can make a big difference in
healthcare.
And as you mentioned earlier, it really
spans the spectrum. We see some really amazing products are being
developed to in the education arena to help train people to not just
do surgical procedures, but also to work as a team on difficult
problem, how to deliver distressing news to a family or to a patient,
how to interview a patient in an effective way. A large selection of
training applications. But beyond training, we also have a whole new
wave of systems — like you described, for strabismus — that help
therapeutically with clinical problems. And those range from– well,
you listed a few of them, treating stroke and traumatic brain injury,
treating anxiety and depression, post-traumatic stress disorder,
helping with addictions, helping with autism. There’s a very long
list of interventions. And one of the things I’m really excited about
is VR allows us also to do a better job of assessing people, to
measure how they move, measure their mood and their behavior in ways
that we really didn’t have tools for before. We can now do better
objective assessments, instead of subjective measurements. And that
gives us some very powerful tools.
Alan: One of the tools that I
think we’re just starting to see come online is eye tracking and
motion tracking, where we’re really able to get data points about
humans that we’ve never had before. Is that some of the things you’re
working on?
Walter: Absolutely. And what I
find very exciting is that because we can measure how people move,
because we can measure where they’re looking, what they’re paying
attention to, things that before we’re collected in a subjective and
analog way, we now are able to collect in a reproducible and
objective way, and that gives us both new tools for research, but
also new tools for assessment. Let me give you a few examples. If
we’re trying to understand, let’s say, a neurodegenerative disease
like Alzheimer’s or Lewy body disease or perhaps Parkinson’s. And we
are asking people subjectively — or their family members — to
subjectively give a report on how someone’s doing cognitively. It’s
very, very hard to measure. And that means that developing new
pharmaceutical interventions, new behavioral therapy interventions
are all limited by the fact that we don’t have very precise tools.
But if we can measure how people move, if we can measure what people
are looking at, if we can measure what people are attending to, if we
can measure behavior, then we can do a better job of coming up with
new interventions that — either pharmaceutical or through cognitive
behavioral therapy — that can help with whatever problem is that
they’re dealing with. And it’s really putting us in a better position
to move forward in terms of both research and product development.
Alan: It’s incredible. The work
you were doing 30 years ago, how has it progressed? How has it
changed, from where you were starting out in this technology, to
where it is today, and where you think it will go in the next five to
ten years?
Walter: Boy, what a big
question. In terms of the progress we’ve made, we knew back decades
ago that VR could be an effective tool to help with some very
difficult problems, for example, treating phobias or post-traumatic
stress. We can use a simulation to help do what’s called exposure
therapy. So, for example, someone who might have a fear of flying, we
can develop a simulator that allows him to go through the experience,
or fear of heights, or fear of spiders, or really anything that there
is a fear reaction to. The counselor or commission can gradually, in
a controlled manner, expose a patient to what they are afraid of, and
teach them the skills to manage those fears and to habituate
sometimes what’s a learned fear response. And we were able to do
that, for example, with– there was a– Larry Hodges and Barbara
Rothbaum developed a virtual Vietnam — back in the early 90s — that
was very effective at treating post-traumatic stress disorder for
Vietnam vets, some of who had been suffering from PTSD for decades.
And with the help of this method of exposure therapy, it was able to
make a big difference.
So to answer your question, we’ve known
for a long time that we can treat problems like addiction, problems
like post-traumatic stress, problems like phobias, a whole selection
of clinical problems using VR, but it really wasn’t affordable. And
it also wasn’t very comfortable to wear. Spending a lot of time in VR
would sometimes cause simulator sickness. And it was so expensive,
and a head-mounted display could cost $70,000. A computer that was
used in the research could cost four or five hundred thousand
dollars. But now we have better systems that I can order on Amazon.
It’s just amazing. What’s happened is some of the paths that were
plowed back in the early days showing what works, what doesn’t work,
what’s a fruitful path of endeavor, what’s not. We know that. And we
can go deeper now with the technology. We can do larger scale
studies. We can reproduce the original research that was done with
small sample sizes and with what appears now to be very crude
equipment. We can really go out and build it up better and probably
more importantly, get it to people who need it.
Alan: So we’ve seen a reduction
in the cost, reduction in the time to make these things. The one
thing that we’re not seeing — and maybe you can speak to this — is
a massive adoption across all the industries. And medical is really
adopting this technology more so than in the other one. What do you
think are still the underlying reasons for the hesitation? And it
might– I have a theory on this. And my theory is that we’ve been
crying wolf for so long. “VR is going to be great!” Another
five years, “VR is going to be great!” Another five years.
People are just like “Yeah, yeah, whatever. VR, sure.” How
do we break through that?
Walter: I think that’s part of
it. I think part of it also has been that VR is somewhat of a glib
phrase. I think it’s a very descriptive phrase. But it’s also a
phrase that causes some people to think, as you described, that it
maybe it’s a little too light-duty and not an effective tool for
addressing large problems. But it’s changed. We now say VR or AR, and
people know what we mean, much like when we say AI. Whereas five
years ago, if somebody said VR, you wouldn’t necessarily know what
that stood for. So we’re getting used to that phrase. And in my
opinion, what the problem is — yes, what you described — but also,
it’s the fact that it’s sort of what we call a virality factor of
one. You almost have to see it and try it before you really get it.
And that means it’s not as contagious as some other things that catch
fire really fast. And I think it is going to really take off very
fast. But what we have to do is build the practical applications for
the enterprise. We’re in the process of doing that in architecture
and finance and skilled labor force training, soft skill training and
medicine. Now, medicine in particular, though, has a little bit of a
very appropriate barrier. We need to show what’s effective and we
need to show what’s safe. It takes a little bit time to do the
studies to demonstrate safety and efficacy, but we’re in the process
of doing that. And there’s a lot of early adopters out there who have
brought VR into their clinics, into their hospitals, and are doing
very good work with it. So I think things will catch fire very soon.
But I think we’re on schedule, now that the prices are reasonable.
And what’s slowing things down, I think, is we just need more people
to build out the practical tools that can be used in the enterprise.
Alan: So there is some massive
opportunities for enterprising students coming out of universities
like Stanford who are– I think they’re kind of benefiting from the
three decades of work that you and your colleagues have put in,
because in my opinion, what I’m seeing in the market is that now it
seems to be just ripe for explosion. It looks like we’ve got, the
rocket is on the track and all thrusters are going. And is that the
kind of sentiment that you’re feeling in the market right now?
Walter: I think it’s a good
analogy. I think that because the groundwork has been laid and
because the infrastructure is already in place. VR and AR technology
leverages the Internet and broadband that’s in place. 5G is on its
way. We’re leveraging distribution mechanisms. And also keep in mind
that at least in the medical arena, a big part of what we’re doing
relies upon other technologies such as machine learning. I mean, we
collect a lot data, but we need to have the tools to analyze it. It
relies upon avatar technology that look much more realistic and have
facial expressions and nonverbal communication aspects to them, so
that they look much more realistic than what we’ve been able to
generate in the past. So I think things are poised to take off
because there’s a convergence of technology. AR and VR technology is
going to leverage AI technology, it’s going to leverage simulation
technology that’s being used in a variety of other arenas other than
healthcare. And it’s all converging into one spot. So, yes, I think
it is poised– and I use the term poised, but I don’t mean poised for
taking off in ten years, and I don’t mean even five years. I think it
is in the process taking off now, and it’s going to move really fast.
Alan: I couldn’t agree more, to
be honest. I’ve been studying this industry inside and out and I
subscribe to Google Alerts for virtual and augmented reality. And
five years ago, I’d maybe get one alert every two days and have a
couple of things in it. I’m getting three alerts a day and they’re
packed. Let me read something. I think this is interesting:
“Medical practitioners should suit
the virtual reality application to the patient, not the patient to
the technology. The VR technology agenda in medicine. All
organizations face the problem of dramatic increases in volume of
data that they must manage to conduct their daily affairs.
Increasingly, this data is in a variety of media formats,
particularly in medicine, where data formats include CAT, MRI, EEG
and X-ray images, as well as real time communication with
consultants. Many technologies have been offered over the past 40
years to help with this escalating information resource management
problem. And now we have virtual reality.”
That is from Virtual Reality
Magazine, from 1993.
Walter: [laughs] That doesn’t
surprise me.
Alan: [laughs] I happen to have
three copies of this magazine. It’s just mindblowing that the promise
was there and it was 20 years too early. And now with the advent of
Oculus Quest and these really inexpensive headsets, I think it’s just
opened up the world to developers, and clinicians, and medical
practitioners.
Walter: Well, let me mention, to
snap on top of that point. I think there’s another trend that’s
worthy of paying attention to which is, at least in medicine, the
continents are colliding. The consumer electronics companies like
Apple and Samsung and to some extent Google and Microsoft, along with
companies like Amazon, are jumping nto healthcare. And that’s really
changing the game. The fact that with their speed of product
development and they’re savvy about good user interface design, to
have these groups jump in and team up with pharmaceutical companies
like Novartis and Sunovion and others, and to team up with medical
device companies like Penumbra, it’s really a wonderful time to see
the confluence of speed and savvy that is coming from the consumer
electronics company, combined with the experience of the medical
product development and distribution channel. It’s like a tidal wave
roaring down a racetrack. So I think things are going take off faster
than any of us will expect.
Alan: I’m going to read
something else. This is from a different magazine:
“Virtual Interface Technology
offers many applications for assisting disabled persons: augmenting
reality in rehabilitation medicine.”
By Walter J. Greenleaf and Maria A.
Tovar, from Spring 1995, Virtual Reality Magazine Special
Report.
Walter: Uh-huh. That’s right,
that’s right. And finally, the things that we were excited about back
then are now affordable. We’ve been able to do it in research
laboratories for decades. But now we can start moving it out into
clinical care.
Alan: So, people listening…
let’s say, for example, hospitals or clinicians or– what is the
practical first steps for them to start applying these technologies?
What would you recommend? I know you founded the IVRHA, so maybe you
can talk about that and how these hospitals and clinicians and
medical practitioners can– how can they get into this and start
using it right away?
Walter: Well, first of all, I’m
glad you asked about practical applications as opposed to asking for
“the killer app,” because we don’t use that phrase in medicine.
You’d be amazed at how many startups coming from the technology.
Alan: [laughs] Probably the
worst phrase ever for medicine. [laughs]
Walter: Yeah, but you know,
people still are talking about the killer app for medicine. But to
talk about how people can get started, I think going to the IVRHA
website is a good spot. You’ll see many of the medical product
companies and startups and service providers that are teaming to
explore the applications of the technology, it’s a great, great
starting spot. We also put on a yearly conference on medical VR and
that’s a great spot to come and meet other people that are working
the arena and try out demos.
Alan: Where’s that going to be?
Walter: It’s going to be in
Nashville. There’s a link to the website for it at the IVRHA website.
I think another way really for clinical groups to get started is to
well, I think take a look at some of the startups out there that are
doing pioneering work. You described how VR is being used to help
treat strabismus and amblyopia. For example, I would think for really
whatever indication that you’re interested in, be it depression, be
it anxiety, be it stroke rehabilitation, be it pain distraction, a
good way to get services, to just, of course, do a simple web search
to find the principal groups that are out there doing research, and
then see which product development companies are basing their
products on a research backed initiative and see who is citing and
using as part of their advisory group established research groups.
There are a lot of people that are jumping into the clinical VR arena
by translating their skills in game development or their skills in
sensor development into products. But not all of them understand the
medical ecosystem. Not all of them understand the ergonomics of how
to bring a product into a clinical environment, so that it doesn’t
slow down the process, so it doesn’t create additional burden and
work for the clinicians and the ancillary care staff.
So I would say as my filter, I would
look for those product development companies that have either brought
onboard research scientists and experts who know the medical
ecosystem to help them advise their technology direction and probably
more importantly, conduct validation studies. And I would also look
for those that have formed alliances with some of the existing
experts in the medical ecosystem, like some of the pharmaceutical
companies or medical device companies. They’re very selective. And of
course, the ultimate criteria is if somebody has made the effort to
go and receive FDA certification, has a product to demonstrate their
safety and their efficacy. That’s really the main watchword I would
look forward to, to see where to start is to find those companies
that address the problem you’re interested in that have gone down
that pathway.
Alan: Absolutely. You touched on
a couple of use cases, one of them being pain distraction. One of the
studies I read, it was something like 25 percent reduction in opioid
usage in debridement of wounds. And if you think about that, that’s a
massive reduction in medications that have detrimental effects on
people, and can lead to addictions and other things, just by using
VR.
Walter: It is very exciting.
You’re you’re talking about Hunter Hoffman’s work. And Hunter has
done a great job of demonstrating not only how we can use VR to
reduce the need for narcotics in a burn treatment clinic, for
example. But how the reduction in the need for narcotics translates
out into less possibility for addiction after people are discharged.
And that’s a really significant thing. A large proportion of the
problems we have with opiate addiction in our country right now are
because people get caught up in using narcotics as a result of being
in a hospital and appropriately getting pain reduction medication.
But if we can augment the pain medication and reduce the need for it
by using virtual environments as a gating distraction from the pain,
then all the better. And also VR been used for helping people with
chronic pain learn through cognitive behavioral therapy and other
approaches how to manage the chronic pain. So it’s not just during
the acute phase of a painful process, but also for the post
discharge, post acute phase.
Alan: When people are thinking
about this, what kind of, in business terms we’d say key performance
indicators. How do you measure the success of something over another
modality? What are the typical measurements if you’re going to start
using VR versus something else used in your clinic? How would you
measure that success? What does that look like?
Walter: That’s a very good
question. I would say for a medical product, it really depends if
we’re teaching a procedure that’s very different than if we have a
clinical assessment, and that’s very different than if we have a
clinical intervention. They all have different metrics. If you’re
measuring the ability to use VR to improve a training process, one
thing you would look at is not just how much more proficient the
training process is in terms of mastering a skill — such as maybe a
surgical skill or how to diagnose a patient — but you also would
want to know comparing it to the traditional methods. Is it less
expensive? By and large, it would be. Often medical schools have to
employ actors and actresses to help in part of the training process
or use very expensive simulating machines. You’d also want to know
what the retention is. Do people retain the lessons learned in a more
experiental matter with VR compared to other ways of learning, such
as looking at a videotape or reading a textbook or dissecting a
cadaver? And by the way, there are some medical schools that are
switching over to all electronic cadavers now, which is not only less
expensive, but in many ways more dynamic, they can overlay the image
of the cadaver with extra information and have a more structured way
of training. So for training, we would have those metrics.
For diagnosis, it would be are we able
to do a less expensive, more efficient, more accurate diagnosis to a
better differential diagnosis, maybe come up with less false
positives and false negatives as a diagnostic criteria? We would use
that. And also in all these things, we need to look at the cost. Does
the extra cost of deploying a VR system– and not just the cost of
buying the equipment, but the cost of putting it into the hospital
ecosystem, maintaining it, supporting it. Maybe there’s a need for
extra personnel to keep the batteries charged. So you really have to
look at the whole effect of bringing new technology into the medical
ecosystem. So that needs to be part of it. And same thing when for
when we use VR as a clinical intervention, does it save us money?
Does it save in the long run? Does it produce better healthcare
results? And you know, I have to tell you, Alan, when I’ve looked at
the metrics on all three of these things, training, assessments and
interventions, VR really can make a big difference. It really is cost
effective. But again, it has to be positioned the right way and not
brought into the clinical ecosystem in such a way that it produces an
undue burden. You want the clinicians to be able go home earlier, not
have to say a little bit later to do more paperwork.
Alan: It’s interesting you say
that because one of the things that came up at LiveWorx was device
management. We’re used to managing cell phones and iPads and stuff,
but this adds a whole new element of device management. It’s
something that people don’t consider until they have 50 of them to
deal with, or 500 of them to deal with.
Walter: Exactly the issue. But I
think the savvy product developers will appreciate that and that they
are building into their systems ways to reduce the paperwork burden,
improve the workflow of the clinic along with having a new tool.
Alan: It’s worth the investment,
in my opinion. From the numbers that you’re seeing, what are some of
the expected outcomes? Let’s say, for example, using this for autism,
it’s very hard to say with a number that this was more efficient or
something. It’s almost anecdotal, but there’s got to be numbers there
that prove that this technology far outpaces anything else we’ve ever
created.
Walter: Well, part of the way we
do that in medicine is by doing long term follow up. If, for example,
we use virtual reality to help with stroke rehabilitation, both
during the acute phase after a stroke and then the post acute, when
people are coming into a clinic for follow-up or doing home
rehabilitation, if we look at how they’re doing a few years later and
compare that to people who had standard treatment, if there’s less
permanent disability, if people are returning to work sooner, if
people are able to be more functional and take care of activities —
daily living is the phrase that we use — more effectively after
they’ve used VR and AR enabled interventions as opposed to the
standard, then that tells us something and we can put a number on
that in terms of dollars saved by preventing permanent disability and
helping people get back to work sooner. Same thing for– well, let’s
use the example of treating someone who has fear of heights or fear
of flying. We can compare those people who used a VR system to master
those problems and see how they’re doing a few years later. How many
of them are still flying? How many of them need to use maybe
anti-anxiety drug in order to get on a plane, versus those who don’t?
If the ones who had the VR treatment are doing better, then that
tells us a lot and we can quantify the value of that.
Alan: Let me ask you another
question from a different angle. Have you ever seen a time where VR
wasn’t more impactful or better than the traditional way of doing
things?
Walter: That’s a good question,
Alan. We’re still in a phase where we’re mapping VR to the problems,
and I think– Nothing is really coming to mind. I’m sure there have
been situations where we thought that VR would be a better way of
approaching this problem, and it turned out to not really be cost
effective. But I have to say, I’m really at a loss to come up with a
good example. I think mostly because, again, for many years the
technology was so expensive, we were very careful to really think
through how to apply the technology the right way. Now, I should say
I’ve seen a lot of where I think it’s not necessarily always the best
use. A lot of VR technology to help people learn to relax. And some
of those VR environments I think are not that much different than
listening to an audio tape or just closing your eyes and imagining
that you’re sitting at a peaceful spot. I think that there are very
amazing mindfulness training programs that promote an active
engagement, where you’re not just at a beach listening to the waves.
And those are very exciting. But I think there are some where the
incremental value of just having a VR environment is not as amazing
as it could be, but it’s still a learning phase. We’re learning how
to leverage the technology in the best way.
Alan: One of the things I saw
was — and I think it was the Cubicle Ninjas VR meditation, it might
have been them or another one — you actually connected your
smartwatch to the VR headset so it could read your pulse. You were
able to go through these deep breathing exercises guided, it is like
a guided meditation, deep breathing and you could actually watch your
heart rate go down as you controlled your breathing.
Walter: That is a good example
of a good use case, where we’re leveraging not just the display
technology, but incorporating the measurement of psycho-physiological
signals and using it to dynamically change the environment. So like
anything, there’s a spectrum of innovation that can be applied. And I
think when you asked me where VR isn’t really that effective, all I’m
really finding is to think of our times where, it’s effective but not
as effective as it could be. If we add extra layers of analytics, or
extra layers of social connection, or extra layers of leveraging game
technology to make things more rewarding and exciting. So but again,
we’re really just getting started now. Now that the products are more
affordable, I think you’ll see a surge of people applying them to
clinical issues. And we probably will see some that are sort of
poorly designed and lame, but I’m sure we’re going to see some that
are just incredible, too.
Alan: Yeah. I would have to say
that we’ve kind of passed the point where we know which VR techniques
make people sick. So I get really, really bad motion sickness.
Walter: Uh-huh.
Alan: I’m the guinea pig for the
office. But yeah, there are still some companies out there making
things that make people sick.
Walter: Yeah, and I think that’s
a matter of poor design, if you moved the world around someone, it’s
bound to make them sick. But if you give them agency to move through
the world, it’s very different. Let me tell you about one way we’re
applaying VR and AR technology to medicine that I’m particularly
excited about. There’s a focus now in medicine to move towards more
what we call precision medicine, where instead of a one-size-fits-all
approach for treatments — where you go down a clinical pathway and
say, well, let’s try this and see if it works, and if it doesn’t
work, we’ll try something else — where we leverage the genomic
information about individual, the measurements we can get about their
behavior and their physiology. And we challenge them with virtual
environments, for example, to see how they react. And we come up with
a precision pathway for treatment that is based on their specific
biotype.
One of the projects we are in the midst
of doing at Stanford University is a project headed up by B Williams
called the Engage Project, where we’re looking at people who have
comorbid depression and weight management issues. And we’re following
them over a multi-year period of time, as they learn to manage their
weight and manage the moods. And they have a variety of different
interventions. But as they go down that treatment pathway, we’re
imaging their brains using a functional medical imaging device. And
we’re also measuring their behavior using data collected by– they
opt-in to allowing us to collect data on their smartphones about how
fast they’re typing, and how fast they’re swiping, and their general
activity levels. And then we’re also using virtual reality as a way
of challenging their neural circuits to see– we have a challenge,
much like Beat Saber, where they have to learn to react the right way
and maybe some other times control their reaction. And the idea here
is to come up with a VR based system that can be used to identify
different neuro circuit biotypes, and that can be used to come up
with a more precise approach for treating depression, for example. So
for biotype A, we might recommend that they try this treatment for
depression, and a different biotype — again, identified by how they
react to a stimulus in a virtual environment — might be recommended
to go down treatment pathway B and a third person might go down
treatment pathway C, all based on their assessment that we’re able to
do using virtual reality. So it’s a research project, but it harkens
to the vision that we can leverage not just VR technology, but also
sensor technology, biosensor technology, and machine learning
technology to really come up with a better way of adapting our
medical interventions to the specific individual.
Alan: It’s incredible. You talk
about the three pillars: training, assessment and intervention. But
one of the things that I think must be under the assessment side of
things is being able to visualize your data. As a physician, being
able to take an MRI data and blow it up and get right in there, I
think we’re only scratching the surface of that. Have you seen any
real world applications of visualizing medical data in unique ways in
VR and AR?
Walter: Oh, absolutely. For
example, there’s some groups doing virtual colonoscopies where they
collect the colonoscopy data, but sometimes it collected just using
an imaging machine as opposed to doing the actual physical
colonoscopy and then allows a clinician sort of fly through the colon
looking for polyps. Or same thing for pre-surgical planning. We can
take CAT scan data, fuse it with ultrasound data, fuse it with other
data, and then plan a complex surgical procedure in advance. So
allowing the clinicians to– not just the radiologist but also the
surgeons, to visualize in three dimensions the complex process that
they need to go through in their operation and rehearse it in
advance.
I’d like to add to those pillars that
you mentioned, too, though. It’s not just training and assessments
and interventions. I think the AR technology is also going to be very
helpful for promoting health and wellness, showing us the effect of
our behavior and promoting us to do what we really should do to be
healthy. It’s a very difficult thing to get yourself to exercise, get
yourself to eat the right way, to remember to take your medications,
especially if they have maybe side effects that you don’t like, and
to understand the consequences of not adhering to what you should be
doing to keep yourself healthy. One technique that was pioneered at
the Stanford Virtual Human Interaction Lab was to create an image of
your future self, an avatar of your future self that you could talk
to and who could talk to you, and show you in a shortened timeframe
the consequences of your decision. So if, for example, your
smartphone is sensing that you’re going into a bar and you’ve already
decided that really you want to cut back on your alcohol use, your
phone can ring, and there on your phone is your future self. Someone
who looks like you, sounds like you, but 20 years older saying, “Hey,
I thought we talked about this. Look at what you’re doing to me.”
Or maybe if you’re not exercising enough or if you’re not eating the
right way, it could show up in your future self looking very
unhealthy. Same problem for managing addictions.
So I think prevention wellness is
another pillar. I think another thing to keep in mind is I think VR
and AR technology is going to allow us to reach underserved
populations. I think the focus of care will shift from the clinic to
wherever the individual is. There’s some things, of course, we have
to do at a specialized treatment center where there’s specialized
equipment. But I think for a lot of things, especially in terms of
behavioral medicine and psychology and psychiatry, I think we’ll be
able to start doing a lot more reaching underserved populations and
be able to provide effective treatments by leveraging the
telemedicine nature of VR and AR technology.
Alan: Well, that’s wonderful.
It’s a nice close to this conversation, because if we can use these
technologies to democratize wellness and health, not just medicine,
not treating people when they’re sick, but really just keeping people
healthy, I think that is the fundamental shift that we need to make
as a society. And I think VR and AR lends a very good hand to that.
Walter: Well, and especially if
we can follow the pathways that our colleagues in the computer gaming
arena have pioneered. They know how to grab people’s attention and
keep them involved in a process. So if we can look at how they have
learned to do that in the gaming arena and apply that to some of the
interventions we’re doing in the medical arena, I think we can make
medicine not just more effective, but also more engaging and more
palatable.
Alan: And fun. Let’s not take
away the fun. I want to say, on behalf of all the listeners and
myself included. Thank you so much, Dr. Greenleaf, for taking the
time to be on this podcast. It’s been really amazing.
Walter: Well, thank you, Alan.
Thanks for all the good work you do. Getting the word out and for the
good questions today.
43 min
September 30, 2019 Three Decades of Medical VR, with Stanford's Dr. Walter Greenleaf
XR has come a long way, baby – and we have one of the technology's earliest pioneer's on today's episode. Dr. Walter Greenleaf has been working in the field for 33 years, since the days when VR was little more than a twinkle in research scientists' eyes. Now, he and Alan chat about how far the technology has come, and how far it still has to go.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today's guest is Dr.
Walter Greenleaf, a behavioral neuroscientist and medical technology
developer working at Stanford University. With over three decades of
research and development experience in the field of digital medicine
and medical virtual reality technology, Walter is considered the
leading authority in the field of working in this industry, and he's
been doing this for 33 years. Unbelievable experience. Dr. Greenleaf
has designed and developed numerous clinical systems over the last 33
years, including products in the fields of surgical simulation, 3D
medical visualization, telerehabilitation, clinical informatics,
clinical decision support, point of care, clinical data collection,
ergonomic evaluation technology, automatic sleep staging systems,
psycho-physiological assessment and simulation assisted
rehabilitation technologies, as well as products for behavioral
medicine. Dr. Greenleaf's focus has always been on computer supported
clinical products, with a specific focus on virtual reality and
digital health technologies to treat post-traumatic stress disorder,
anxiety disorders, traumatic brain injury, stroke, addictions, autism
and other difficult problems, and behavioral and physical medicine.
He's currently a distinguished visiting scholar at Stanford
University's Media X program at Stanford University's Virtual Human
Interaction Lab and the Director of Technology Strategy at the
University of Colorado National Mental Health Institute Center. To
learn more about the work that Dr. Greenleaf and his team are doing,
you can visit the Human Interaction Lab at Stanford at
vhil.stanford.edu and a new
organization that he's formed called the International VR Health
Association at ivrha.org.
Welcome to the show, Dr. Walter
Greenleaf. So great to have you.
Walter: Thanks, Alan. I'm
pleased to be here with you.
Alan: That's an honor. You are
considered one of the godfathers of this technology. You've been
working in it your whole life. And I want to personally say thank you
for laying the groundwork that allows people like myself -- the new
people getting involved here -- to really pick up where you left off,
and where you've driven this whole industry forward to, and let us
really build upon your lifetimes of knowledge. So thank you very much
for paving the way for us.
Walter: Well, thank you, Alan.
And really, without everybody else's and my colleagues work and your
work and other people who are helping the trends position from
something that for a long time was a research lab curiosity and
something that really hadn't escaped the confines of academia. Now we
have it out there in the world. And I'm particularly excited about
all the progress has been made in applying VR and AR technology to
difficult problems in healthcare. For me, it's a very exciting time.
Alan: I've been keeping track of
all of the different things that come up in my news feed and I have a
health and medical folder. And it's interesting, because last year I
actually had to break it apart into a mental health folder, in
addition to the traditional health and medical. So there is an
enormous amount of, not just research, but real practical
applications being created for this. You know, one of them, one that
stands out the most to me is being able to use virtual reality to
treat lazy eye or strabismus.
0 min
September 27, 2019 HR in XR, with BrainXchange’s Emily Friedman
As the lead
writer and head of content at BrainXchange, Emily Friedman has had
ample chances to explore a lot of XR-related topics. She lets Alan
pick her brain about a few of them, from getting millennials
interested in trades, to democratizing knowledge, and how humanity
will enter The Cloud.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today’s guest is
Emily Friedman from BrainXchange and Augmented World Expo. Emily
Friedman is a New York based enterprise immersive, wearable and
emerging technology advocate, journalist and facilitator. She’s Head
of Content and the lead writer at BrainXchange, lead journalist and
senior editor at Enterprisewear Blog, and head of marketing and
communications for Augmented World Expo USA and AWE EU. To learn more
about BrainXchange, you can visit brainxchange.com. And if you wanna
learn more about AWE or Augmented World Expo, you can visit
awexr.com.
Welcome to the show, Emily.
Emily: Thank you for having me.
Alan: Oh, it’s my absolute
pleasure. I’ve been really looking forward to this conversation,
because you are writing everyday – or, not everyday, but what, a
couple times a week? — on the enterprise wearables world. So maybe
just kind of give us an overview of what is BrainXchange and AWE.
Let’s start with that.
Emily: Ok, I wish I were
productive enough to write multiple articles a week. But there’s a
lot going on. BrainXchange, we started out as a boutique events
company, and we just happened to enter augmented reality at the right
time. It was 2015, right after Google Glass, quote/unquote failed.
And there were all these headlines, “Glasshole” articles. But if
you read between the lines, it was clear that smartglasses weren’t a
failure, and that enterprises were actually finding good use cases
for it. So today we provide events, content, and other services all
related to facilitating enterprise XR.
Alan: You know, I’ve been at AWE
a couple of times now. I lead the startup track this year. It’s an
important conference for virtual/augmented/mixed reality and some may
say it is the most important conference. It’s where everybody around
the world gathers in. And I made this comment that if the building
happened to collapse, basically the entire VR world would cease to
exist, and we’d have to start over again. It was an amazing
collection of some of the world’s smartest people working in this
technology and enterprise. They seem to be really driving this
technology forward. What are you seeing?
Emily: Well, as for AWE, I think
it’s a very important benchmarking event. Like you said, the entire
industry gets together at that one point. What we’re seeing — and
the reason we gravitated towards enterprise at first — is that
that’s where the money is. I mean, that’s where the money has to be
made, both for end users and the AR/VR companies themselves. At the
end of the day, we cater to the enterprises and we talk to them every
day. We get on the phone with Fortune 500 companies, the innovation
people and all these different companies every day. And we listen to
their pain points. AR/VR happens to offer a solution to a lot of
their pain points.
Alan: So what are some of the
pain points? Let’s unpack that.
Emily: Huge one is a shrinking
workforce, that creates this need to train faster, better. So as the
workforce ages — in manufacturing, I think the average age is like
40 to 50 now — and retires, not only do you need to attract new
talent; you need to train them. As a millennial, this is actually
pretty important to me. Learning a skill today just doesn’t get you
as far as it did half a century ago. Tech advances, business models
change, and much of what I learned in school, I feel like it’s
irrelevant. And for Gen Z, it’s going to be worse.
So the ability to learn new skills
effectively — to upscale, to rescale — is really important. Another
one is remote support. Those are probably the two most enthusiastic
applications today. For us, it was one of the earlier ones, just
being able to connect your team remotely. Lone worker in the field —
say a field service company’s fixing an air conditioner — can access
talent of experts at a home office, and give them a view of what
they’re seeing. And that’s just really powerful.
Alan: Awesome. So remote
support, for example. What is the problem? What is the underlying
problem? Because one of the things you mentioned is attract and train
new talent. I think the key is that attract, because kids today,
they’re waking up in the morning, they’re opening Instagram, they’re
like, “I want to be an Instagram celebrity.” We can’t all
be Insta-famous, but there’s a lot of jobs that are in trades that
are great, good paying jobs that young people just aren’t, maybe
they’re not even aware of it or they just don’t care. I know a lot of
parents push their kids to go to university, even though that if you
look at university now, we’ve got a trillion and a half dollars in
debt in the US from student debt. So trades are a real value to the
economy. And how do you attract and excite people about those jobs?
Emily: So, as for the field
service, remote support question; the pain point there is time. So
not having to do the job twice, you send one person out and then you
need to send another person out to help that first person. As far as
the trades, I think our education system hasn’t really kept up with
the economy, like the actual workplace, but it’s necessary.
Alan: And it’s actually almost
impossible for education systems. If you look at the way they were
designed, they were designed not to change. They were designed to be
steadfast in the face of change. And that, unfortunately, when you
enter into exponential growth phase of humanity, this becomes a real
problem.
Emily: Exactly. So I think that
it’s both. It’s that they’re not learning the skills they might need.
And by the way, trades jobs don’t have to be manual. They’re not low
skill, they’re high skill. And there are now often involving
technology.
Alan: Oh, absolutely.
Emily: In that way, I think our
education system hasn’t kept up. And I think you’re right. My
generation were not aware of the trades. And I think skilled trades
training has dropped off a lot.
Alan: We’re seeing– starting to
see some some new technologies, like VR and AR, that are starting to
bridge that gap. There’s a couple of companies making some really
interesting headway in virtual reality training, and I’ve tried a few
of them. Pretty impressive.
Emily: Yeah, exactly. So one
having appealing technology like AR/VR is definitely attractor.
There’s the other side of this, the other coin, which is that older
workers don’t have to retire, now that there is AR and VR, because
they’re still valuable to the organization, even if they can’t go
onto the factory floor or out into the field. I think there’s both
sides of that, and it’s really important to cater VR training and
adopting AR in order to collect and record all the inborn talent in
your company, and also to share that with new workers and help them
learn fast.
Alan: I had a chance to try this
at PTC LiveWorx recently. I put on a RealWear — it’s basically like
a heads-up display, it’s like having a tablet a foot from your face
and it bends out of the way when you don’t need it, you just flip it
up — and I was standing in a tractor. I pulled down the thing and it
walked me step by step how to change an air filter. And I’ve never
touched a tractor, ever. I don’t know anything about a tractor. I was
able to remove the air filter, check it, change it, replace it, close
it back up. And it was ready to go. And it was interesting because
what I was watching wasn’t some crazy AR overlay. What it was, was
just a video that was captured by an expert on the same device that I
was wearing to see it. It was impressive.
Emily: I mean, it can be as
simple as arrows in your field of view.
Alan: Yeah.
Emily: There’s a range, from
assisted reality all the way through mixed reality. RealWear’s a
great example. They’ve had some really large rollouts, which I think
is such a great sign for the space, and they’re going to be at EWTS.
Colgate, for example. I think BMW recently just rolled out RealWear
devices to a bunch of its plants. So they’re growing and definitely
the companies have matured. I think we’re at third, fourth generation
devices at this point and the software has matured.
Alan: Absolutely. I saw
something, I think three years ago at AWE, and it was a pick-and-pack
type of thing. We put on the glasses and it walks you through picking
things off shelf for distribution warehouses. And it was really bad.
It was laggy, it didn’t work, it was kind of crappy. And fast forward
this year, oh my God. It was like, millimeter accurate. It was just
intuitive. We’ve come a long way in a very, very short amount of
time, which I guess that lends itself to the word exponential growth.
Emily: Yeah. And I think EWTS
offers this opportunity from the start. We’ve kind of curated the
sponsors, the exhibitors at the event, making sure that they have
solutions that are ready to go today. I think it’s really valuable
for them to hear real end users, real enterprise end users on stage,
sharing their experiences, good and bad about their technology. And I
think it’s helped move some of that user experience issues,
ergonomics issues forward.
Alan: So you mentioned EWTS,
Enterprise Wearable Technology Summit in Dallas, Texas, correct?
Emily: Uh-huh.
Alan: So who are some of the
companies that are going to be attending this?
Emily: Well, it’s really the
Fortune 1000 makes up our audience. It’s primarily heavily enterprise
for the audience. Few are solution providers. But it’s been growing
every year, like you said, exponential growth. We have veteran
speakers at this point who return year after year to give us updates
about their experiences. Peggy Gulick from AGCO. Janelle Haines from
John Deere. Josh Shabtai from Lowe’s. Gary Binstock from Colgate,
who’s using RealWear. Dan Jost from Molson Coors is returning this
year, and it’s across the industry spectrum. Every industry, pretty
much.
Alan: I’m looking at the speaker
list here, it’s incredible. So if you want to know more about this,
it’s brainxchange.com and just look for EWTS, or just Google
“Enterprise Wearable Technology Summit”.
Emily: And it’s been fascinating
to have this point of view, to come at it from the enterprise point
of view, and really speaking with the enterprises every day. So Duke
Energy has been with us since the beginning. Boeing as well. And so
to see some of these become large rollouts, thousands of devices —
in Wal-Mart’s case, hundreds of thousands of devices — or standard
work tools like they are at DHL, that’s just standard. What you’re
saying, pick-and-pack software on smartglasses, that’s standard for
them now.
Alan: Wow.
Emily: And to see that is just
such a wonderful thing.
Alan: It’s interesting how it
goes from being this fringe technology to industry standard.
Emily: Yeah, it’s interesting
that that has remained cross-industry. We do tracks by industry, but
right now there’s only a certain number of– AR/VR has a certain
number of applications. And I think every business has a component
that it applies to. So whether that’s heads-up hands-free
information, remote expert, visualization, training, sales, that’s
pretty across the board. So I think this ability to network and learn
from other companies that are starting to bring this into their
company is really helping a lot.
Alan: Absolutely. You mentioned
different kind of aspects of this, and it looks like it’s every
industry. You got Pfizer. You’ve got AGCO, agricultural. Lowe’s,
retail. Dow, chemical. Wayfair, retail again. So what are the ways
these technologies, are these typically in the enterprise? Is this
something that they’re using in their warehouses? Is it something
they’re using to train? Is it something they’re using for marketing,
or are you seeing any one company that’s kind of using it for
everything? Or is it just kind of siloed right now, still?
Emily: Well, all of the above.
Alan: [laughs]
Emily: Companies are using this
for sales and marketing. And that’s actually not something that was
very prevalent a few years ago. It was really heavy on field
services, utilities, logistics, but it’s now coming into sales and
marketing. As consumers right now, we don’t really have access to
AR/VR tech. There isn’t really a great AR smartglasses for consumers
out there right now. And VR is expensive for most people, or they’re
unaware of it. But sales and marketing aspect allows companies to
give their customers a taste of AR/VR, because they can afford it.
That’s kind of new and that’s definitely a new feature of our event.
Like I said, we have veteran speakers and some of those speakers are
really– their use cases have evolved to multiple use cases. So I
know AGCO is using Google Glass all over.
Alan: Really? How are they using
that?
Emily: I’m not sure, because
that one’s kind of new. In a lot of companies, this replaces the need
for going back and forth to a computer, or looking down, or picking
up above. It’s really as simple as going hands-free. I think that’s
one of the most powerful aspects of wearables and smartglasses and VR
headsets.
Alan: It’s interesting. There’s
companies doing this. And the more they start to present their
findings, the more it becomes a no-brainer for this technology. I
believe it was Shelly Peterson from Lockheed Martin. I’m not sure if
she was one of your speakers, but– oh yeah, she spoke at AWE! That’s
right.
Emily: Yeah. And she’s also been
with EWTS since the beginning.
Alan: Brilliant mind. And one of
the things that she mentioned at this year’s AWE event was that
they’re seeing average 85 percent decreases in training times, and 25
to 50 percent increases in retention rates. And this is incredible.
Like, I don’t know that there’s any technology we’ve ever invented as
humans, that have that kind of impact on our bottom line.
Emily: And on our productivity.
Yeah, I think this is just really groundbreaking. This is the first
time you get to put yourself in someone else’s shoes for real. So
whether that’s putting yourself in another culture’s shoes or a job
that you’re trying to learn. It’s just so powerful, that firsthand
experience. And those numbers that Shelly gave, they’re not unique.
Lots of companies are getting numbers like that. And it is really
astonishing. But again, like I said, taking information out of
people’s hands and putting it in front of their face is just
incredibly powerful.
Alan: Yeah, it really is. And
the devices themselves are getting better by leaps and bounds as
well. There’s a bunch of new devices coming out every day, and the
field of view is getting better, the battery life is getting better.
It’s that exponential growth of hardware as well. And so I think it’s
this perfect storm of the timing being perfect for this technology to
impact every business.
Emily: It’s a shame, kind of. I
feel like every year has been the year of AR/VR.
Alan: [laughs] We’ve been trying
wolf a long time.
Emily: Yeah, exactly. I think
2018 was a little disappointing, in terms of the solutions
themselves..
Alan: I agree.
Emily: This year was the first
year I walked around AWE, and was just so impressed with the level of
the technology. This year is also EWTS’s biggest expo. And we back
our exhibitors, because we want these solutions to be ready to go. It
helps that a lot of the big companies, HTC, Oculus, Lenovo, they’re
pivoting to enterprise. So it’s just grown a lot.
Alan: It’s really amazing to
watch. My company, MetaVRse, we’ve been in the business side of
things. We’ve done that from day one. We looked at the business
applications of this technology first and foremost, because the way I
looked at it was like, “OK, this isn’t like a cell phone, where
it’s easy to put in everybody’s pocket and scale. This is something
that’s going to require a use case that you don’t mind looking like
an idiot with these glasses on your head.”
Emily: Right.
Alan: When you go back four
years, the glasses were huge, and they were connected to computers,
and they just weren’t something that would scale. And even the
Hololens, I mean, Hololens 1 is a great device, but man, you wouldn’t
want to wear that all day. But for an application specific, “I
need to look at this machine, fix this machine, get in and out
quickly.” That is a powerful, powerful piece of equipment. And
everybody goes “Oh, it’s $3,500, it’s too expensive. It’s never
gonna be a consumer hit.” It shouldn’t be a consumer hit. It
should be something that is used by enterprise, because $3,500 to
outfit a factory with one or two or ten of these devices is a drop in
the bucket to the downtime caused when these machines, these big
manufacturing machines are down. If you’re down for a day that’s
multimillions of dollars in downtime.
Emily: Exactly.
Alan: And if this device can
save that, then you’re winning.
Emily: Especially for an
airline. Having a plane out of commission is so costly. Time really
is money in business. And while I don’t think the use cases are
really there for consumers yet, and the devices aren’t quite there–
although I was really impressed with Unreal’s mixed reality glasses.
Alan: Oh, those are great.
Emily: But they’re not out yet.
So like we’re moving forward a little bit. But in enterprise, it’s
not just about getting inside of a machine and having these really
powerful visual images that help you get to know what you’re doing in
front of you. It’s also design, cutting down the design process, and
I think it will unleash new creativity from designers, whether that’s
engineers, builders, products. I think being able to create your
product in mixed reality is just going to have such an impact on that
process. It’s usually really long. If you think about a building
project, there are so many stakeholders in a building project. And
not everybody understands the plans, especially if it’s a public
building, and now you have to bring in people from local government.
It’s such an amazing way to quickly refine. It’s like testing out
things — refine and go, refine and go — and helping others to see
what your vision is.
Alan: We’re seeing similar
aspects in car companies, in aerospace and design. It’s really
incredible. Then you have companies like Spatial who are allowing
people to collaborate in augmented reality or mixed reality in
different spaces with people from around the world.
Emily: Yeah, it’s just an
incredible time saver and it’s more powerful. It’s easier to
understand something that’s in front of your face, something that you
can experience, and it cuts down on physical models. That’s really
where the time saving is.
Alan: Yeah.
Emily: It’s communication and
those physical models. Being able to iterate. It takes so much less
time. You don’t need those physical products. You don’t need to
return to a plan, and print something new, or get everyone together
again. It’s just an incredible time saver. I also think for designers
themselves, like I said, that it will unleash new forms of
creativity. And I think this is important as product cycles get
shorter. New products are coming out at a much faster rate and there
is a lot of connected products, too. So I think this has all been
just really great timing.
Alan: I agree. I’m going to
shift gears a little bit because one of the articles that you wrote
was talking about XR in HR.
Emily: Yes.
Alan: Or Human Resources. What
are some of the things you’re seeing in that? Because this is a
totally different way to use this technology,y so the technology
doesn’t change. You’re still using VR/AR/MR, same glasses, same
headset, same production methods. But a completely different use
case.
Emily: Yeah. So this still in
the pilot phase, I don’t think Fortune 500/1000 companies are here
yet, but I– reason I wrote about it is, it’s just, again, this is
the first time we’ve ever been able to step into someone else’s
shoes. You can form memories in VR. There’s been tons of studies at
Stanford. You can change behavior. For how long, I don’t really know.
And I’m looking forward to the studies that will be coming out in the
future. It’s just so powerful. And today, traditional HR like sexual
harassment training, unconscious bias training, it’s just not
effective.
Alan: So you talked about XR in
HR. There are companies working on this, there’s a company, I think
called Uptale? I want to say Uptale. They’ve created this experience
where you are an HR manager and you’re talking to somebody. And then
after you deliver your talk, you actually get to sit in the other
person’s eyes and look at yourself, giving you the advice back.
Emily: Exactly.
Alan: What a powerful tool.
Other than video, you can record yourself in a video talking to a
camera. But talking to another person and being able to sit in that
person’s eyes and watch yourself, your body language, your eye
contact, everything. That’s crazy.
Emily: And it’s also, again, a
financial thing. Workplace discrimination costs businesses over
$60-billion a year. McKinsey has predicted that we could add
$12-trillion to the global GDP by simply advancing gender parity and
diversity in the workplace. So I’m really hoping these XR startups
that I’m seeing equal reality vantage point — Morgan Mercer is
really inspiring — I really hope this becomes more standard. But
again, there’s a lot of studies in all aspects of using AR/VR for any
kind of applications. There are studies that need to come out. Long
term effects. Does it really change your behavior? Can you be
traumatized in VR? Can you be bullied in VR? There’s a lot of work
left to do. But I think XR for HR is such a promising application.
Alan: I agree. One of the other.
There’s so many articles, if you’re listening and you want to learn
more about this stuff, Emily is a prolific writer and maybe you don’t
write once a week, but there is a lot of content here. One of them
that I was reading was “Home
on the VRange: Immersive Technology in Residential Retail.”
The reason why I picked up on that one is because we have a program
called XR Ignite, which is a community hub, an accelerator to connect
startup studios and developers with corporate clients. And we were
reviewing the applications this weekend and it’s been amazing. First
of all, we’ve had over a 130 applications in the last couple of
weeks, but one of them was this home AR app where you can take a 3D
CAD model of a building, of a house, drop it in your space, and then
you can walk around it. You can shrink it down to dollhouse size. You
can have it as full size. You could literally see what your new house
is going to look like, and not only see it, but walk through it, and
do that all using your phone. And eventually it’ll be a pair of
glasses, but for now, it’s the device in everybody’s pocket. And I
thought that was just an incredible tool for visualizing real estate.
So what other things have you seen?
Emily: Also in terms of real
estate, that was an early sales in marketing, real estate was pretty
quick to this game. And I think one of the reasons is that there are
CAD models, there is the CAD information, there is BIM. So they had
more of a foundation to create VR and AR experiences. I know right
now I’m looking for an apartment, I’m doing the hunt. I’m moving from
Manhattan to Brooklyn. And pictures are deceiving. So when it comes
to luxury items, or high ticket items, big ticket items, cars, luxury
goods, an apartment, furniture, things that are hard to return or you
can’t return it, you get stuck in your lease for a year. This adds a
whole new aspect that enables remote shopping for these kinds of
things.
Alan: One of the podcast
interviews I did today was with Mohamed Rajani from Macy’s, and
they’re using VR to give people the experience of seeing new
furniture, and the stats that they’re seeing are absolutely
incredible. I mean, you’ll have to check out the XR for Business
Podcast to find that link. But wow, the results are astronomically
high and they’ve rolled it out to over 100 stores now.
Emily: Yeah, and Lowe’s was
pretty quick, also.
Alan: Yeah, Lowe’s has been
working on this for a long time.
Emily: Wayfair speaking in our
event. I love the Lowe’s case, though, because it really gets at the
providing AR/VR to consumers, at a time when they can’t or won’t buy
it themselves. That’s a gateway to consumer AR/VR picking up. The
chance to experience AR/VR for yourself in a store to connect with
the brand, I think it’s gonna help the exposure problem. A lot of
people just haven’t been exposed to AR/VR.
Alan: I agree and I think I
really love what Lowe’s did with their training. I got to try it
at… maybe it was AWE, but I got to try one of their training
simulators and I was tiling a wall in a bathroom. I had to mix the
mortar, and then I had to– in my brain, I’ve done that. I’ve
actually tiled a wall. It may have been in virtual reality and
physically haven’t really tiled a wall. But in my mind, I’ve done
that.
Emily: You’ve formed a map for
it.
Alan: I did.
Emily: Yeah. [laughs] What’s
interesting also, there are work force facing applications and
customer facing applications. So Bose is one of those companies
that’s looking at this from all aspects.
Alan: They really are. They’ve
been working on this for a long time. I remember their original
caves, where they had these kind of markers all over the wall to
track where you were in 3D space. They’ve come a long, long way since
that.
Emily: Yeah. And I think what it
shows is that, like your example, learning how to tile a wall. It
democratizes knowledge and information. It’s going to shift jobs,
definitely. You can look at a video and watch a tutorial in front of
your face and fix your own sink. That’s gonna be one less job for the
plumber. So it’s interesting to see these shifts happening and how
putting information in consumers hands is really important. And
that’s actually a big part of real estate is putting the agent and
the buyer or renter on the same level, as far as being able to
picture an experience.
Alan: Yeah, indeed. One of the
articles that you wrote, and I think this cuts to the heart of why
we’re not seeing a wider adoption. I mean, you know, if you look at
the Fortune 1000 companies that are coming to BrainXchange events,
these are the early adopters. Let’s be honest. There’s thousands and
thousands of companies that haven’t even tried AR in their factories.
Emily: Yeah.
Alan: One of the articles you
wrote is Building
a Culture of Bottom-Up Innovation and how to get this adopted
within a company. So what are some of the tips that you would give to
people listening? How would they get started? How do they get that
foot in the door to find those use cases to really develop that,
internally or externally?
Emily: So this is something that
I’ve watched for five years, and this is a really, really strong
suggestion. Start with your workforce. Go to the users right away.
Oftentimes your end users, your workers probably created hacks for
themselves to make their job easier, that you’ve never considered.
They know what their pain points are. They know what they wish. For
big companies like GE, creating an innovation hub or something, where
workers can come every day, workers can come and try out new devices,
and there are lots of good ideas. That’s great. That is building a
culture of bottom-up innovation and it helps a lot with rollout.
You’re going to get less backlash from your employees. The other
aspect of this — and this is something I learned from Ron Bellows at
AIG — is a bottom-up culture also means that all departments kind of
have to start working together. You know, traditionally operations
and IT are very separate, IT and HR are very separate, EHS very
separate. And I think what this does is it brings together, everybody
got to be at the table to make it work.
Alan: I think one of the things
that’s really intriguing is the fact that the tools, both hardware
and software, are just getting so much easier to use.
Emily: Yeah.
Alan: A few years ago, we were
coding things in the hundreds of thousands that now we can do for the
tens of thousands.
Emily: Yeah, definitely.
Alan: And the costs are
dropping. There is also now way more people around that can do it.
That’s another thing that’s interesting is that now there are more
people that know how to build this stuff. So it’s not just 10 people
around the world that understand how to make this. It’s getting
there. But I think there’s going to be a shortage of talent. As more
and more companies realize, “We’re getting 60 percent cheaper
training or whatever and we’ve got to to ramp this up.” So one
of the reasons we started XR Ignite, again, was to help facilitate
acquisitions of small studios into these, because companies are going
to want to do this fast and they’re going to want to scale, we’re
already starting to see studios being acquired by Accenture and
Walmart acquired a studio. And so there is not just the technology
part of the hardware and the platforms, but also the studios that are
creating the content. How is that relationship all working between
platforms and studios and content providers and independent
developers and all of that?
Emily: So this is kind of one of
the things I was talking about when it came to attracting new talent.
Certain jobs are going away, but tech jobs are moving into the
skilled trades. Needing content creation is a way to get younger
professionals involved. As far as studios, AC companies had it really
easy, because they had all that information, that data, CAD models to
start with. One of the biggest hurdles for companies is where to get
the content. And if you’ve been using manuals, if you’ve been using
computers, spreadsheets and checklists, it’s hard. That is really
hard. So yeah, you do need a studio, but I think those applications
are slightly easier to create than like training or something
consumer facing. I’m hoping that as AR/VR becomes more and more
popular and the big companies like they’re creating tools that make
it easier for developers or anyone to build AR/VR experiences. Google
has some easy tools. Mozilla has easy tools. And as the big companies
come in, I think they’ll probably have their own services as well.
Like the cloud.
Alan: Yeah, I agree. Microsoft
has actually– it’s interesting, because the Hololens was in their
devices division and they actually moved it last year over to cloud.
So the Hololens is now a cloud product, which is interesting when you
think it’s a device, but it’s really a device that enables their
cloud, is really what it enables. And I think that’s where you’re
seeing the shift of like, “Wait a second. You know, these
headsets are great, but they’re just a tool to show data. And the
amount of data that they consume or generate is enormous.” It’s
got the telco companies and the cloud computing companies salivating
for what’s next.
Emily: Exactly.
Alan: Because once you get to 4K
TVs and 8K TVs, what’s next? And calculating spatial computing, being
able to put everything into 3D. That is a huge amount of data.
Emily: Yeah. We’re gonna need
consortiums, whether that’s in the form of an Amazon type company or
Google or Facebook stepping in. We’re going to need that. I also see
AR/VR devices like Hololens is the way that the human being is
brought into industry 4.0. It’s how we’re connected to all this
digital transformation and IoT that’s going on. It’s how we enter the
cloud. So I think in addition to content creation, there are going to
be a lot of data analytics jobs that are needed.
Alan: Oh my goodness, it’s going
to be crazy. One of the things that I heard at a conference once was
based on eye tracking, head position, pose estimation, how you move,
how you– they can even tell how you’re breathing by the way your
head moves, because when you breathe you kind of move ever so
slightly. We have sub-millimeter accuracy head tracking and eye
tracking. We’ve never had that kind of data, no matter what we’ve
done. We can study people left and right, but we’ve never been able
to study them at that micro level until now. And one of the speakers
said Google will know that you’re gay before you are, because of what
you look at.
Emily: Yes.
Alan: It’s an interesting
thought. But what will this unlock when we know everything about the
intent of a person before they do?
Emily: Yes. I’m not trying to
get political, but our government really does need to enter the
picture. I kind of operate on the fact that Google knows everything
about me. But like you said, with wearable technology, putting things
on our faces, the information that we’re giving becomes way more
personal and way more sensitive. There’s good and bad to all of it. I
think this is going to need policing. Facebook needs policing anyway.
The other side is that we’re learning things and changing assumptions
we had in the past. So Accenture worked with Kellogg’s, I think on
product placement and they did this. They tracked people’s eyes and
how their head moves, and they found that like everything they
thought about where to put a cereal box in a grocery store was wrong.
Alan: Wow.
Emily: So I think it opened up
these great opportunities to get more in touch with human behavior.
But it’s also creepy.
Alan: [laughs] It definitely has
this Orwellian feel to it that. Wait a second. We’ve already given
all of our data to Google. Let’s be honest. I use Drive, I use Gmail.
So they know everything about my buying habits. They know everything,
between Google and Amazon they probably know everything about me. We
don’t even go shopping any more, things just come to the house. But
they still don’t know about my personal life. Well, I guess we have a
Google Home and an Alexa in our house, so I’m sure they do know about
our personal life. The question becomes, do we trust them? And so far
we do, Facebook being the one. They just got slapped with a
$5-billion fine because of privacy violations.
Emily: Yeah.
Alan: So I think governments
really need to step in, especially in the age of AI when you can–
it’s one thing to know this information about an individual. It’s
another thing to act on it, and to be able to take that information
and make it relevant. Right now, I think we’re still in that phase
of, “We collect all this information, shit tons of it. But to be
honest, we can’t use half of it, because we don’t even know how to
process it.” So while we’re collecting data, there’s huge
amounts of data that we don’t use. Companies– the Age of Big Data, a
few years ago, “Oh, big data, we’ve got to collect everything.”
And then they realize, “Oh my God, we collected all this data.
We don’t know what to do with it.” So I think AI will solve that
problem, but also create some real privacy and security issues.
Emily: It’s also frightening to
think about the fact that all this data we’re collecting, if we run
it through AI, they could be used to make major decisions that affect
our lives. And we don’t really understand the data right now. So I
actually just read a really interesting book called “Invisible
Woman: Data Bias In a World Designed For Men”, I think. And one
of the examples was Amazon traded this AI tool for hiring. The
information that they put into the system was the last 10 years of
resumes that were submitted to them. This was for a technical
position like a developer or something. What ended up happening was
the system was biased against women because–
Alan: The system was.
Emily: Yeah. Because the tech
world is notoriously male dominated. And so if you’re looking at the
last 10 years, most of the resumes are going to be from that. So they
had to shut it down. So there’s things like that, that are a little
frightening to me. Policing is one thing, like policing our privacy
and getting government involved. Hacking is crazy. That scares me,
that’s what keeps me up at night. Our information is constantly being
stolen and preventing the use of data for the wrong unintended
reasons. So understanding the data is a big part of this.
Alan: Yeah, no kidding. Wow. We
could discuss this forever, I’m sure. I think we both realize that
the potential for unlocking humanity’s potential is unlimited, but
the potential for it to fall into the wrong hands — not even the
wrong hands but the wrong actors within certain subsets of brands and
companies — really becomes a challenge. And we also have the
unintended consequences of depression and antisocial behaviors and
stuff like this. And there are unintended consequences of an always
on computing platform that is glued to our face.
Emily: And that’s really dumb. I
mean, there aren’t studies yet.
Alan: No, we don’t know.
Emily: It’ll take a generation
for us to get really meaningful insight into some of those.
Alan: We’ve only had smartphones
for 11 years.
Emily: Yeah.
Alan: We are already realizing,
“Oh shit, this is not good for us.” We need to take a break
and turn off our notifications. I actually wrote an article, “11
Ways to Reduce Smartphone Related Stress” because I was
researching for myself and my kids and my wife were, you know, how do
we cut back on looking at the phone all the time? And my biggest
thing was turning off all the notifications and sounds. I just turned
off, blanketed everything. And when you have glasses in the future,
we’re gonna be wearing glasses where the whole world is our computer.
How do we select for what we want to see, what we don’t want to see
and when? So I think there’s a huge road, and I’m kind of glad
enterprise is leading this versus diving right into the consumer
market.
Emily: Yeah, I agree. And it’s
for that kind of reason. In a way, it’s almost like a controlled
environment. You’re not just releasing AR/VR into the wild.
Enterprises are actually finding ways to use it. They’re working with
IT to secure the information, even things like hygiene, passing a
device from one worker to another. Those are things that are gonna be
worked out, thankfully, in enterprise first.
Alan: Yeah, I mean, we’ve been
doing demos since 2015 and we’ve done probably 500 events, so
thousands and thousands of people putting on these headsets and we
started off with replaceable covers and all this stuff. And we
finally got to the point where for VR and AR we use these VR covers
that have like a leatherette and then we just wipe them with alcohol.
And I mean it has a little bit of a smell to it, but at least you
know it’s clean. There’s nothing growing on that thing. But yeah.
These are all really interesting challenges that– the big one that
was just evident, or became evident is the security of the platforms
is non-existent. I mean, three of the major collaboration platforms
got hacked a couple of weeks ago.
Emily: Yeah, I read that.
Alan: We need to figure these
things out and they’re going to figure it out. I mean, this is what
technology does. We find a problem, we solve it. So I’m really
excited about it. Yeah. We could go on forever, but we’re we’re
running out of time. So I want to ask you one last question. And
first of all, thank you so much for being on the show, Emily. It’s
been an amazing conversation. What is one problem in the world that
you want to see solved using XR technologies?
Emily: Good question. I want to
see XR help women in the workforce. So if XR is the future of
training, it’s the future of learning, it’s potentially the future of
our education system. And it has this great potential to democratize
information and skills. I would hope it could be leveraged to address
inequalities in the workplace. At the same time, I want to see XR
companies pay more attention to the user experience for women. I
personally find a lot of VR devices uncomfortable. I’m not alone,
I’ve spoken with many women who work in AR/VR. This is the future of
training and I believe it is. And you believe it is. So user
experience for women cannot be inferior to that of men. Again, we’re
talking about wearable technology. It’s incredibly intimate. And
optics. Men and women don’t perceive depth the same way. So these
kinds of things I’d like to see XR companies pay attention to and
physiological differences, different ways that we perceive depth.
That’s important and I haven’t seen that brought in yet.
Alan: It’s an interesting point.
I believe there’s a reason behind that. Most of the technology
hardware is designed by men and I don’t know why that is. It just
doesn’t seem like– it almost seems like they need to hire female
designers to finish the product, like you got the product to working,
it’s good. OK. Now let’s have somebody with an eye for design and an
eye for comfort across both sexes and all sizes. It’s a hard problem
to solve when you have 95 percent of the people working on the
problem are men. It’s an issue, for all the women listening, if you
want to get an understanding of what this industry is like, go to CES
in January and you’ll be in a literal sea of men. It’s kind of
ridiculous actually, but I think it’s getting better, and more and
more women are warming up to tech and it’s slow, but I think we can
get there.
Emily: Yeah, there should be a
woman in the room whenever any design decision is made.
Alan: There should be a woman on
the board of every company.
Emily: Yeah.
Alan: One, minimum. And if we
did that we would– and, we should only have women world leaders,
because that would eliminate a lot of the ego and bullshit and war.
Emily: I would love that.
Alan: Wouldn’t that be great, if
we just made women the leaders of the world? There would be no war.
We would cancel all military actions and all military spending and
apply that to education and food. And here’s to hoping that happens
in the future.
Emily: Definitely.
46 min
September 27, 2019 HR in XR, with BrainXchange's Emily Friedman
As the lead
writer and head of content at BrainXchange, Emily Friedman has had
ample chances to explore a lot of XR-related topics. She lets Alan
pick her brain about a few of them, from getting millennials
interested in trades, to democratizing knowledge, and how humanity
will enter The Cloud.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today's guest is
Emily Friedman from BrainXchange and Augmented World Expo. Emily
Friedman is a New York based enterprise immersive, wearable and
emerging technology advocate, journalist and facilitator. She's Head
of Content and the lead writer at BrainXchange, lead journalist and
senior editor at Enterprisewear Blog, and head of marketing and
communications for Augmented World Expo USA and AWE EU. To learn more
about BrainXchange, you can visit brainxchange.com. And if you wanna
learn more about AWE or Augmented World Expo, you can visit
awexr.com.
Welcome to the show, Emily.
Emily: Thank you for having me.
Alan: Oh, it's my absolute
pleasure. I've been really looking forward to this conversation,
because you are writing everyday – or, not everyday, but what, a
couple times a week? -- on the enterprise wearables world. So maybe
just kind of give us an overview of what is BrainXchange and AWE.
Let's start with that.
Emily: Ok, I wish I were
productive enough to write multiple articles a week. But there's a
lot going on. BrainXchange, we started out as a boutique events
company, and we just happened to enter augmented reality at the right
time. It was 2015, right after Google Glass, quote/unquote failed.
And there were all these headlines, “Glasshole” articles. But if
you read between the lines, it was clear that smartglasses weren't a
failure, and that enterprises were actually finding good use cases
for it. So today we provide events, content, and other services all
related to facilitating enterprise XR.
Alan: You know, I've been at AWE
a couple of times now. I lead the startup track this year. It's an
important conference for virtual/augmented/mixed reality and some may
say it is the most important conference. It's where everybody around
the world gathers in. And I made this comment that if the building
happened to collapse, basically the entire VR world would cease to
exist, and we'd have to start over again. It was an amazing
collection of some of the world's smartest people working in this
technology and enterprise. They seem to be really driving this
technology forward. What are you seeing?
Emily: Well, as for AWE, I think
it's a very important benchmarking event. Like you said, the entire
industry gets together at that one point. What we're seeing -- and
the reason we gravitated towards enterprise at first -- is that
that's where the money is. I mean, that's where the money has to be
made, both for end users and the AR/VR companies themselves. At the
end of the day, we cater to the enterprises and we talk to them every
day. We get on the phone with Fortune 500 companies, the innovation
people and all these different companies every day. And we listen to
their pain points. AR/VR happens to offer a solution to a lot of
their pain points.
Alan: So what are some of the
pain points? Let's unpack that.
Emily: Huge one is a shrinking
workforce, that creates this need to train faster, better. So as the
workforce ages -- in manufacturing, I think the average age is like
40 to 50 now -- and retires, not only do you need to attract new
talent; you need to train them. As a millennial, this is actually
pretty important to me. Learning a skill today just doesn't get you
as far as it did half a century ago. Tech advances, business models
change, and much of what I learned in school, I feel like it's
irrelevant. And for Gen Z, i
0 min
September 25, 2019 Building the AR OS for Enterprise, with RE’FLEKT’s Dirk Schart
Most businesses
have the information and infrastructures they need to be more
efficient and competitive — it’s just a matter of having it all at
their fingertips. RE’FLEKT is working at making that process easier
by creating a modular, scalable, open-source operating system for
businesses to build their own in-house AR applications on top of.
President and CMO Dirk Schart drops in to explain how.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today’s guest is Dirk
Schart RE’FLEKT President and CMO. Dirk Schart is a marketing and
growth driven tech executive, whose current mission is to build the
operating system for enterprise augmented reality. Dirk leads the US
operations for RE’FLEKT. Dirk is one of the first employees of
RE’FLEKT and is a funded company by global companies such as Bosch
and BASF, and today plays a leading role in enterprise AR.
Previously, Dirk built the Digital Innovation Lab for Hyperloop
Transportation Technologies, for which he continues to work as an
advisor. Within four months, he built a team and developed the first
MVP, and he also directs the VR and AR team for Hyperloop
Transportation Technologies. Dirk helps technology startups such as
VisualX, VYON, and RiseStep. He’s the author of two augmented reality
books, albeit they’re both in German, but you can get them. And he
also contributed to Metaverse, the book by Forbes writer Charlie
Fink. He founded the first AR blog in Germany, “WE ARE AR”
and has been interviewed and quoted by leading media such as Tech
Crunch, Venture Beat and Huffington Post. For more information on
RE’FLEKT, you can visit re-flekt.com.
Dirk, welcome to the show.
Dirk: Hey, Alan. Thank you very
much for having me.
Alan: My pleasure, my friend.
The last time I saw you was at Augmented World Expo, and I think we
were taking silly photos.
Dirk: That’s true. But was fun.
[laughs]
Alan: It’s always fun. And it’s
becoming like a family, this whole augmented reality family and
everybody’s working together. And it’s been wonderful. And the work
that you guys are doing at RE’FLEKT is world class. And I really want
to dive into the benefits of this technology and how you guys are
using it. Let’s talk about that.
Dirk: Absolutely. So you already
mentioned it in the intro; what we’re doing is we’re building the
operating system for enterprise AR. So what does that mean? That’s
for us the foundation of how enterprises will use AR in the future.
Let me go back quickly and talk a little bit about how we started.
That makes it easier for the listeners. Back in 2012, it was really,
really difficult to create any kind of augmented reality
applications. So it was completely different to what we have today.
And we started to develop individually programmed applications. There
were no platforms. There was nothing at the time. And one of the
first things we did is, we build an application for Range Rover. And
the reason for that was that they came and said, look, we have a very
complex repair. It’s about a few return line of a Range Rover car.
And the mechanics, they always have problems doing that because it’s
complex. They don’t do that every day. Documentation doesn’t seem to
be perfect for it for that use case. They try PDF, they tried videos,
but result was the same, they couldn’t fix the problem. So we built
an application.
We showed it to them and helped the
mechanic, because it guided them step by step through that repair and
showed them exactly at the point where they had the problems. “Wait.
Now focus on it. Have a look. Double check before you do the next
step.” And that at the end helped them to perform the task much
better and reduce their error rates. And then, of course they asked
about, “okay, so how can we do that for all our car models?”
And at the time — that was at 2013, 2014 — there was no scalable
way to do that. And that was for us the starting point to build our
platform, which is called RE’FLEKT 1, which allows enterprises to use
your existing content.
We’re talking about everything
enterprise have already, because it’s existing. They have it in your
documentation, in a technical documentation for owner’s manual to
repair manuals, maintenance manuals, all that stuff. It is somewhere
in the cloud or on a server, but not used for AR. So we enabled that
the enterprises to reuse that with a platform. And like that, we
started and we rolled that out. We added remote support and things
like that. And over the years we realized, okay, now that that’s
working even if there are a lot of challenges and you know that,
we’re not there to say it like that. In a way they say, “well AR
is kicking off in a way that everybody’s using it with the CRM.”
But the next step for us, and now coming back to the initial point
about “what does it mean to build the operating system for
enterprise AR?” is, to have a platform is one thing. But there
are many vertical specific requirements today. Whether you go into
medical or pharma, where you have regulation, or you going to
automotive or aerospace. The scenarios are completely different, and
the requirements are different. And we as an AR startup — and that’s
the same for other AR startups — we cannot solve all of these
problems. We are technology companies. So this is why we said we take
our platform, we make it modular, we add SDK and API to make it open,
and building that operating system so that partners, system
integrators, solution providers, but also enterprises on their own
can build solutions they need. And this is exactly what companies ask
us for: to have a solution where they can customize their stuff,
where they can integrate into their solutions. The funny thing is we
always think as AR company and you mentioned it, it’s a community and
it’s cool. We are all so cool in AR, but in enterprise business,
they’re not going to change their infrastructure and their
architecture and the systems because of AR. It has to happen the
other way around. And that’s why we’re doing what we’re doing.
Alan: What do you mean by that?
“They’re not going to change their infrastructures is off
because of AR. It’s the other way around?”
Dirk: What we tried is to offer
our solutions and then go into the enterprise and say, “Well,
look, here it is. You have an AR solution and now you need to
integrate into that. You need to provide your data and all of that.”
But enterprises have existing infrastructures and architectures. They
have a CRM, they have an authoring system, they have a PLM system,
they have a CAD tool. They have all of that. So AR is just one little
thing, and AR has to integrate into those kind of systems, not the
other way around. No company is going to change a lot of their
infrastructure or their CRM or their CAD tools because of AR, that’s
not going to happen. And that’s exactly why we deliver it as an
operating system.
And now we allow and say, okay, for
instance, we connect to Siemens PLM Team Center. So that means a user
who already works in something like Team Center does not need to
leave that environment. Our platform kind of runs in the background.
It is used for publishing, for helping with the technical stuff,
making that easier, like building a tracking configuration. All of
those things are integrated into our platform, but the author can be
at the used environment which he or she already knows. So it’s not
about learning all of that new. It’s just to have kind of an AR layer
on top of it. And that makes it much easier. And that’s — for me and
that’s for us — the key to make AR scalable and also to have finally
acceptance on a higher level. Because the technology is available,
it’s not a problem at all. We can realize all of those things the
enterprises need. It’s much more about the acceptance among IT people
and also among the frontline workers.
Alan: Interesting. Yeah,
somebody mentioned on one of the other podcasts that we don’t have a
technology problem anymore. We have an adoption problem.
Dirk: Right. That’s how it is. I
fully agree on that. Well, look, as I said, we can realize it. I
mean, give you an example. We had one little gap in the publication
process. You could do everything. You could upload your data, you
could add your AR content. And then at the end you could publish. The
only thing you could not do was to build your own tracking
configurations. So it means you need to tell your AR system and your
camera and your algorithm when it sees a machine. What does the
system have to place on top of the machine? What is said to be
overlaid here? And that’s called a tracking configuration. That had
to be done by an engineer or by a software developer. We closed that
gap; we created a tool. And with that tool, everyone can do that,
without any kind of coding skills. You just use your existing data.
You compare it where now expressing a very simple way and that’s it.
So it closes the gap.
That supports what you just said. We
don’t have a technology problem. We can fix all of that. Now also
with machine learning on top of it, which improves the tracking and
the usability of the whole system. But don’t forget that, especially
when we talk about the frontline workers, the actual users, then we
have to talk about change. And you can just go and say, “Well,
look, you don’t use your hammer anymore, now we give you an iPad and
that’s it.” That does not happen from today to tomorrow. We have
a new report we worked on, why so many POCs or trials get stuck in
that early process and do not proceed to a deployment, to a rollout,
what we all want to have in the AR community. And it’s about change,
it has nothing to do with the technology, maybe with the usability.
But at the end it’s change, and you need to tell the people why they
should use it, what is the advantage, and how can they use it?
Alan: All right. So let’s unpack
that. Why should customers use this? How do they use it? And then
what are they going to get out of it?
Dirk: Talking about the why,
they’re the obvious things, which we repeat over and over. Talking
about enterprise use cases, you can reduce your downtime, you can
improve your first time fix rate, reduce your error rates. There’s so
much more than only that. I’m thinking about that you can really
improve your customer experience. You add a lot of value across the
product lifecycle, things you could not do before. And I said when I
explained that example, when we did it for Range Rover, that that
showed it clearly. Take it from the consumer side: you buy a new
product — let’s take a coffee machine. So you unpack that. What do
you want to do? You just want to use it. You don’t want to flip
through pages and read how that works. And it’s an awkward
experience. It takes you forever until you get that coffee, finally.
Alan: Don’t mess with people’s
coffee.
Dirk: Yeah. [laughs] My coffee
machine did not work in the morning. I’m lost without my coffee! But
nobody wants to read the printed manuals anymore. If you find them,
then they’re outdated. It takes a lot of time to understand what’s
written there. You don’t want to have that anymore. And let me add
that to the why. I think it has a lot to do also with what we know
from consumer side. Even that many people say, “Oh, well, you
know, but it’s enterprise, and enterprise is okay to be boring.”
No, it’s not. We are used to Netflix. We’re used to iPad. That kind
of experience. And then we go to work, and then we have to use the
old tools which do not provide anymore what we need.
Alan: It’s crazy, right? We have
these beautiful UX/UI for everything in the consumer world. And then
you go into these enterprises, and it looks like Windows 91.
Dirk: [laughs] Exactly! And
that’s the problem. People don’t want to have that anymore. You come
from home and you go to work and you have to go into another world.
And that’s not acceptable anymore. And that’s what we see, that
transformation happens. And also we think about the next generations.
If I see my son — and he’s eight years old — and I showed him the
computer and I gave him a computer mouse and he looked at me and
said, “Dad, what’s that?” And I said, “that’s a
computer mouse, for your cursor here.” And he just left. And I
said, “What?” He said, “No, I don’t want to have that.
I will use my iPad.” That’s a change in behavior. And that’s why
I say we need to bring the consumer grade experience to the
enterprise sector. And AR helps us a lot to do that. Now coming to
the how, well, that’s a bigger part. I mean, we could do several
episodes here about how. But let me quickly given all of you about
that. I think it’s a lot about how you can get started. In the past–
the last years, many enterprises kind of wanted to get started with a
huge thing. Everybody was thinking about that rollout. How do you
decide that if you were a manager in a large enterprise, are you
going to decide and say, “I will invest 500k into a new
solution, where the ROI is not proven or I don’t know what’s going to
happen exactly. I see there is a potential, but I don’t know exactly
how it’s going to work. And I have to report that.” Nobody’s
going to make that decision. And that’s, I think, one of the problems
we had in the past.
Alan: I always tell the story
how when we started meeting with customers, when we started, they’d
go, “Great! This is amazing technology, I love it, it’s really
mind-blowing! Who else is doing it? What are the ROIs and how much do
they cost?” And we would say “Nobody else; we have no idea;
and a lot. Still want to buy it?”
Dirk: [laughs] Yes, that’s
reality. Not augmented; only reality. That’s exactly how it is,
right? And those are the questions that we have every day. It’s not
enough anymore to have some interesting logos on your slides and say,
I work with ABC. No, you have to prove it. How many users do you
have? How did they do the rollout, as you said it? What is important
is you need to do it in steps. You need to start with that with a
trial. Go do a POC–
Alan: Ok, so what are the steps?
I’m a new customer. I own a factory. You guys come in. What’s step
number one?
Dirk: So step number one is, for
us, we start with a little package and do a usually a scoping
workshop, where really to check what are the requirements? Is there a
company who just wants to test something? Is there a company who’s
already thinking about a bigger solution? And then specifically, what
is the use case? I really want to see a use case when we get started.
It does not need to be any complex thing, but just one thing. And
then in the workshop, we define “Okay, that’s the use case.
That’s the way we do it today. And then we have KPIs. And that’s the
way we’re going to do it with AR.” And then we can measure.
That’s the key. So and that gives the enterprise a small package
where they can define what they do together and with the front line
workers that’s important. It needs to go bottom up, not only defined
by the management, because that will be a failure. And then you have
that scoping and then you have the measurement. You can say, “Okay,
this is how we did it before, and this is how you do it with AR. And
now we can see that’s successful. Does it make sense in that use case
[garbled]
?” So once you have that, you go to the next step and
you go into a POC where you can really test that internally, within
several machines or several products and really start to scale it.
Not only in a test environment, but in a real world scenario, which
makes it different or even involve your customers and get the
feedback there. And then that’s the second step. And then to come to
the third step where you finally roll it out. Take all the experience
you have from step one and step two, take that into consideration.
And then finally, you’re able to roll it out. Because between 1 and 3
dictate a lot of different steps.
Step 1, it’s more about figuring out
how it works, maybe already involving the IT a little bit and say,
“Okay, so what kind of connections do we have here? How do you
install all of that?” And step 2 is then completely focused on
the business case itself. “Who’s going to use that? How is that
compared to the current solution? What is the customer or the user
feedback?” And step 3. It’s a tough part to roll it out. It’s
about security. It’s about integration. It’s about all of those
things. And that’s a completely different story. I mean, it takes you
a month to really get the security done and get approved as a vendor
with a large enterprise to fulfill all the requirements.
Alan: So this scoping workshop,
POC rollout, is this a 12-month rolling timeline? Or longer, or
shorter?
Dirk: It depends a little bit on
the enterprise side. There are some there, they’re faster. But there
is some that need more time. So bringing that into a timeline that
the scoping workshop and the results out of that. That’s something we
do. Let me say maximum 60 days. That’s the timeframe. What we also do
is, we do the scoping but we also give them the product to work on
that, to get a feeling. What can they do? How can they create content
on their own or with our support? So we really train them, we have
our own education program where we train them. So our promise is up
to 60 days. A company knows how to create an application on their own
and they create the first application on their own. And then, for the
POC, you can add another 90 days, because that has to be prepared. So
60 or 90 days for the first two parts. And then rollout, I would
definitely plan a six month time period. That’s what you definitely
need to prepare all of that. It’s a lot of communication and a lot of
things to check internally. Month in, you can finally roll it out.
And even then, you probably do it in a few countries or with a few
users before you do that globally. It’s a big thing, to do that you
have a lot of different requirements. Might think about things like
GDPR we have in Europe, which you do not have. And the same in the
States or in Asia. Think about Asia and China. You cannot just go
there and host a server. It’s just not possible. We tried that and
luckily we have a partner like Bosch, our investor and partner,
Bosch, and they helped us with lots of stuff there, because you
cannot just go there and say, “OK, here we are. No, let’s get
started.”.
Alan: I guess not.
Dirk: No, it’s not. It’s not
that easy. It’s possible. And the market is growing fast.
Alan: What are the KPIs? What
are the measurements that we’re going to measure? What are some of
the metrics that that you would test against them? And is there a
cost to phase 1? What are the costs associated with phase 1, phase 2,
phase 3?
Dirk: Yeah. With phase 1, it’s
usually a wrench between, let me say, $10k and $20k. POC and rollout
is a little bit harder to say. POC depends on what exactly is the
scope of it. Let’s say there is between 20 and 50. But in rollout,
it’s hard to put in numbers, because that’s very specific and depends
on what kind of solution you want to have. But of course, then it
goes into licensing and subscription. That’s tough, but it’s also a
50-50 plus then, to have that in numbers. So KPIs you asked for. So
that’s something that– the interesting thing is the enterprises that
they know their KPIs, they know exactly what they have and what they
need. The typical things, what you have is error rates, first time
fix rates, down times, when you think about the manufacturing sector
especially. That’s a very clear KPI. But of course, there are many
more if you look into the training sector. How do you measure
knowledge obtainment? How can you measure knowledge transfer? When we
think about the skill gap and the workers gap in the manufacturing
sector. And they have the big issue that they need to transfer the
knowledge from the people getting retired to the young guys.
Sometimes you have really hard numbers. Like what I said before, they
reduce time to downtime or the error rate. And sometimes it’s a
little bit more difficult to say, “OK, how can we measure things
like knowledge transfer?” and we have to define that together
with the customer. It’s very important to find a base, to compare
that. We had a customer and they did it a completely different way.
We asked, “So how did you measure the success?” and the guy
who led the project, he said, “Well, it was simple. I asked my
team what they think, and the team said we’re super happy, and it’s
better to work than before.” And I said, “Okay, I
understand. But what are the metrics behind that?” And he said,
“No, there are no other metrics. It’s just the team is happy and
works better than before.” I said, “OK, that’s an
interesting way to define a KPI.”
Alan: [laughs]
Dirk: Well, this shows you we’re
still early stages, right? We’re building KPI frameworks for our
customers. But it’s not that you have one framework which you can use
for all kind of customers. It’s just [garbled] word right now. I had
a meeting last week with a large enterprise in Florida and for them,
other things were super important. For them, it was important how can
we automate the content creation? How can we customize the UI, that
it fits to what we have already? We want to integrate that. We don’t
want to have that our service technicians have to get used to a new
UI or something new. So that was for them much more important than
other matters. They said, “Well look, that’s what’s for us now
super important, because that helps us to integrate the system and to
show the employees and the workforce, well look, that’s a good tool,
use it. And then we will look for the other KPIs once they’re using
that.” And I like that approach because as I said, it’s about
change. So the first step, you have to make sure that people want to
use it. What do we have right now in 70 percent of the cases, we
create a tool which is very helpful, but maybe workers or technicians
use it once a week, but they don’t have that frequency that they use
it every day. So it does not have the value it’s supposed to have. It
really depends on what is your goal. And then you can define your
KPIs.
Alan: Now, you’ve run a number
of these in this protocol. So this 1-, 2-, 3-step process — scoping
workshop, POC, and then roll out. How many of your customers are in
the rollout phase now? Or maybe already rolled out?
Dirk: Several are in the rollout
phase. The interesting thing is — and that’s also the sad thing for
us enterprise AR guys — is that were not allowed to talk about all
that stuff.
Alan: [laughs] I figured that.
Dirk: That that’s one of the
most difficult things. But we’ve already rolled out. So like the
biggest rollout we had was an application, which is called Daimler
Rescue Assist. And I think it’s probably one of the biggest rollouts
in AR history, because it is available in 27 languages and for all
Mercedes cars since 1990.
Just to give the background quickly,
why it is so big: it’s an application for first responders. And when
they come to an accident, they have to figure out where are the
critical parts like battery wires, fuel tank, things like that. And
they have to make a decision, in seconds, where they cut into the car
if there’s someone inside. And before, they had to do that with PDF
instructions, and you can imagine that’s quite tough if you have to
compare that. You have deformation in the car. Then you have to cut
into that car. That’s not really a good decision-making basis. That’s
the reason why we have that application. And Mercedes and Daimler,
they wrote that out globally, together with our partner Bosch. So
that’s one of the big examples. And that shows also it’s possible to
do that. And the others, we have currently between five to ten
rollouts in preparation, communicate more about that soon. There are
different companies from the chemical sector, but also from the
energy sector. And what you can see is, it’s completely different.
There is not one rollout. I mean, you have some things that always
happen, it’s like security as the major topic–
Alan: What are some of the
things that are always consistent? From what I’m hearing, you’ve got
one major rollout, a couple more coming. So, we really still are in
the earlier stages of this technology. And again, it’s an adoption
issue, not really a technology issue. But that is only recent that
the technology has caught up to our visions.
Dirk: Yeah, absolutely. I mean,
I think it’s a lot about once we have seen the first five, 10, 15 —
that will be the trigger. I think many enterprises are waiting,
they’re ready to go, but they’re waiting to see others. And then they
will quickly turn into the–
Alan: Fast follow.
Dirk: Yeah, exactly.
Alan: I live in Canada; we’re a
country of fast follow. We watch what America does, and we wait for
them to make their mistakes, and then once it’s proven, we jump on.
Dirk: That’s like the Apple
strategy, right? [laughs]
Alan: Exactly. Never be the
first; always be the best.
Dirk: Exactly. But that’s what
we see. They want to seed, they invest, but they don’t want to make
the decision about the really big deal. And then on the other hand,
also, it is a big deal to do a rollout like that. And if you have
like several thousand of users involved there, a lot of questions you
have to answer. And that starts. You have to involve like branding,
who is going to host your app? Who’s going to do that? What our
security requirements and all of those things. Where are we going to
start? Which country is the first one? What kind of devices do we
have? And don’t forget about by large enterprises.
We often have old operating systems,
whether it is about Windows or even tablets and stuff like that. So
you cannot really compare it. There are some basic patterns. Security
is one. I would say less like 70 percent of the securities that
that’s basically the same large enterprises. Then there are some
specific rules. But basically that’s the same. So what you have done
that. And you know how to handle that and you have the right tools
for it, then that becomes easier. But the other things that’s really
different because organizations are different.
Alan: So what are some of the
considerations? Let’s just run through a list and we’ll just compile
a big list and we’ll put it in the show notes for people so they can
at least have a list of considerations. Right now I’ve got security,
branding, hosting location devices, device management, integration.
Dirk: Right. Countries. That’s
something we have to check because there are different devices we
already mentioned. Those are the well precise and the league terms in
terms and conditions. So that part is very, very important. How do
you do that? And then it depends also on is that an internal pronoun
or John Roland where he goes to the customers? That’s also completely
different. That’s what we what we have right now, where the customers
are invoked. Then you need to think about, okay, how do we provide
access to customers? What kind of systems do they use? So it becomes
kind of a narrow level of complexity on top of that. But you learn a
lot in different units, have different requirements. And then I think
one of the biggest parts when you when you get started with that is
to find the right people for it. You really need the right people in
these large enterprises who know how it works. That’s for me, always
a major point. We come from our AR world and we think we know
everything and then we come to an enterprise, and the first thing we
have to do is we have to listen. Because we have to learn about, how
does that enterprise work? What are the structures? How is the
organization organized, and what were the specific things they can
tell you? There are so many different things that they have to look
for. And then it’s about customization. If you can, define steps for
the rollout. But that does not mean that it really fits for all
enterprises. You need to have an individual plan for an enterprise to
do a rollout, that it fits. Otherwise, it’s not going to work.
Alan: All right. So what are
some of the challenges that you guys have seen that you’ve had to
overcome, over the time that now, is just part of your process? What
are some of those challenges that people getting into this [can
expect], that seem to be general challenges?
Dirk: So I think one of the
biggest challenges is what I mentioned earlier: the way we get
started. Instead of saying, “well, look, here is a license,
here’s the product. You get a training: now get started. Create
applications and roll them out.” That’s what we thought we can
do. Just more or less, you sell a CRM and enterprises install it, and
they get started. But we’re not selling a CRM. We’re far away from
selling CRM. When you buy a CRM, it’s more about, “which one do
I need,” but not whether you need one. With AR, it’s about, “doI
need AR: yes or no?” It’s not about which kind of solution, the
first step. And most of the enterprise, they even don’t think about
it. And most of the enterprises I see, they even don’t know about the
different solutions or vendors. That was one of the biggest
challenges for us to learn and understand. What is the right model to
get an enterprise started? And large enterprises, mid-market,
industrial market — it’s just the same. There’s no difference from
that point of view. So that’s why we came up with the different
stages, right? Scoping workshop, POC, and then rollout. That helped
us a lot to learn together with the enterprises, and take the
learnings, improve it, and then do the next step. Enterprises are not
ready to buy a license right away and get started with it. There are
[some]
, of course, but that’s not the typical way to do that.
Alan: So, you’re a platform.
Dirk: Right.
Alan: You’re also a content
provider, and you’re also a service provider. You’re kind of having
to do everything, is that what I’m hearing?
Dirk: Yes.
Alan: That will change as this
technology becomes more adopted, and people can start to realize the
benefits of it. But I think where we are right now in this
technology, from what I’m hearing from you, is that we’re still so
early that we still need to handhold every customer through the
process of deciding that, “this is something we want to do,”
and then trying it. “Yep, it does do what we say it’s going to
do,” and then, “okay, it’s time to roll it out.” But
the companies are still ill-equipped to do this themselves. Selling
just the CRM system is not possible just yet, but it will be in the
future, as more and more case studies come out and people are able to
say, “okay, I want that,” right?
Dirk: Exactly. That’s what you
describe is reality today. The interesting thing is that applies for
all our startups. If you have investors, and investors — of course
— look for recurring revenue. So, you shouldn’t say, “okay, we
will sell licenses and nothing else.” But in these — and that’s
not only for AR; I see exactly the same in IT, in block chain, in AI
— we are not at a point I can just sell the licenses and the
customer takes it and does whatever. We need to have the education
and the service part. There are many, many enterprises. They ask me,
“what kind of person is that, who would create the content? What
kind of skill set do they have need to have? Whom do I need to hire?”
That was, for us, the trigger, and we started now with our own
education program, where you can get your own modules, and companies
can book that, and they can train and educate their own people to do
that, [and ensure] that they have also the same level of education
and same level of knowledge. We need the services until, as you said,
we reach a point that everyone knows. But even look at the big
systems. If you look at SAP, or even a CRM — take salesforce,
something like that — you need a lot of customization. You need a
lot of service to implement that in the right way. And AR is no
different.
Alan: It’s interesting you
mentioned that, because this is one of the reasons we started XR
Ignite, of which you’re a mentor, as well. XR Ignite was started as a
community hub to foster the relationship between startup studios and
developers, and corporate clients. Because what we realized — same
thing you just mentioned — is that the content creation is a
separate thing from the platforms, and venture capital invests
typically only in products and platforms. But what they’re missing is
this sweet spot of content creation. And one of the things that we
realize is that independent developers and small studios, they’re
going to start to get acquired, because companies — exactly what you
said earlier — they’re going to say, “who do I need on my team
to build this content? Oh, I need 10 people with Unity skills. Well,
I don’t have time to hire 10 people. Why don’t I just acquire this
little studio here that can do what we need, or work with them for
six months? Test them and then acquire them.” I feel that
there’s going to be a lot of these small micro-acquisitions of $5-,
$10-, $20-million acquisitions, where teams are just basically
acquiring talent.
Dirk: Absolutely. And you can
see that already, that companies are thinking about that. How can
they solve that? Do they have own teams mostly? No, they don’t have.
They have like one or two–
Alan: No, and it is going to be
hard, because right now, there is a shortage of this talent. And if
you project out two years, when all of the companies all of a sudden
wake up, the talent pool is not going to be much bigger, because
schools and universities and colleges are not teaching it yet. And I
think we’re going to find this issue right across the board with
exponential technologies, that our current school systems are
ill-equipped to retrain people. How do we train people fast enough to
become very, very proficient in creating this technology, so that
companies can start to either acquire studios or startups and really
build them in, or hire them?
Dirk: Absolutely. And that’s
what we need. And it’s different often, right? I mean, content
creation is something… especially when it comes, now, to the two
big things we do: repair and maintenance in areas, in training
environments, and in the manufacturing or medical sector. You take
your CAD — people know how to do that part — and then you add your
AR layer on top of that. But then it already starts, and is okay, how
can I do that? But I think is there is a shift from 2D people to 3D
people today. Content creators, developers. I used to build 2D
interfaces. What we have in our mobile phones; tablets. But now,
handling that in the 3D space and make that a great user experience
and guide the people and help them to use it? That’s a completely
different story. And we don’t have that. There are almost no people
— think about UX designers — being able to think in a 3D space in
that way, especially in the enterprise sector. There may be in the
consumer area, but in the enterprise sector, there is a lack of
people.
Alan: I agree, and I would
punctuate that with the fact that ARKit — which is Apple’s framework
for developing AR — and ARCore — which is Google’s — majority of
the really interesting, robust content that’s coming out of ARKit and
ARCore is consumer-driven. It’s games, it’s experiences. Even if it’s
a business application, it’s usually some sort of marketing. These
applications — while also very, very valid — designing enterprise
AR is not as sexy as designing a video game. It’s more lucrative, but
less sexy. So you’ve kind of got these game developers over here that
are making amazing games in three dimensions. But there’s a skills
gap in between the enterprise and the gaming world.
Dirk: Yeah, you need to know how
enterprises work. You need to see how one of these frontline workers
is using the tools. And that brings me back to how we started with
AR. I remember some of the apps, where we showed a car mechanic how
to change a tire. I mean, that’s not going to work. We need to see,
where does AR make sense? Where does it provide value? And that’s
something that you have to combine that with a good user experience,
because it has to be interesting. It has to be also fun, even though
it sounds weird for an enterprise sector. But it is like that, coming
back to what we what we discussed earlier; you need to have that
Netflix and Apple experience to have a good work result. And that’s
completely different to what you do for games. You can learn from
games, right? Not to underestimate that. You can learn a lot from
games: how to keep people interested, how to keep people at a level,
how to provide information and transfer information. There are so
many different things. See a firefighter: how they hold the tablet
and use it with their thumbs. Maybe they have gloves even. Then go to
a car mechanic. This is a completely different story, and that’s what
you have to understand, too.
Alan: Oh, man, there’s some so
much here. So let me ask you, on a personal note: what is one problem
in the world that you want to see solved using XR technologies?
Dirk: Well, that’s a great
question. There are a lot of things I see, but I think for me, even
that it sounds a little bit global now. What I want to have is I want
to enhance my work environment. I don’t want to be tied to little 2D
screen. That’s where I see the biggest impact of VR. Give me my 3D
work space, where I can define my environment and I can take it with
me. And I always have my office with me wherever I am without having
a little 2D screen.
Alan: And then your office does
not need to be limited to a desk with four walls anymore. You could
create an an an environment… like, I like working on the beach in
Bali. So, every time I put my headset on, I’m in Bali on a beach,
working on a beautiful desk with 100-inch screens around me. The one
thing that we don’t have yet is the ability to have a keyboard and
type in the area, which is crazy. One of those startups that applied
for XR Ignite, actually, is working on that. They’re working on a
keyboard that allows you to see your fing– well, not your fingers,
but see your keys in VR. And it allows you to type in a more
comfortable way in VR.
Dirk: That’s definitely
something we need. And the things I’ve tested so far were not really
usable in the way I want to have it. So right now, I think we only
can enhance our current desktop office, but we really cannot replace
it. But what you mentioned is exactly right. I mean, you see that
often: the keyboard is really a key element in the way we work.
Absolutely. Yes.
Alan: Yeah. And actually,
Logitech introduced their their three dimensional pen recently. I got
a chance to try it at Augmented World Expo. And then there’s another
company called Massless. And basically what they’ve created is an
input device, looks like a pen, but it’s tracked in 3D space. So now
you can draw in three dimensions; something like Z space, which is a
computer that allows you put on really inexpensive glasses and see
three-dimensional things come out of the screen. They’ve also got
this kind of stylus effect, where you can use a pen to grab things
and bring them into 3D space. So I think the user input devices — be
it a keyboard, or voice, or gaze even (looking at something), or a
pen — we’re only just starting to figure these things out.
Dirk: Yeah, we’re early stages.
On one side, we want to reinvent the whole thing, which makes sense
because we need it. It’s a different environment. On the other side,
we also need some elements which we’re used to. Right? I mean, that’s
always funny. Many people also said, “so you tested a lot of
variables; a lot of AR glasses. So, why do we always have the typical
rectangular screens in that 3D space?” I said, “well, look
it’s just the beginning. But also keep in mind, if you learn
something new — if you have to get used to something — then maybe
so it’s good to have some things you are familiar with, right? And a
rectangle screen, with an X at the top right corner that lets us
know, okay. I know how to deal with that. And that’s also a good
feeling.” So it’s a transition phase somehow, where we have to
learn. And it’s interesting that you mentioned the stylus. Yes — I
played a lot around with the Apple pencil. Everything is digital
today. And I came back to that and I didn’t want to use paper,
because I don’t want to have another notebook in my backpack. So I
use that. And I have to say, it’s really a good way to work. I can
imagine that it helps us to work in space.
45 min
September 25, 2019 Building the AR OS for Enterprise, with RE'FLEKT's Dirk Schart
Most businesses
have the information and infrastructures they need to be more
efficient and competitive -- it's just a matter of having it all at
their fingertips. RE'FLEKT is working at making that process easier
by creating a modular, scalable, open-source operating system for
businesses to build their own in-house AR applications on top of.
President and CMO Dirk Schart drops in to explain how.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today's guest is Dirk
Schart RE'FLEKT President and CMO. Dirk Schart is a marketing and
growth driven tech executive, whose current mission is to build the
operating system for enterprise augmented reality. Dirk leads the US
operations for RE'FLEKT. Dirk is one of the first employees of
RE'FLEKT and is a funded company by global companies such as Bosch
and BASF, and today plays a leading role in enterprise AR.
Previously, Dirk built the Digital Innovation Lab for Hyperloop
Transportation Technologies, for which he continues to work as an
advisor. Within four months, he built a team and developed the first
MVP, and he also directs the VR and AR team for Hyperloop
Transportation Technologies. Dirk helps technology startups such as
VisualX, VYON, and RiseStep. He's the author of two augmented reality
books, albeit they're both in German, but you can get them. And he
also contributed to Metaverse, the book by Forbes writer Charlie
Fink. He founded the first AR blog in Germany, "WE ARE AR"
and has been interviewed and quoted by leading media such as Tech
Crunch, Venture Beat and Huffington Post. For more information on
RE'FLEKT, you can visit re-flekt.com.
Dirk, welcome to the show.
Dirk: Hey, Alan. Thank you very
much for having me.
Alan: My pleasure, my friend.
The last time I saw you was at Augmented World Expo, and I think we
were taking silly photos.
Dirk: That's true. But was fun.
[laughs]
Alan: It's always fun. And it's
becoming like a family, this whole augmented reality family and
everybody's working together. And it's been wonderful. And the work
that you guys are doing at RE'FLEKT is world class. And I really want
to dive into the benefits of this technology and how you guys are
using it. Let's talk about that.
Dirk: Absolutely. So you already
mentioned it in the intro; what we're doing is we're building the
operating system for enterprise AR. So what does that mean? That's
for us the foundation of how enterprises will use AR in the future.
Let me go back quickly and talk a little bit about how we started.
That makes it easier for the listeners. Back in 2012, it was really,
really difficult to create any kind of augmented reality
applications. So it was completely different to what we have today.
And we started to develop individually programmed applications. There
were no platforms. There was nothing at the time. And one of the
first things we did is, we build an application for Range Rover. And
the reason for that was that they came and said, look, we have a very
complex repair. It's about a few return line of a Range Rover car.
And the mechanics, they always have problems doing that because it's
complex. They don't do that every day. Documentation doesn't seem to
be perfect for it for that use case. They try PDF, they tried videos,
but result was the same, they couldn't fix the problem. So we built
an application.
We showed it to them and helped the
mechanic, because it guided them step by step through that repair and
showed them exactly at the point where they had the problems. "Wait.
Now focus on it. Have a look. Double check before you do the next
step." And that at the end helped them to perform the task much
better and reduce their error rates. And then, of course they asked
about, "okay, so how can we do that for all our car models?"
And at
0 min
September 23, 2019 Flexing Your Brain in XR, with Cognitive Design’s Todd Maddox
We often talk about
how XR technologies are great tools for education and training on
this podcast. But why is that? Like, physiologically? Turns out, XR
tickles the thalamus in ways traditional learning strategies never
could, and that’s not us just whistling Dixie.
Today’s guest —
Cognitive Design & Statistical Consulting, LLC CEO Todd Maddox —
has a PhD in Computational and Psychological Science, meaning there’s
no one better to explain why XR and your brain are a match made in
heaven.
Alan: You’re listening to the XR
for Business Podcast with your host, Alan Smithson. Today’s guest is
Todd Maddox. He is a cognitive design specialist. Todd is a PhD, and
the CEO and founder of Cognitive Design and Statistical Consulting
LLC. He’s also a learning scientist and a research fellow at Amalgam
Insights. His passion is to apply his 25 years of psychological and
neuroscientific expertise gained by managing a large human learning,
memory, and performance laboratory to help build better education and
training solutions. Todd has published over 200 peer reviewed
scientific articles, resulting in over 10,000 academic citations and
hundreds of speaking engagements. During his 25 year academic career,
he’s awarded $10-million in federal grants from the National
Institute of Health, National Science Foundation, and the Department
of Defense to support his research. Since entering the private
sector, Todd has embarked on a mission to translate the amazing body
of research conducted in the ivory towers into plain English and help
companies leverage this research to build better products. Todd is
especially interested in applying his expertise in the psychology and
neuroscience of learning, memory, and performance, and to use
immersive technologies in manufacturing, health care, corporate
training, and retail, to name a few. You can follow Todd on LinkedIn.
Just look for “Todd Maddox PhD.”
Todd, welcome to the show.
Todd: Hey, Alan, it is fantastic
to be here. Thank you.
Alan: It’s such an honor. I’ve
been reading your posts and your articles, and trying to get through
some of your scientific papers is a challenge. It’s so much
information there.
Todd: Yeah, I hear you. And to
be honest, my recommendation is to sort of skim the peer-reviewed
stuff, because it does seem like it’s written in a foreign language,
even though it is English. And the LinkedIn post and the more recent
stuff, where I really try to talk in plain English, because if a
scientist can’t present their work in plain English then there’s
something wrong. So that’s what I’m trying to do.
Alan: I love it. And one of the
articles that was recently published was a report on VR as an empathy
builder, through Tech Trends.
Todd: Yeah.
Alan: Here, I’m just going to
read a quote from it:
“Any profession that requires
interpersonal interaction, such as education, retail, food service,
call centers is better served with strong empathy.” Let’s start
with that.
Todd: Totally, yeah. Every one
of those examples is a people example; people interacting with other
people. I know we’ve got amazing technologies; we’ve got robots,
we’ve got all these wonderful things that are making our lives
better. But let’s face it, in the end, it’s about people interacting
with other people and caring for other people, walking a mile in
somebody else’s shoes. That is really just so critical.
These technologies — in particular
virtual reality, I would say — this is an immersive technology. I
could be dropped into any environment. That’s amazing, that’s very
cool. But now imagine: Todd, a middle-aged hetero white guy gets
dropped into an environment where Todd is now a young
African-American lesbian woman. And whatever’s happened to her has
happened to her and hey, it’s happening to me now. Now, granted, is
this the same as a lifetime of “go back where you came from?”
Of course it isn’t. But it’s a start, and it’s visceral. It engages
emotion centers in the brain in a way that is rare. And of course,
you can do this over and over and over again in virtual reality,
because I can be plopped into any environment, into any person’s body
that we like. So when it comes to empathy and understanding, I really
think that’s one of its many sweet spots, I’ll say.
Alan: There’s been some recent
research around using this for human resources, for hiring, and then
also one thing that I saw was really neat is, as a manager, you were
able to sit in virtual reality and reprimand somebody. You had to
give a disciplinary action. You sat there and you talked to the
person and it recorded your eye tracking, your hand motions, your
head pose. And then afterwards, you got to sit in the seat of that
person and watch yourself give yourself the reprimand.
Todd: Yeah. And that’s what it’s
all about, right? Just to start to dive a little bit into the
neuroscience behind all this, because that’s really where I reside
and am most fascinated. It’s one thing to study what it’s like to
give somebody a reprimand. “Here’s a PowerPoint on how you give
somebody a reprimand,” or even, “here’s a video — watch this
person getting a reprimand. What did they do right? What did they do
wrong?” All of that information is processed by what’s called
the prefrontal cortex, which is right behind your forehead. I call it
the “What System.” It’s the part of your brain that learns
what you’re supposed to do. OK, that’s great, and it’s really
important to know what to do. But let’s face it: I know that I
should eat a little more healthy than I do. So, I know what to
do. But do I do it all the time? Knowing what to do, and
knowing how to do something are two completely different
things. They’re mediated by completely different parts of the brain,
and the processing characteristics of those two brain systems are
very different.
So what you’ve got in the VR example
that you’re suggesting is, you’re actually in there doing, instead of
“here’s what I should do — watch this PowerPoint, hopefully
you’ll go do that stuff — you’re in VR and you’re actually doing it.
And I love that it’s measuring things like body language, non-verbal
communication. What’s fascinating about non-verbals is, it’ll
actually provide an enormous amount of information about whether
somebody is being genuine, showing empathy, whether what they’re
saying is not really what they mean. These are absolutely powerful
tools.
And then, yeah, when you can then swap,
you’re now getting the reprimand from you. It’s truly remarkable.
You’re just so broadly engaging so many of these parts of the brain
that are critical for these kinds of tasks. It’s just an amazing
technology.
Alan: And in this podcast,
there’s always the same things that come up: “I want to be able to
empower my field service workers hands-free.” “I want to use
virtual reality for training.” Now, the training aspect of things
is starting to accelerate much faster than anything else, because the
results that we’re seeing across the board in companies like Boeing
and Lockheed Martin and Wal-Mart, these big companies are seeing
real, tangible results using this technology. Why do you think it’s
so powerful? What is it about this technology that’s creating those
lasting synapses in the brain that feels like you’ve done it?
Todd: Yeah, it really comes down
to the brain. And there are a couple of things. Traditional ways of
training – really, what I did for 25 years when I was a college
professor educating undergrads — you have butts in the seat and
you’re droning on at them. Well, what’s happening in your brain?
First — again, we’re focused exclusively on the prefrontal cortex —
if you ask your grandmother, you know, well, what does it mean to
learn? Ask just any old buddy. That’s really what they’re talking
about. They’re talking about the prefrontal cortex. “Okay, I
have to sit down. I have to concentrate. I have to process this
information. I’ll mentally rehearse it so that I can, quote/unquote,
remember it.” Everything is driven through the prefrontal
cortex. It’s part of the brain that sets us apart from,
quote/unquote, lower animals — which is the term actually I don’t
like that much — but in an evolutionary sense, it allows us to have
what we call metacognition. That is, we can think about the fact that
we think.
But let’s take VR — or AR, for that
matter. First thing is, I’m learning through experience. I am in
the environment while I’m learning. I’m not in a classroom. I’m not
at a desk studying. I am in the environment. Why does that matter?
Well, that broadly engages perceptual representation regions in the
brain — the back of the brain, the sides of the brain, the top of
the brain; all these parts of the brain that represent the
environment that you’re in. So, you’ve already got that activation
alone.
Now let’s take, say, a hands-free AR
device. Now you’re actually generating the behaviors of whatever your
job is. And you’re being guided, let’s say, with AR assets in the
Hololens: you’re actually generating these behaviors in a guided
fashion. You’re not just reading a textbook that tells you what the
behaviors are — that hopefully you’ll remember when you’re out on
the field — but rather, you’re being guided through these behaviors,
and you’re doing them in real-time out in the field.
So, I’ve got the behavioral centers of
the brain that are activated. These are the centers deep down in the
brain in a region called the striatum. Simultaneously, if it’s an
engaging and rich environment — or like with Wal-Mart, if it’s Black
Friday is the environment that you’re in — you’re going to have
emotion centers lit up. It is as if you’re plopped down in the middle
of Black Friday. So, emotion centers in your brain are lit up — like
the amygdala and some other limbic structures. Long story short:
traditional training engages one part of your brain — the prefrontal
cortex — which is a part of your brain that cognitive load is a
problem; working memory capacity is a problem; attention span is a
problem. Versus virtual reality or augmented reality, where you’re
engaging experiential learning centers, cognitive centers, behavioral
centers, and emotional centers. So, much more the brain is being
engaged in synchrony.
This is why you learn more quickly, you
make fewer errors, and you retain the information more. This is
really what is driving these amazing return on investment for forward
thinking companies that are using these technologies.
Alan: That was actually…
[chuckles]
you know how to speak my language on this show! The XR for
Business Podcast is all about ROI — what are the best investments we
can make as businesses, to drive our business forward using
virtual/augmented/mixed reality technology? So, my next question was,
of course: what are some of the ROIs being seen by businesses?
Todd: Yeah. So you mentioned
several of them: Lockheed, Walmart. I’ve actually been following PTC
quite a bit. Actually, that’s where you and I met. I’ve been
following some of their technology and actually met several people
who were using one of their products called Expert Capture, and
getting… trying to remember the exact numbers. But, training times
cut in half, standard operating procedures being generated ten times
faster than they were before — you’re seeing numbers like this from
company after company after company. And again, from a neuroscience
of learning and a neuroscience of performance perspective, none of
this surprises me, because we’re talking about engaging one system in
your brain, versus engaging three or four systems in your brain, and
engaging them in this… almost like a ballet. It’s this beautiful
synchrony, all in the interest of achieving whatever it is that your
goal is. Whatever you’re learning task is, your training task,
whatever it is you’re trying to perform. And so you’re seeing these
ROI that’s off the charts.
I will say — and I actually just wrote
a report on this last week, also at Tech Trends — that I’ve been
making this case. I’m going to continue to make this case so people
listen to me. These technologies are fantastic. They broadly engage
all these parts of the brain, which is super. And you’re seeing
amazing ROI. My very, very strong belief is we can double those at a
minimum. Wow, how can we do that? I mean–
Alan: Hold on, hold up! So,
we’re already seeing 50, 60, 70 percent increases right across the
board. Retention rates, memorization techniques, faster training
time. You’re saying we can even double that?
Todd: We can do better on those
metrics, and we’re going to get other value that we’re not even
really measuring. For example — and I think Expert Capture’s a good
example — you’re going to retain this information better. You’re
going to learn more quickly. Okay, that’s great. You’re gonna know
what the standard operating procedures are, and you’ll be able to
regurgitate those back to me. I think you can — and I don’t want to
get too into the weeds here — but there are aspects of, in
particular AR, that are not optimized for training behavior.
Most of these AR assets… let’s say
you have the Hololens on. You’re working on a machine. The Hololens
says “move your left hand over here, and turn this knob.”
Okay, that’s great. That’s called “guided learning.” And
guided learning is solid. There’s nothing wrong with it. But if you
start tweaking and incorporating other ways of doing this, that are
less guided and more discovery-based by the learner, and –
critically — involve real-time reward and punishment. You are now
going to be training the muscle memory correctly. It’s actually
better for you to discover these things. So, be guided partially.
Then you have to generate a response and you get reward or
punishment. That is how you’re going to train the muscle memory.
You’re going to be able to train
expertise. I have 25 years of expertise in manufacturing. I believe
that with these tools, we could create experts much, much faster. And
that’s not really the goal of a lot of these technologies. It’s
really just “translate that expertise into a series of steps
that I present on the Hololens, show it to the workers to get them up
to speed.” So, they’re up to speed. That’s great. But they’re
not experts. They don’t have all of the expertise that that baby
boomer who’s retiring has. I believe we can impart that as well. But
we’re gonna have to make some changes to the way that these
technologies work. We’re going to have to optimize the AR assets to
the way the brain learns best. AR works very well now, because it
broadly engages all these parts of the brain. And that’s great.
That’s a good starting point. Now we need to optimally engage each of
those parts of the brain.
Alan: Yes. How do we do that,
then? You’ve got an AR experience that guides me through fixing a
machine — and I’ve been able to expert capture, I’ve been able to
put on a camera on one of my experts, because here’s the rub: as the
workforce starts to age and retire, the problem isn’t so much that
there aren’t jobs, the problem is that the jobs are changing
slightly. The experts who are the best in the world at that job are
starting to retire. How do we have the skills transfer from one
generation to the next?
Todd: The first thing is we need
to capture that expertise, that boomer expert who’s in manufacturing.
If you tell him, “go sit at that computer and write out your
standard operating procedures,” for one, he’s not going to be
happy. And for two, he’s not going to do a very good job. And you
might ask, “well, why is that the case? He’s got all this
expertise.” He has behavioral expertise. His expertise is in
how he interacts with that machine. That’s muscle memory. That’s
understanding situational awareness. So he actually has a feel
for when that machine isn’t working right. He can’t even describe it,
he just sort of knows it’s off a little bit. And you’re asking him to
sit down and type out standard operating procedures, which is
knowledge. So you’re asking him to use his prefrontal cortex to write
out procedures, when his expertise is behavioral and situational
awareness.
So that’s one thing: we have to do a
better job of capturing it. Expert Capture does a good job on that!
I’m not saying it doesn’t, because it says, “hey, go do your job
and I’m just going to capture it.” But how we curate that, and
how we present that to young workers is absolutely critical. And
we’re not optimized. That has not been the focus to this point.
Alan: I think a lot of the focus
has been just trying to get the technology to work, let’s be honest.
Todd: Absolutely.
Alan: We’re only 12 months into
having headsets that actually turn on when you want them to do it,
and object recognition. It’s a very new field. And I’m really excited
to listen to the next part of what you were going to say, and that
is, how then do we optimize for this? Because we have the very basics
worked out. I can put on a pair of glasses. I can recognize a
machine. I can walk you step-by-step through how to fix that machine.
Now, what you’re saying is, let’s create a more discovery-based, or
maybe even gamified experience, where I learn, but I’m learning while
doing and making mistakes.
Todd: Absolutely. To your
initial point, I want to be clear that I am not making the case that
people [in the industry] are too slow. No, not at all. And you’re
right, we are we’re in the very early stages. But we have these
technologies, and it’s time to say, “okay, I’ve got this great
technology. I can present any AR asset I want, I can present it
anywhere I want, and I can present it anytime I want.” The
question is, “when do I want it? What do I want? And where do I
want it?”
That’s where science comes in. And
first: Neuroscience. I actually had an interesting discussion about
this with Jim Heppelmann months ago at PTC, he was talking about “we
use green for some things, and red for other things.” And I
said, “Yeah. Do you know why that is, Jim?” And he’s like,
“well, green means go and red means stop.” I said, “well,
there’s a reason for that.” The cells in your brain that respond
to green are the same cells in your brain that respond to red. But
they are excitatory to green and inhibitory to red. In other words,
it is impossible not to discriminate green from red. You can
never confuse the two, because it’s built into the neurons.”
We need to leverage more things like
that. We need to go back and look at how the brain processes the
what, the where, and the when. And we need to build that into these
AR assets to optimize these things. Gamification, yeah. Start with
directed learning and then start weaning this worker off of the
directed learning, toward more discovery-based. That is going to
speed the muscle memory. That also requires rewards and punishments.
So there’s gamification right there. That’s going to be the next set
of major upgrades, I think.
Alan: What about artificial
intelligence? I just read an article recently that was talking about
a company that’s using AI for recruitment; they’re able to run these
people through a couple of games, a couple of questions, and a video.
The AI’s analyzing the video, it’s analyzing their results on the
game. And then it’s saying, “out of these hundred thousand” — it
was Unilever, by the way, that it’s doing this.
Todd: OK.
Alan: —
“out of 100,000 applications, we’ve narrowed it down to
3,500, based on just this algorithm for this particular job.” And
then they’ve also started using it to then unlock what does the
potential look like for, “maybe you’re not right for this job,
but hey, you scored in the 98th percentile on this other job that you
didn’t apply for.” It may make for much better recruiting.
One of the things that I think is
necessary that we haven’t even scratched the surface of: we use
Netflix algorithms every day to give us better movies to watch. We’re
not even touching the surface of what’s possible, when we create that
sort of AI algorithm to give us better learning in a way that works
best for us.
Todd: Totally. Definitely
preaching to the choir on that. I mean, big picture, you do an
assessment. If I was involved in something like that, I’d say, “okay,
great. So what is it that we’re trying to train for? What is the task
that the worker will be doing?” And that’s, of course, what
we’re selecting for. What are the aspects of that task? Oh, it’s a
very behavioral task. Or it’s very cognitive-heavy. Or, well, you’ve
got to have situational awareness, you’ve got to deal with any old
thing that might happen to you. Or all three.
I’d look at what are the
characteristics of the perfect worker: you’ve probably got some [in
your company]. We’re going to measure all those aspects of those
workers. And in particular, I would be guided by the neuroscience of
performance and learning. Then what you do is you use AI. And of
course, you’ve got new data coming in, so you’re constantly updating
your algorithm. But then what you do is, you have a potential new
hire or recruit, you get measures from them. It’s not a
template match per say, but it’s a match. You’re basically
correlating that new recruit’s scores with the scores for the ideal
employee, and you get sort of a measure of fit.
And I love the idea that you apply for
a job A; you’re actually not a very good fit for job A. Well, we have
job B over here; you’re a really good fit for. It’s such a more
efficient way of doing things, because let’s face it: people don’t
know what job they want, what job they’re going to fit for. If we
could use AI to help guide that process and put people in the jobs
that are best-suited for them, whether they, quote/unquote, know that
or not, right? We’re not that great at introspecting about ourselves.
We really don’t know that much about ourselves. We think we do, but
we don’t. Whereas these kinds of tools can actually give us a better
insights into what we would be good at. So, AI machine learning,
these kinds of algorithms: awesome, awesome future.
Alan: So we’ve got all of these
new technologies. They’re happening fast. We’re seeing great results
across enterprises. You said we can do more; we can do better. How
does a company even start to evaluate or look at these tools? Because
you’ve got a handful of companies working on manufacturing and
industrial, and those seem to be getting really great traction. But
what about sales training, or HR training, or soft skills training,
or retail, or… there’s so many other aspects of business that maybe
aren’t as obvious at front, but what are some of the other ones that
you see these techniques working on?
Todd: I’m really glad you
brought that up, because we’ve been talking a lot about the
manufacturing and industrial sector, and I think these tools are
taking off there, for one, because the ROI is so clear.
I was actually having a discussion with
a colleague a couple days ago, that the beauty of manufacturing is, I
can have you use one of these AR tools and I can see how quickly you
complete the task. Boom. I mean, there’s my data; it’s right in front
of me. It’s a short-term ROI. I see it immediately, and I can see it
from all of the people that I have use this AR tool.
For HR, for sales, for what people call
soft skills — I’ll use the term people skills – but, interpersonal
skills. These are harder to measure. They’re slower to develop and
evolve. They’re a little mushier to measure. And so it’s been more of
a challenge to measure the ROI for, let’s say, people skill training
with VR, than in the manufacturing sector. But it’s still there. And
I think maybe that’s where the scientists can really come in and try
to identify some short-run ROI, but also really that longer-run ROI.
Let’s face it: it’s a lot easier to interact with a machine than it
is interact with people.
Alan: People are… this
morning, before we jumped on this, I said, “you know, I feel like I
have a weekend hangover, without the alcohol.” I mean, people are
not like a machine. The machine wakes up Monday morning and just
works. It doesn’t have three days of camping, or it didn’t go to
Vegas for the weekend. Machines don’t do that. But humans, we have
all of these complicating factors. And it really complicates
business. And it’s part of the interconnected web of humanity. But at
the same time, if we can better attune ourselves to watch out for
these things using this technology. I think it’s really powerful.
Todd: I mean, let’s take
healthcare. Let’s take law enforcement, firefighters, call center,
retail — anybody who has to deal with, quote/unquote, putting out
fires with adversity. The term I use is situational awareness. It’s
an ability to always know what you need to do, right now. It’s like
you always make the right choice, and this uncanny ability to kind of
predict what’s going to happen in five minutes. There are people that
have that. And guess what? It’s trainable.
Alan: Now you’re getting crazy,
Todd. Come on now.
Todd: Well, I–
Alan: You can train a sixth
sense?
Todd: You can. You absolutely
can.
Alan: OK, let’s unpack this.
What you’re saying is, using situational awareness and situational
training in environments such as VR — where you can recreate that
environment — that situational awareness becomes built up. But how
can you build that sixth sense in somebody to predict what will
happen?
Todd: OK, so yesterday — or I
guess it was Saturday — it was the 50th anniversary of [the Apollo
11 moon landing] — those
guys had a sixth sense. They had these simulators that cost
equivalent of billions of dollars; over and over and over again, they
got thrown at them every possible situation, no matter how likely or
unlikely. They were prepared for anything. There is a use case right
there. It has been trained, and it can be trained.
“Yeah, but gosh, that was like, a
handful of guys, and it cost a ton of money. So that’s not too
realistic, Todd, in the real world.” No, you’re right. Okay, but
now imagine putting somebody in VR. You could throw all the same
situations at them. You can measure their physiological responses. VR
focuses mostly on where you look, and of course we have auditory, but
there’s no reason we can’t — actually, we should — start
looking physiological responses.
Alan: Yeah, I saw something,
somebody had taken the Gear VR — which is Samsung, you slot your
phone in and put it on your head — and it was a meditation app that
used the Samsung Gear Watch to keep your pulse rate. As you’re
meditating, you can watch your own heart rate inside the VR headset.
If we start thinking about… most of the headsets right now, the VR
headsets don’t have eye tracking. I think the only one that really
has eye tracking is the HTC Vive Pro Eye. Something so simple as
being able to know where exactly the person is looking has
dramatic effects on the knowledge base of how we move forward with
this technology. Then, when you increase that with skin response or
heart rate? We haven’t even really touched on that stuff.
Todd: Totally. And I think
that’s… I’ve talked to a number of people recently, some pretty big
companies that are starting to bring VR and/or AR into their
companies. I’ll talk to them about the neuroscience, and they’ll say,
“we’d really like to do some studies where people get a magnetic
resonance imaging machine or put the EEG on them.” And I said,
“I’ll be really honest. Those are really, really great tools.
But why? Why do you want to run one study — that’s going to cost you
an enormous amount of money — to learn one thing?” I mean, you’ll
have some pretty pictures of the brain, but there are strengths and
weaknesses of all of those techniques. And that’s a time for another
discussion.
Alan: Well, I think you can get
really deep with this technology or you can just take it one step at
a time and say, “okay, let’s just measure your heart rate.”
That’s simple with a watch band.
Todd: Totally. And if we’re
talking about situational awareness, and we’re talking about dealing
effectively with stressful situations, knowing that you’ve got EEG
activation, or that your amygdala is lighting up is not really
relevant. What’s relevant is: are you calm? You can measure that with
heart rate, galvanic skin response. We can measure that with things
that are on the market today, that are super effective.
Alan: And inexpensive too, let’s
be honest.
Todd: And inexpensive.
Alan: Skin response and heart
rate is cheap.
Todd: Very. And they’re
everywhere, right?
Alan: The ubiquity of
sensors.
Todd: And you’re seeing more and
more of… certainly, the military spending a lot of money on this,
but understanding the physiological responses: hey, perfect
application of AI. I can determine whether — to use a military
example — whether a war fighter is ready to go to war today. I can
put them through a simulation; I can be measuring their physiological
response. “You’re not up to snuff today.”
Alan: Military, they’re way
ahead. They’re using this technology, they’re studying these types of
things. But let’s just take it to even the most basic aspects: a
K-to-12 learner; some kid in grade 6. We send our kids to go to
school from 8:30 in the morning until 3:30 in the afternoon, and they
learn science from 10 to 11, math from 11 to 12. We just run them
through this gamut. But have we ever really looked at what is their
optimal time for learning? Maybe those kids learn math better at 8:30
in the morning, or maybe phys-ed in the morning. Have we ever really
looked at that? And of course, every individual learner is going to
be different.
Todd: Yeah. OK, so there’s a lot
to say and there’s a — I’m actually really glad you brought up kids,
and I’m going to add middle-aged and older adults to the mix as well,
so–
Alan: All the people.
Todd: Basically, anyway.
[laughs]
Well, okay.
Alan: You know, to be honest,
we’re all lifetime learners now.
Todd: We are.
Alan: Gone are the days where
you go to school, you graduate, you go into a job, and that’s the end
of your learning career. Now we’re entering the exponential age of
humanity. We must maintain our lifetime learning status.
Todd: Totally. There’s no doubt
about it. And we need to use learning tools that are optimal for
where we are in our lives. Kids — actually, the prefrontal cortex is
not fully developed until you’re about 25 years old — yet we have
kids learn math, like you say, from 10 to 11 in the morning. And what
are they learning it with? Their prefrontal cortex. We are training
children to learn information with a system that’s not fully
developed. That’s crazy. We should be using immersive technologies
that more broadly engage more parts of the brain, that are fully
developed. I’m not saying we shouldn’t still work the prefrontal
cortex, but relying exclusively on the prefrontal cortex with
children is incredibly suboptimal.
Move to the other end of the lifespan.
Your prefrontal cortex actually starts declining in your 40s. On that
end of the spectrum, we shouldn’t be relying purely on the prefrontal
cortex. Healthcare examples are great: I go into my doctor, they tell
me I need to have some procedure. They give me some bunches of pieces
of paper to take home to read about the procedure. Incredibly
ineffective. How about if I go into a VR experience, I see what their
procedure’s like? I’m going to be less stressed, I’m going to have
more knowledge, I’m going to be more prepared for it, and even more
satisfied. Guess what? I’m going to be more likely to heal well after
that surgery. There’s actually data on this.
Alan: There’s a sick kids
hospital here in Toronto, that did a partnership with Samsung,
originally. And what they did was, they just put a simple 360 camera
on a gurney and they wheeled it through, as if it was a kid. They
wheeled it right through to surgery so that you know which hallways
you’re going to go down, and which room you’re going to go in, what
the room looks like, and what the sounds are like. They let kids wear
that in VR before the surgery, and it decreased their stress. And
when you’re going in for surgery, stress is actually the opposite of
what you need. When you’re going into surgery, you need to be calm
and relaxed, and let your body heal.
Todd: Yep, you don’t want to be
stressed out. Reading a piece of paper does not provide the
information that you want, so you’re still uncertain. You’re still
stressed. Put on a VR headset: again, you’re broadly engaging more
parts of the brain than just the one. I don’t have to generate a
mental representation of what the hospital’s like, like I have to do
with the piece of paper. No, it’s in front of me. I know exactly what
the hospital is like. I know where the surgery room is, I have the
feel of it. And I’m going to be less stressed because of that. I’m
going to be more likely to heal. No brainer. And that’s cheap.
Alan: Really cheap. So, to put
it in perspective: in all of human history, we’ve never had a device
that can deliver training and education as efficiently and
effectively as virtual and augmented reality. Now, you combine that
with studying our emotions through eye tracking, head pose — that
sort of thing — our movements, and then also through our biometric
responses — our heart rate, our galvanic skin response — when you
combine that all together and then run it through AI to deliver this
in a personalized manner: we really are setting ourselves up for what
I believe to be the most powerful tool we’ve ever created to train
people. And I think it’s the only way that we’re gonna be able to
remain competitive in a world where everything is increasing
exponentially.
Todd: Yeah, I mean, I completely
agree. And the beauty is that each of the parts of that puzzle, we’re
making progress on. We’re making progress on AI. We’re making
progress on wearables that tell us all kinds of stuff. The VR
headsets are getting better, and the cost is going down.
We need to put all that stuff together
— which we’re doing — and we need to use science. We need to look
at how the brain processes things, do some experimental science as we
optimize these tools. And you’re absolutely right: the savings,
people are going to learn better, they’re going to be less stressed
in medical situations. We’re going to have better-trained police
officers and, I mean, you name it: everything. And it’s just around
the corner.
Alan: One thing we didn’t touch
on, which I think is timely right now, is neural link, or
brain-computer interfaces. Elon Musk’s company, Neuralink, announced
that they’re going to be able to embed these little microfibers that
act as almost like a brain stimulator, and also capturing the data.
So we’re gonna be able to capture and read-write onto our brains.
What are your thoughts on that?
Todd: Yeah, I actually have a
colleague who’s really into it, so we’ve been talking a lot about
this. I actually read a really cool article about neural link a
couple days ago. You know, it’s insane. OK, so look. So what we’re
trying to do is we’re trying to link to the brain. OK, that’s great.
Well, what is step one? “Step one: understanding how the brain
works.” Hmm, we really don’t know how the brain works all that
well, do we? “No, we don’t.” So step one is to understand how the
brain works a lot better than we do today.
Now, I think we can do that in concert
with the goals of neural link, but I think we have to always keep our
eye on the ball, which is: how does the brain process? What do these
signals mean? Understanding that there are different parts of the
brain that provide different types of information and signal
different things, and making sure that we combine that information in
the same way that the brain does. So, the engineering side is great.
We need good engineers. But we also have to always pay attention to
the “neuro” side of neural link, and really understand how the
brain works. Then we’re gonna make some real progress. And it’s
incredibly exciting.
Alan: It really, really is. I
read this quote in one of your articles that you wrote, and it really
stuck with me. And I think this kind of will cap off this
conversation:
“Learning is an experience.
Everything else is just information.” -Albert Einstein.
Todd: [laughs] A colleague of
mine, Tim Fitzpatrick, who was actually the CEO of a VR health care
company. He exposed me that quote nine months ago, and I was just
blown away by it. I mean, I’m so enamored with Einstein anyways, and
was like, “yeah, he’s a learning scientist, too, because it’s
brilliant.”
But absolutely. Learning is an
experience. You learn while doing. You learn in your environment.
Why? Because it broadly engages so many parts of your brain at once,
that the information sticks; it’s retained. You learn the behaviors.
Everything else is just information. Information is the prefrontal
cortex. It’s how we focus almost all of our learning tools. We try to
drive everything through the prefrontal cortex to the information
system, when we should be training people through experience.
Alan: Well, thank you, Todd.
We’re at the end of the XR for Business Podcast with your host, Alan
Smith, and this has been an amazing exploration of what’s possible
when we use virtual/augmented/mixed reality for training, artificial
intelligence, and biometric responses.
I have one more question to ask: what
problem in the world do you want to see solved using XR technologies?
Todd: Oh my goodness, there’s a
lot of problems in the world, but I guess the one I’ve been most
focused on — and there’s some personal reasons for this – but:
senior care. We have — some people call it the silver tsunami —
running down the train track. We’ve got boomers, myself included, who
are aging, and we do not have enough people to take care of seniors.
And we need to do a better job of training not only the seniors on
what’s to come; their family members on what to expect; and experts
to help them. And I really believe there’s so much empathy involved
there, and so much detailed understanding and training: these
technologies can absolutely solve that problem. So that’s one of the
ones that I’m most excited about and want to put a lot of my time and
energy on. But many, many more. Many, many more.
“Cha-ching”
sound effect by Muska666
41 min
September 23, 2019 Flexing Your Brain in XR, with Cognitive Design's Todd Maddox
We often talk about
how XR technologies are great tools for education and training on
this podcast. But why is that? Like, physiologically? Turns out, XR
tickles the thalamus in ways traditional learning strategies never
could, and that's not us just whistling Dixie.
Today's guest --
Cognitive Design & Statistical Consulting, LLC CEO Todd Maddox --
has a PhD in Computational and Psychological Science, meaning there's
no one better to explain why XR and your brain are a match made in
heaven.
Alan: You're listening to the XR
for Business Podcast with your host, Alan Smithson. Today's guest is
Todd Maddox. He is a cognitive design specialist. Todd is a PhD, and
the CEO and founder of Cognitive Design and Statistical Consulting
LLC. He's also a learning scientist and a research fellow at Amalgam
Insights. His passion is to apply his 25 years of psychological and
neuroscientific expertise gained by managing a large human learning,
memory, and performance laboratory to help build better education and
training solutions. Todd has published over 200 peer reviewed
scientific articles, resulting in over 10,000 academic citations and
hundreds of speaking engagements. During his 25 year academic career,
he's awarded $10-million in federal grants from the National
Institute of Health, National Science Foundation, and the Department
of Defense to support his research. Since entering the private
sector, Todd has embarked on a mission to translate the amazing body
of research conducted in the ivory towers into plain English and help
companies leverage this research to build better products. Todd is
especially interested in applying his expertise in the psychology and
neuroscience of learning, memory, and performance, and to use
immersive technologies in manufacturing, health care, corporate
training, and retail, to name a few. You can follow Todd on LinkedIn.
Just look for "Todd Maddox PhD."
Todd, welcome to the show.
Todd: Hey, Alan, it is fantastic
to be here. Thank you.
Alan: It's such an honor. I've
been reading your posts and your articles, and trying to get through
some of your scientific papers is a challenge. It's so much
information there.
Todd: Yeah, I hear you. And to
be honest, my recommendation is to sort of skim the peer-reviewed
stuff, because it does seem like it's written in a foreign language,
even though it is English. And the LinkedIn post and the more recent
stuff, where I really try to talk in plain English, because if a
scientist can't present their work in plain English then there's
something wrong. So that's what I'm trying to do.
Alan: I love it. And one of the
articles that was recently published was a report on VR as an empathy
builder, through Tech Trends.
Todd: Yeah.
Alan: Here, I'm just going to
read a quote from it:
"Any profession that requires
interpersonal interaction, such as education, retail, food service,
call centers is better served with strong empathy." Let's start
with that.
Todd: Totally, yeah. Every one
of those examples is a people example; people interacting with other
people. I know we've got amazing technologies; we've got robots,
we've got all these wonderful things that are making our lives
better. But let's face it, in the end, it's about people interacting
with other people and caring for other people, walking a mile in
somebody else's shoes. That is really just so critical.
These technologies -- in particular
virtual reality, I would say -- this is an immersive technology. I
could be dropped into any environment. That's amazing, that's very
cool. But now imagine: Todd, a middle-aged hetero white guy gets
dropped into an environment where Todd is now a young
African-American lesbian woman. And whateve
0 min

About XR for Business

From the publisher's feed

Meet the leaders who are changing the face of virtual and augmented reality

Share XR for Business

Sign up to save your podcasts

XR for Business

XR for Business episodes

About XR for Business