XR for Business

By Alan Smithson from MetaVRseBusiness Technology Arts

XR for Business episodes

February 25, 2020 Creating a Dialogue Between Innovators and Educators, with VirtualiTeach’s Steve Bambury
Using VR in the classroom is a no-brainer. It’s immersive tech, and can teach kids in new, innovative ways. But if the people developing the technology don’t understand how kids’ brains learn, it’s not going to take, no matter how innovative. VirtualiTeach’s Steve Bambury drops by to explain how he’s trying to bridge that gap.
Alan: Hey, everyone, my name’s
Alan Smithson. Coming up next on the XR for Business podcast, we have
Steve Bambury, founder of VirtualiTeach. We’re gonna be talking about
digital literacy, the virtual/augmented reality platforms, and the
question on everybody’s mind: What are the key barriers to adopting
VR and AR in schools and how to overcome them? All this and more,
coming up next on the XR for Business Podcast. Welcome to the show,
Steve. How are you?
Steve: I’m good, man. It’s good
to speak to you.
Alan: It’s really great. The
last time we saw each other, we were in Dubai — where you live —
and you took me to the Dubai Mall, and we went in and we went to the
VR Park, the giant VR Park. And I was just blown away by how big and
ostentatious everything was. And it was a really great experience. I
can’t thank you enough for your warm hospitality in Dubai. But today
it’s all about you. So let’s talk about what you’re doing, and how
did you get into this? And what are you doing now?
Steve: I’ve been in Dubai for 11
years. And for those 11 years, I’ve always worked at the same school.
I was working a school group here known as GESS — which is the
acronym for Jumeirah English Speaking School — also broadly referred
to as GESS Dubai now. GESS is one of the leading schools in the
Middle East. It’s a very old school, at least in terms of
international schools in this region. It’s only, I think four years
or three years younger than the UAE as a country. So it is very well
established. And yeah, so I worked there for 11 years. I worked as a
class teacher in one of the primary schools, and curriculum leader.
Eventually become head of computing at the primary school. So I was
teaching digital literacy and computer science content to four year
olds, 3 to 11 year olds. And I ended up in that role primarily
because of all the work I’ve been doing to integrate the iPads in the
classroom. From 2011, we were one of the first schools in the Middle
East to to roll out iPads in the classroom. And then three years ago,
I moved into a role that was created for me, which was the head of
digital learning and innovation, working underneath the new director,
Mark Steed, who’d just come in from the UK. Mark had the pedigree in
terms of digital learning from what he’d done at this very, very
prestigious school in the UK called Berkhamsted. He’d also chaired
the Independent Digital Strategy Group for eight years there. And so
Mark created this role and this role took me out of the classroom
most of the time. A lot of it involved training with staff. It also
involved going back into departments and helping them with enrichment
projects. And it was kind of in parallel to that. I mean, part of the
reason that my work with virtual reality really took off is because I
moved into this new role, and had this freedom to innovate and to
explore new technologies. My first VR headset was just a [garbled]
headset I imported from the States in 2014. But it was not long after
I started this new role as head of digital learning at GESS that I
got my first Vive. I took that Vive into the school and started
looking for ways to integrate it into different curriculum areas. In
actual fact, I’ve just recently started writing a series of guest
posts for Vive on the Vive blog. You can go into Google, like “HTC
Vive blog Steve Bambury” or something, you’ll probably find
them. But I’ve been writing a series of blogs about my journey using
and integrating the HTC Vive headsets at GESS. The first one focused
on all those initial trials that we’ve run. My background is in film.
So before I was a teacher, I worked in the film industry. So I’ve
always done a lot of film projects with kids. So one of things that I
was doing throughout our journey with virtual reality was documenting
all of the projects that we were carrying out. I was capturing
student voice. I was capturing staff feedback. And I think that was
invaluable in terms of the success of our deployment of higher end VR
at GESS, because it enabled me to then use these — not only
internally but also externally — to promote what we were doing at
the school. But internally it meant that I had this evidence that
there was definite impact in terms of the integration of this
technology, and the results that were tangible when you were using
the Vives in the classroom. And then from there, we ended up
investing in more Vives. We brought in Acer mixed reality headsets,
as well. I ended up doing trials with the Vive Focus and other
hardware. The one thing obviously that is already clearly missing
from this mix, is the word Oculus. For those that are outside of the
Middle East region, just for context, Oculus has next to no presence
in the Middle East at all. The Rift never launched here. The Go never
launched here. The Quest has launched here, but only in shops. 100
percent markup price from the Europe and US price.
Alan: Oh wow.
Steve: So it’s been– yeah. I
mean, Facebook are here, but they don’t seem to value the region, so
everything tends to be Vive, with the WMI headsets sort of in the
wings somewhat. So yeah. So I did a lot of cool stuff with VR and
started becoming kind of known for the work I was doing with VR.
Couple of years ago I set up VirtualiTeach as a platform to share
best practice and share my ideas, my theory, my projects. And…
yeah. Broadly that became all I ended up speaking about at
conferences around the region and internationally. I was becoming
“the VR guy”. So–
Alan: You are the VR– You’re
the VR education guy!
Steve: So yeah. And it is
something that I’m really passionate about, and it’s something that I
see the future of education, in terms of where spatial computing will
take us. It isn’t just another gadget that schools need to consider
weighing up buying into or not. This is the evolution of computing in
general. So yeah, so I did that role — the head a digital learning
— for three years. And at this point I’d worked for the company for
11 years, and I was looking for the next step. And Mark — the
director — he was headhunted to go off to Hong Kong and work in Hong
Kong, which he now has. And so in parallel to that, I decided to
strike out on my own and set up my consultancy, which is broadly how
I’ve spent my summer. Normally in the summer, everybody leaves Dubai,
because it’s so ridiculously hot. And there’s so many expats here —
80 percent of the country is expats — so everybody leaves and
Dubai’s a ghost town in the summer. Very hot ghost town.
Alan: Oh, it’s so hot. Oh my
God. I was there this summer and oh my God…
Steve: So I was stuck here this
summer, because I had to deal with all the logistics and the
paperwork of getting my company license, my new visa, and everything
like that. And yeah, and this is me now, a couple of months in,
working for myself the first time in my life. I set up my consultancy
called Digital Inception. Anecdotally, the name refers to– it was
actually the name of a remote keynote presentation that myself and my
friend Luke Reece delivered in 2013, I believe, to the University of
Southern California from here in Dubai using the Nearpod platform.
And the presentation kind of riffed on the movie Inception and the
ideas from the movie Inception, and the idea that if professional
development is good enough that you walk away, and you feel like the
idea was there all along, that you already knew it, it isn’t
something you’ve been preached, you’ve had something unlocked that
was already within you. And I always kind of liked that title, I
thought it was cool. And it was fitting that I could use it to
encompass the work that I was going to be doing with immersive
technologies, but also broad enough to encompass some of the other
areas that I work with. I mean, I was a distinguished educator. I do
a lot with Apple technology. I’m a Microsoft master trainer, so I do
a lot of work with Microsoft technology. So I needed something that
was kind of broader to encompass all of those things. So yeah, that’s
where I’m at.
Alan: [chuckles] So you’ve done
all of these things. You’ve been a pioneer. You also do the CPD talks
where you interview people. So by all intents and purposes, you are
an expert. Well, I would say the world’s leading expert on VR and AR
in education. And you also have VirtualiTeach. What is that platform
all about? Was that just a repository, a place for you to kind of
store everything that you saw? And then it became a website. How did
that happen?
Steve: Yeah, kind of. So in
2012, when I was doing a lot of work with iPads in the classroom.
Myself and my friend Luke — who I mentioned just now — who also
works at JESS with me. And he’s still at GESS, he’s one of the deputy
heads there. We were speaking at events, and we were getting loads of
people asking us questions and stuff. So we figured it would be a
good idea to tackle a website where we could share our ideas and
direct people to. So we set up a website called iPadEducators.com,
which went on to win an award, and it led to both of us becoming
Apple distinguished educators. And VirtualiTeach is– the approach is
broadly the same, both of these websites are completely not for
profit. I turn down advertising offers on a weekly basis. It’s never
been about making money. It’s just about sharing best practice, which
outside of the education industry, some people see it’s kind of
weird. If you’ve got ideas why you’re not charging for them? But
educators as a community, we’ve always been very, very much of the
kind of pay-it-forward mentality, that you share what you’re doing,
and we grow together. To be honest, it was 2017. And my youngest
daughter had been sick. And so the summer plans were canceled that
year as well. And I– essentially I was considering writing a book.
Ironically, the book was going be called Digital Inception. I was
going to write a book about all this stuff I was doing with virtual
reality, but the more I considered the time that I would need to
spend writing a book, and the delay in this kind of gratification of
sharing the content with other people. In the end, I just thought,
you know what? Let’s just put another website together, because I can
start putting content out quickly. And I like to produce content.
Alan: Writing a book. By the
time I read the book, first of all, it’s obsolete. Second of all, you
can’t keep it up to date. So we’re struggling right now. We’re
writing a book and– two books, “XR for Business” and “XR
for Education”. And by the time they’re finished, they’ll be 100
percent obsolete.
Steve: Yeah. Yeah. You know why?
In fact, last year in the end, I did put a book proposal together and
submitted it through a colleague to a publishing house. And that’s
basically what they come back to me and said. They said, “Look,
we like your idea. But by the time when you actually write this and
we publish it, we’re conscious of the fact that it will be out of
date.” And to a degree that’s true of all books based on
technology, because everything’s moving forwards.
Alan: Yeah.
Steve: So, yeah. So that’s
basically why the website became about. Now, I will give a little
caveat to anyone that’s listening, around about the time that we’re
recording this, back end of October 2019. So with everything that was
going on, with me setting up the company and various other bits and
pieces, I took a couple of months break from the website, and
unfortunately what’s happened is the website’s designed through Wix,
and over the summer Wix have launched a new blog feature. And
essentially my website is entirely built– the entire structure of it
is built around the old blog feature. And then I use custom feeds to
pull theory content onto a page, and Vive related content onto a
page, and guest articles onto a page. And it was all done with a
tagging system, which has been made redundant with the change to
their site. And essentially, I’m now in a position where I’ve got a
load of new content ready to publish. But at the moment, I’ve got to
find the time to go back and essentially re-label everything that
I’ve ever posted, which is like 150 articles. I’ve got to manually go
into all of them and and re-tag them. So my site’s kind of sitting
dormant at the moment, but hopefully we’ll be back on its feet before
the end of the year. Because I do have a lot of new content ready to
go on. But I have to say, sadly, the old content needs to be
re-tasked first, and I need to do some some layout adjustments as
well, to fit the new style that Wix have deployed.
Alan: I’m on your site now and
you’ve got– this is like the compendium for anybody looking to do.
And it’s VirtualiTeach, but “Virtual I teach”, it could
also be.
Steve: Yeah, that tends to be
what people say to me. You know what? When I was trying to come up
with a name, I’m going backwards and forwards with different ideas.
And I’d been reading a lot around the theory of the virtuality
continuum. And I had obviously the Curiscope Virtuali-Tee, you know,
the augmented reality t-shirts where you can look inside the human
body. And I was like “Oh, VirtualiTeach! That’s a really good
play on words!” And a couple of people I said it to was like,
“No, don’t use that. It’s too confusing.” And I was
stubborn and decided to stick with it.
Alan: I like it. So Steve, on
the VirtualiTeach site at the very top, there’s an image of you
sitting with a bunch of people, but you’re in avatars. What’s that
all about? What platform is that?
Steve: So that’s one of the
images from the CBD and VR events that I host. It’s ENGAGE, by
Immersive VR Education They’re the guys that made Apollo 11 and
Titanic VR. The funny thing is people tend to think that I work for
Immersive, which I totally don’t. I’ve never worked for Immersive.
I’ve got a lot of love for these guys, they’re great guys, and I work
closely with them. But ENGAGE, it just is the platform that I chose
and continue to choose to use for my professional development events
inside VR. So I kick them off. Just before I launched the website in
mid-2017, when I first got my Vive, I’d been lucky enough to have a
sort of guided tour of the ENGAGE platform from Dave Wiehl and the
CEO of Immersive VR Education, and was blown away. It was the first
multi-user VR platform that I’d ever been inside and obviously —
especially back then — it was the only one that was dedicated to
education, and it was just stunning. And I was conscious of the fact
that schools weren’t necessarily in a position to be harnessing this
technology with a whole class full of students at that point — and
to a degree, most schools are still not right now — but a lot of
schools had begun to invest in maybe one or two Vives or Rifts or you
know, educators like me, who is a bit techie, had gone out and bought
their own one. So I decided in June 2017 to test the waters and offer
out this free professional development event inside virtual reality.
For clarity, one thing I hear quite often — just like I hear the
confusion with the VirtualiTeach name of the website — in Europe and
in the UK in particular, it tends to be referred to as CPD, for
Continuing Professional Development. And quite often I get American
educators asking me, like, what CPD stand for? Because in the States
you tend to just refer to it as Professional Development or PD. And
you know what?
Alan: Not cardiopulmonary
disease?
Steve: [chuckles] Exactly. Or
some sort of medical procedure. But at the end of the first year, I
considered rebranding it to “PD in VR”. And at that point,
even American educators like Steven Satter were saying to me “No,
don’t change it, man. Don’t change it. It’s got a legacy now.”
And it– something about it, it just didn’t sound right to change it.
So we ended up just leaving it the same. But yeah, so I did that
first session in June 2017. I actually delivered it three times in 24
hours, to three different groups of educators, some of whom accessed
via PC, and some of whom were in headsets. Obviously, there’s a
limit, and even more so back then, there were limits to the number of
avatars that we could have inside the same virtual space. And it was
tons of fun, despite all the obvious technical glitches that you face
when you’re at the tip of the spear. So I decided to start hosting it
monthly, and decided as well to start bearing them up. So I started
doing some where I would deliver presentations, normally virtual
re-enactments of presentations that I was doing locally here in the
UAE or across the Middle East region events. But I also started
hosting panel discussions and fireside chats and had some truly
phenomenal guests on, building up to the one year anniversary show we
did six hours straight last July.
Alan: Wow.
Steve: Opening with Alvin
Graylin from Vive. You know, Charlie Fink was on, and Bob Fine. And
we had the whole virtual reality podcast crew were on there.
Alan: It’s so funny you say
that. We’ve had Alvin, Charlie and Bob as guests.
Steve: Yeah.
Alan: They get around, these
guys.
Steve: They do, yeah. So, yeah.
I mean, beyond that, I’ve had guest from the BBC, I’ve had from HTC
and ClassVR, and all kinds of speakers as well — obviously — as
educators. And then after that first year, we were trying to come up
with a new idea for another kind of variation on it. So I kind of had
this idea to host a chat show. So I do this live from Dubai chat show
format under the CPD in VR banner, which I’ve hosted about half a
dozen of those. And actually, I’m in parallel to taking a little
break with the website, I also took a little break from the CBD in VR
events, but just did the kind of comeback show about two weeks ago,
which was kind of laughingly jokingly dubbed the Season 3 premiere,
because this is the start of the third year of me doing them now. And
that was the live from Dubai format. I had Steve Grubbs from Victory
XR on and–
Alan: Grubbs’ been on my show,
too.
Steve: Yeah, he’s a great guy.
Yeah. Dr. Angelina Dayton, The VR Lady, who does a lot of amazing
work with the Cherokee Nation, and now the Navajo Nation as well. And
Suzanne Lee from Pivotal Reality up in Scotland, too, who does some
stunning work with elderly people with dementia. And that kind of
encapsulates my approach when I’m doing the panels — or now the chat
show format, as well — is try and get a broad spectrum. Somebody
said to me once, “Why are you not just getting educators on?”
And to me, there needs to be a lot more dialogue between developers
and educators, and there needs to be a more open door approach to
helping developers understand what it takes to make something that’s
effective in the classroom, which is something I wrote about last
year for VR Focus. You know, my advice to to developers, if you’re
creating an educational app, how is this actually going to work in a
classroom setting? You might make the most beautiful experience of
all time, but if I can’t harness that effectively with a classroom
full of kids, then it is not something I’m going to go back to. It’s
not something I’m going to be able to integrate successfully in the
classroom.
Alan: Absolutely. It’s almost
like you have two camps, you have educators, and then you have
technologists or technology providers. And you need both to deliver
content in a way that makes sense in schools. What are you seeing, as
far as adoption of this on a broader scale. Is this something like–
if you looked at– are we’re gonna have VR in every school? Is that
in five years? Is that ten years? Is that ever?
Steve: Obviously, my viewpoint
is somewhat skewed, because I’m based in Dubai. I would underwrite
that with just because I’m in Dubai doesn’t mean everyone’s
absolutely minted. You know, schools here don’t necessarily have
endless piles of cash just because we’re based here in Dubai, like
the GESS group I worked for, was not a for-profit organization, as a
lot of the older schools are. So there are obviously somewhat limited
resources. Schools in general are very cash poor organizations and
they’re also reticent to jump on to what can be perceived as
bandwagons. That being said, I mean, at least here in the UAE, what
I’m seeing is more and more schools dipping their toes in the water.
A friend of mine works in a kind of technology director role across a
large school group in Abu Dhabi here in the UAE. And they’ve just
rolled out the ClassVR solution across all of their campuses. Two of
the schools that I’m currently working with through my consultancy
that have already deployed ClassVR as a solution as well. Obviously,
some of the schools have looked already into the higher end solutions
like Vives, and they may be invested in two or three. And another
school that I’m working in, they’ve already marked out a space
specifically to be a VR lab, painted big murals of VR headsets and
stuff on the wall, but they were conscious of the fact that they
needed to make sure that their Office365 deployment and their day to
day technology integration was firmly deployed and all accounted for,
before they then look at how they’re going to deploy, and what type
of VR they’re going to deploy. And a lot of it comes down to the
various frictions, as Alvin Graylin refers to them, frictions of VR
in education, whether it be the costs or the fear of adopting
something that is outdated within a year, or the health and safety
concerns, and various things like that are potentially holding
schools back right now. I think cost is a huge one, especially when
you know that the PCVR solutions rely on–
Alan: See, let’s lay out the
factors here, and how — in order to be kind of useful to people
listening — what are some of the major factors hindering the
adoption, and what are some of the steps that schools and school
groups can do to overcome that?
Steve: Ok. I’m going to cheat
here, and I’m going to open my website in front of me [chuckles]
Because one of the presentations that I did at the biggest education
event in the region. So the biggest education event in the Middle
East is an event called GESS Dubai, which — as you can guess — is a
global brand now, they’ve got events in South America, in Indonesia.
And I’ve got a great relationship with these guys, I’ve presented
there for probably about seven, eight years now. And this year
actually hosted a whole VR stage for three days straight. We did a
lot to Tilt Brush demos and hands-on workshops and stuff.
Alan: Cool!
Steve: But my keynote
presentation, my prestige presentation — I always try and give them
something brand new, some sort of theory content — was this session
around the five key barriers to VR adoption in schools, and how to
start breaking them down. And I’ve since published this on my site
back in March. So if you go on VirtualiTeach, you can access this.
Alan: It’s virtualiteach.com.
Scroll down about halfway the page. “The five key barriers to VR
adoption in schools, and how to start breaking them down.” Go,
Steve.
Steve: You like my wrecking ball
graphic?
Alan: I love it. It’s amazing.
Steve: So. Right. So number one,
lack of understanding, OK? The simple fact that people just don’t
really understand what virtual reality is and what it can do. And I
think paired with that, there was a… I can’t remember who published
it, but it was a really interesting article earlier in the year. I’m
sure you read it, Alan, about whether or not in the long run the
Google Cardboard did more harm to the VR industry than good. Because
it was brilliant entry-level device, but it then contained what
people’s perceptions of what virtual reality was capable of. And I’ve
had so many people that, you know, you put a VIVE on them for the
first time and the words that come out of the mouth, I’ve heard the
same sentence multiple times, “I didn’t even know this was
possible.” And that’s partly because people’s understanding of
VR is, “I can look around to 360 image. I can potentially look
around a 360 video.” And that misconception that’s been built
through the use of mobile VR, it kind of needs to be unpicked. You
cannot explain virtual reality. It’s experiential technology. You’ve
got to put people in the headsets, and particularly in schools,
you’ve got to put people in headsets who are the game-changers,
they’re the people that have the sway to actually implement change.
One of the first people that I stuck the headset on was was Mark
Steed, who I met before he was the director of the school. I mean, we
were blessed in that Mark was the director. And he’s a very, very,
very tech-savvy guy. And he’s open to new technologies. But I stuck
Mark on the plank — as I want to do for most people that come to me
who want to try VR for the first time — stuck him on the plank. This
is a guy with three master’s degrees and he couldn’t walk out on it.
Two years later, he wrote a blog article for the Tez in the U.K.
about that experience. How much that affected him and how that moment
made him realize the power of the medium in general, because he had
that visceral, emotive reaction to it, which is not something he’d
ever experienced from a different form of technology before.
Alan: It’s interesting that you
talk about that particular one, because we actually built a training
scenario where you’re in a warehouse and you have to go across a beam
and turn off a power supply. And when we built it originally, the
beam was three feet across and you were maybe 20 feet off the ground.
And it wasn’t scary at all. We decreased the beam to one foot and
increased the height to 30 feet, aAnd wow. It is terrifying. And
everybody who goes across it is just tiptoeing across. And it’s this
mind melt because you’re walking across something — you know you’re
safe because you’re in a room — your brain can’t comprehend the fact
that you’re 30 feet in the air; you can’t decipher between reality
and not. And at that point when, in our particular instance, you
actually fall — you don’t fall, you just kind of fall and then it
goes black and you start over again — and it says, “don’t
forget to put your safety gear on.” And it’s in that moment
where you’re like, “man, I will never, ever forget to put my
safety gear on again.” Terrifying people, it turns out, is very
good education!
Steve: From the psychology point
of view, the one thing that I’ve seen again and again and again as
well, which — I remember reading Jeremy Violence’s book Experience
On Demand — and in that first chapter where he’s talking about
putting Zuckerberg on the plank experience at Stanford, and he refers
to something that I’d seen in person myself; the fact that you can
have a group of people standing there watching someone doing the
plank experience and laughing at them and going like, “oh,
what’s wrong with you? You know it’s not real,” blah, blah,
blah. And then they put the headset on themself, and even though
they’ve seen it from a third-person perspective, once they are in it
from a first-person perspective, the subconscious takes over and have
their reaction to it is completely their own. It doesn’t matter that
they’ve already seen somebody else do it. One of them, in fact, one
of the early sessions that we did with the first five I got — which
if you go in, you can find it on my site. If you scroll right, right,
right back to that to the earliest articles, it’s one of the first
ones on there. But also the video is linked on that five blog that I
mentioned earlier on — so I took the plank experience into the sixth
form psychology department at GESS and along with my friend Dr.
Joseph Bell, who is one of the psychology professors there. We pull
out of 16-, 17-, 18-year-old students through the plank experience.
And we captured footage of them going through the experience as well
as their reflections afterwards, and Joe provided some commentary in
terms of what was happening from a psychology perspective. It was
absolutely fascinating. The other thing, in terms of that that I find
fascinating, is that — and I wish I was talking to Dr. Sara Jones
the other day because she’s actually writing a book about VR, and
somebody I’ve known for a number of years — I was saying to her, “I
wish I had carried through with this idea that Joe and I had to do
this study, because I probably find that more adults can’t do the
plank thing.” Kids will walk out on that plank generally with
not a fear in the world, whereas adults, there’s a good proportion of
adults that just cannot do it. You know, my dad is a builder. He does
loft conversions, so he takes people’s loft — or attics, whatever
you want to call them — and he turns them into additional rooms. He
spent his life on roofs. He’s been spending his life climbing up
ladders and walking across roofs, and he couldn’t do that plank
experience at all. Couldn’t do it.
Alan: Wow.
Steve: Which I found
fascinating.
Alan: That’s weird.
Steve: Yeah.
Alan: Makes no sense.
Steve: You’d think so. Steering
back towards what we were saying about the kind of barriers; the
second one that I covered was the cost and the ROI — the return on
investment. And we kind of touched on this already. The idea that,
you know, schools are reticent to invest in technology that they
potentially see as just another thing. And you’ve got lots of
different vendors and companies trying to hawk their wares, so to
speak. You know, STEM is massive. Now, you know that everybody’s
like, “do we need to buy robots?” Do we need to buy 3D
printers? Do we need to buy VR content?” And I think the
difference is — I kind of touched on before — is that we’re not
talking about just another gadget when we’re talking about augmented
and virtual reality. We’re talking about the evolution of computing.
We’re talking about the change in the way the technology is
interacted with, full stop. And if you look at that article, you’ll
see an image that I painstakingly sourced from Google for the
original presentation, which is a group of students from the… I
think, from the 80s. At this single PC, which, you know, this was the
experience that I had initially when I was in primary school. That
there was one computer in the school, and it was on a cart and it was
wheeled into “oooohs” and “aaaaaaahs”, because
the magical computer’s here and–
Alan: I’m not going to ask how
old you are, but I still remember the first time I went to the
library and there was three Mac computers. It was like, “Whoa!”
[laughs]
“So awesome!”
Steve: Ataris and stuff. But I
specifically chose this image because I think this is the fear of a
lot of schools, is that, “well, we’re gonna buy– we can only
afford one, we’ll buy one. And then we’ll end up with the whole class
sitting round, watching one person interact with it.” My counter
to that is, no, you won’t. Because as educators, we now understand
better. Pedagogy has evolved and we understand that that isn’t what
you should be doing with technology. You shouldn’t have a group of
kids all sitting around watching one person interact with the
technology. There are so many more ways. We have evolved in terms of
our use of technology in the classroom and our understanding of
digital learning has evolved so much, especially since the advent of
tablets and the first deployments of iPads in the classroom. We’ve
learnt new ways to integrate limited amounts of technology and
explored bunches of different ways where, I’ve maybe had half a dozen
mobile VR headsets deployed for an activity in parallel to a VIVE or
mixed reality headset. I’ve had work where students are not just
taking turns, but they are specifically paired for a reason; so that
one person’s the hands in the physical space, and the other person,
they’re immersed in the virtual experience. That was kind of my
counter to that. Paired with that is the other one that we mentioned
before, which was this rate-of-change fear, and the graphic that you
see on the site there, which I always give a shout out to my — sadly
— dearly departed friend Chris Long, who died earlier this year.
Chris originally showed me this graphic of Martec’s law. It was
during the one-year anniversary event for the CBD and VR. He
delivered a presentation and this was part of it. And it was one of
those graphics, where I saw it and I was like, “this makes
perfect sense. How have I never seen this before?” And I’ve
started using it in a lot of presentations–
Alan: I’m actually stealing this
for my presentations. Thank you, Chris!
Steve: Essentially the premise
is that, because technology changes at this rapid pace, this
exponential rate, but organizational change tends to be slower. What
happens is that the gap between the two increases over time. And
ultimately, if a company sits and procrastinates for too long, the
gap between the state of technology and the organization and their
use of technology becomes so large that an organization can actually
need a full reset. And you’ve seen this. You see it in organizations
all the time. It might mean that they have to lay off a whole bunch
of staff. It might mean that they need to source a huge chunk of
investment to catch up. And this was something that I really
learned– I mean, I learned so much working for Mark Stevens, in
terms of digital strategy and vision. He came into GESS, which was
incredibly ahead of the time in terms of technology you use in the
classroom. But he came in and was like, “well, look, you’ve got
no refresh cycles built in for your tablet or your whiteboards or
this. There’s no standardization.” That was what he ended up
bringing to GESS, was a much better strategy, and a much better
plotted roadmap in terms of technology integration at the school, and
where the school was at and where the school was going. So then the
fourth one — the fourth barrier to the VR adoption, the fourth
common thing — because just for context, again, this article, this
presentation was based on the conversations that I’d had both at GESS
and with other educators worldwide, and with people that I’ve been
using VR and their feedback, in terms of the barriers that they were
hitting. So this wasn’t me just plucking these out of thin air. This
was based on common concerns that were being thrown at educators.
People will come to me and say, “I’m being asked about this.
What do I say?” And health and safety concerns is obviously
something that was coming up quite a lot. “Is VR say for kids?
How long should they use it for? Will it hurt their eyes? What age
should kids start to interact with VR?” I wrote a piece off the
back of the Commonsense Media Report contesting some of the data that
they used, and not because they were talking about content concerns,
saying that parents had concerns about sexual content or violent
content in VR. OK, I can understand that concern.
Alan: Yeah.
Steve: Yeah. You’ve got an
infinitely higher probability of finding that type content on
YouTube, which, these same parents have got their kids sitting on
their phones with unrestricted access to YouTube because they have
got parental restrictions set up. So I thought that was somewhat
ironic. “You know, too much time with VR is bad for you.”
Well, too much time with anything is bad for you. You know? It’s like
me saying eat too many cakes is bad for you. If I read a book for six
hours straight, I’m gonna get eyestrain. Too much of anything new, an
educated person will tell you that too much of anything is bad for
you. Social isolation is another one that was coming up. And, you
know, until we get into that place where the social multi-user VR
experiences are more common, I think that, you know, there is some
credence to that. It’s just a very insular thing when you’ve got a
headset on and you know, if you’ve got a room full of people.
Alan: To be honest, my kids have
unlimited access to VR and they — I mean, we’ve got a quest, we’ve
got Vives, we’ve got Hololens, Magic Leap, we have all the toys —
and they don’t want to go in it. And when they go in it, they want to
go in with other people. They want to be in social VR, which makes
sense if you’re playing Beat Saber or something, but it’s really
great when you’re with other people and then engage the platform, or
these types of things. So I think socialize– So– that’s really hard
to say! Social isolation.
Steve: Yeah. Yeah. I mean–
Alan: Say that five times fast.
Steve: These are things that I
pulled out from there, from the Common Sense Media Report.
Alan: Now, the next one is the
health concerns of bumping into something, that is actually probably
one of the ones that I think actually could be a problem; if your dog
walks in the room when you’re in VR and you trip on the dog or hit a
table. I’ve seen people get punched in the face.
Steve: Yeah. I mean, devil’s
advocate: I approach this from an educator’s perspective. And if you
want my take on the Common Sense Media Report, that article is an
in-depth breakdown of the whole report and my response to it is there
on the site. And if you look at the theory section, you find out,
yeah, 13 percent have bumped into something. My response is, of
course they have. If you’re not watching kids and they’ve got a VR
headset on, of course they’re going to bump into stuff. I’ve had kids
using mobile VR headsets where there was no, you know, the three
degrees of freedom headsets where they can’t move anyway, and they
still stand up and start walking. You have to monitor kids using this
kind of technology. If you don’t monitor them, again, yeah, of course
they’re going to bump into things. But I’ve used VR headsets with
hundreds and hundreds and hundreds of kids. And I’ve never had a kid
bump into anything because I control the situation. You don’t just
stick them in a Vive and then walk off and make a cup of tea.
Alan: It’s interesting. We had
an event a couple of days ago, we just got — this is two months ago
— we had just got the Oculus Quest and we had an event and people
were playing the sword fighting and… it was our fault, we didn’t
put a barrier around them, but somebody walked by as they were
swinging. Got punched right in the face. And I was like, oh, jeez,
that happened. But that was our fault for not putting a personal
physical barrier around the person, even though they have a digital
one. So when the person inside didn’t know… it just was our fault
and it was only… because we use the Vives, we normally set up an
extension so that people can’t get in to the person. But because the
Quest was so new and we didn’t know, we just put them in a room and
assumed that nobody would walk near them. Somebody swinging their
arms around like an idiot. You would think people wouldn’t walk near
them. But that happened.
Steve: Yeah. I mean, one very
low-cost solution I’ve seen people start to implement is they bring
in — and you can actually point ones that are marketed as
specifically for virtual reality now — but they essentially put mats
of some kind on the floor. You get students to take the shoes off.
You’ve got very simple haptics; I can feel that I’ve stepped off of
the mat, so therefore, I need to stop. And obviously making sure that
you’ve got, whether you call it your guardian or a chaperone, you’ve
got that that set up properly. I really like the Quest, obviously,
the way that the passthrough camera kicks on once you step outside
the guardian, because it draws that full stop. I’m outside of my
space. But again, little things like if you’re marking your
chaperone, don’t take it right to the very, very limits of the space
that you’ve got. So there it goes right next to a pane of glass or
right next to a solid brick wall. Give yourself a kind of a border so
that if somebody does get lost in the moment and you happen to not be
watching them, that they’ve got that kind of leeway, that little bit
extra more… I mean, we could do a whole podcast just around this
kind of VR and health and safety and stuff with kids. But a lot of it
comes down to common sense, ironically, considering this was a
response to the Common Sense Media. It comes down to limiting the
lengths of experiences and limiting, obviously, moderating the types
of experiences that you’re using with students. One of the things
that Mark and I started to put before we both left GESS, and once the
dust settled in both moves, we do hope to pick up was we were trying
to put together a kind of white paper and look at sort of creating an
actual formal approach to this. What age would you potentially start
using a Google Cardboard with a student? And how long would that
experience maximum last? What about a Windows mixed reality headset?
Vive? If I’m working with a 13 year old, what would be the
recommended maximum length of an experience? Especially with
something like Tilt Brush, I’ve seen it happen. You stick a kid in
and they get lost in that world. Well, do you pull it out and say,
how long do you think you been in? And then say “about five
minutes” and you say, “oh, you’ve actually been in there
20.”
Alan: Time dilation is a proven
fact in VR, actually. I was reading a study on this early on and the
time dilation can be as much as 25 percent that people think they’re
in VR a lot shorter than they actually are. And one of the things
that you can actually do to completely mess with people is you can
put a virtual clock inside VR that moves slower than real time and
you can actually increase that to about 50 percent. So people think
they’re in for an hour and they’re in for two.
Steve: I’ve never heard of that
done before. I like that. That’s nice. So then the last one is, it’s
kind of the biggie, from my point of view, is that the benefits alone
and ultimately, as somebody who’s worked for a long time now with
various forms of education technology, there’s gotta be benefits to
learning. There’s gotta be some point to deploy in this technology.
When I when I first started doing this stuff, I was doing with with
the high end VR in early 2017, the kind of party line for myself and
other pioneers like Jamie Donnelly and Steven Sarto, it was like
“there hasn’t been enough studies yet. The jury’s out,”
kind of. But we’re two years on and the jury is starting to come in
now and we’re starting to see more and more evidence come in from all
corners of the world. There’s all kinds of data that you see there on
the site. From Beijing University and Warwick University and Cornell
and Stanford, there’s all these studies taking place showing that no,
VR is more engaging than than other traditional forms of media, that
it leads to significant retention of learning. The one that came out,
Alvin Graylin tagged me in a tweet last week. There was a new study
from another university in China looking at–
Alan: Yes, I saw that, too. I
asked him for the study. I didn’t get it yet.
Steve: Now, I haven’t seen the
actual study, but with statistics looking at language learning in VR,
and then from a theoretical point of view, I then start thinking
about some of the big theoretical educational models, and the one for
me over the last decade that has become very prevalent is the SAMR
model from Ruben Puentedura. This is going to become like, if you’re
playing ed-tech bingo at conferences, the first thing that you would
put down on your bingo card would be the SAMR model on a slide
because you can’t get through any sort of ed-tech presentation these
days without somebody going into the SAMR model, one talking about
the different levels of SAMR. When I was putting this presentation
together, this is the first time I’d actually looked at SAMR for a
while and the first time I ever looked at it with specifically my VR
head on. And for those that maybe don’t work in the education
industry, this is a two-phase, four-step model created by Dr. Ruben
Puentedura specifically about technology integration in the
classroom. The lower phase, the enhancement phase, has the two steps,
the lowest step is substitution, where technology acts as a direct
substitute; type something on Word rather than you hand-write it.
Then it goes augmentation, so your technology’s still a substitute,
but, you know there’ll be an additional functional improvement. So
you’re using a digital thesaurus, or you’re adding in a clipart image
or something simple. And then the second higher phase is the
transformation phase, which has the modification and ultimately
redefinition. And the idea being that you’re moving towards using
technology to create new tasks and access content in ways that was
previously impossible. You take the lowest level, use that analogy of
having to write something; the lowest level would be just, I type on
word instead of handwriting it. And highest-level might be that I
collaboratively write something on a Google doc and then we post it
to a blog or a wiki, and then we take live feedback from people from
around the world. That’s me redefining that task, using technology.
And when I looked at this through the VR goggles — pun intended —
when I looked at it with my VR kind of mindset, what I found
interesting was that it’s like a leap frog, because I can’t see any
instance where VR is being used for things like substitution, because
by its very nature, because of the experience or nature of VR
applications, you’re automatically giving students the ability to do
things that they had previously never been able to do before. Whether
that’s painting with light or defying gravity with the way that you
build a sculpture and tilt brush, or flying around the world in
Google Earth, or stepping back in time in the Titanic experience.
It’s a transcendent technology in a way that other technologies have
not offered before. And I thought that was quite telling. I’m
actually at the moment working towards finishing something I started
in early 2018. So nearly two years ago, I started looking at VR and
another very famous education theory, which is Bloom’s taxonomy and
Chris Long, who sadly no longer with us. I, myself and my friend Alex
Johnson in India and Stephen. So we were kind of informally looking
at this on a shared document and we were going backwards and forwards
with it and it kind of got put on the backburner a few times. And
then Chris Long and I dive back into it maybe six months later and we
really thought that we had something there, something interesting —
something somewhat controversial — but interesting nonetheless. And
then it went back on the backburner. But now I kind of feel like I
need to finish that. I need to get that published, not least because
it was kind of the last project that I was working on with Chris. And
I’d like to see that through to completion. So 2020, I think at this
point, will be when I’m looking to publish that, because as I
mentioned earlier, I’ve got to rebuild my entire website before I can
publish anything.
Alan: Well, Steve, I mean,
there’s so much we can talk about. It’s been really amazing to listen
to these. You are one of the world leaders in this. So thank you so
much for joining us today.
Steve: Thank you, Alan. It’s
always a pleasure.
51 min
February 25, 2020 Creating a Dialogue Between Innovators and Educators, with VirtualiTeach’s Steve Bambury
Using VR in the classroom is a no-brainer. It’s immersive tech, and can teach kids in new, innovative ways. But if the people developing the technology don’t understand how kids’ brains learn, it’s not going to take, no matter how innovative. VirtualiTeach’s Steve Bambury drops by to explain how he’s trying to bridge that gap.
Alan: Hey, everyone, my name’s
Alan Smithson. Coming up next on the XR for Business podcast, we have
Steve Bambury, founder of VirtualiTeach. We’re gonna be talking about
digital literacy, the virtual/augmented reality platforms, and the
question on everybody’s mind: What are the key barriers to adopting
VR and AR in schools and how to overcome them? All this and more,
coming up next on the XR for Business Podcast. Welcome to the show,
Steve. How are you?
Steve: I’m good, man. It’s good
to speak to you.
Alan: It’s really great. The
last time we saw each other, we were in Dubai — where you live —
and you took me to the Dubai Mall, and we went in and we went to the
VR Park, the giant VR Park. And I was just blown away by how big and
ostentatious everything was. And it was a really great experience. I
can’t thank you enough for your warm hospitality in Dubai. But today
it’s all about you. So let’s talk about what you’re doing, and how
did you get into this? And what are you doing now?
Steve: I’ve been in Dubai for 11
years. And for those 11 years, I’ve always worked at the same school.
I was working a school group here known as GESS — which is the
acronym for Jumeirah English Speaking School — also broadly referred
to as GESS Dubai now. GESS is one of the leading schools in the
Middle East. It’s a very old school, at least in terms of
international schools in this region. It’s only, I think four years
or three years younger than the UAE as a country. So it is very well
established. And yeah, so I worked there for 11 years. I worked as a
class teacher in one of the primary schools, and curriculum leader.
Eventually become head of computing at the primary school. So I was
teaching digital literacy and computer science content to four year
olds, 3 to 11 year olds. And I ended up in that role primarily
because of all the work I’ve been doing to integrate the iPads in the
classroom. From 2011, we were one of the first schools in the Middle
East to to roll out iPads in the classroom. And then three years ago,
I moved into a role that was created for me, which was the head of
digital learning and innovation, working underneath the new director,
Mark Steed, who’d just come in from the UK. Mark had the pedigree in
terms of digital learning from what he’d done at this very, very
prestigious school in the UK called Berkhamsted. He’d also chaired
the Independent Digital Strategy Group for eight years there. And so
Mark created this role and this role took me out of the classroom
most of the time. A lot of it involved training with staff. It also
involved going back into departments and helping them with enrichment
projects. And it was kind of in parallel to that. I mean, part of the
reason that my work with virtual reality really took off is because I
moved into this new role, and had this freedom to innovate and to
explore new technologies. My first VR headset was just a [garbled]
headset I imported from the States in 2014. But it was not long after
I started this new role as head of digital learning at GESS that I
got my first Vive. I took that Vive into the school and started
looking for ways to integrate it into different curriculum areas. In
actual fact, I’ve just recently started writing a series of guest
posts for Vive on the Vive blog. You can go into Google, like “HTC
Vive blog Steve Bambury” or something, you’ll probably find
them. But I’ve been writing a series of blogs about my journey using
and integrating the HTC Vive headsets at GESS. The first one focused
on all those initial trials that we’ve run. My background is in film.
So before I was a teacher, I worked in the film industry. So I’ve
always done a lot of film projects with kids. So one of things that I
was doing throughout our journey with virtual reality was documenting
all of the projects that we were carrying out. I was capturing
student voice. I was capturing staff feedback. And I think that was
invaluable in terms of the success of our deployment of higher end VR
at GESS, because it enabled me to then use these — not only
internally but also externally — to promote what we were doing at
the school. But internally it meant that I had this evidence that
there was definite impact in terms of the integration of this
technology, and the results that were tangible when you were using
the Vives in the classroom. And then from there, we ended up
investing in more Vives. We brought in Acer mixed reality headsets,
as well. I ended up doing trials with the Vive Focus and other
hardware. The one thing obviously that is already clearly missing
from this mix, is the word Oculus. For those that are outside of the
Middle East region, just for context, Oculus has next to no presence
in the Middle East at all. The Rift never launched here. The Go never
launched here. The Quest has launched here, but only in shops. 100
percent markup price from the Europe and US price.
Alan: Oh wow.
Steve: So it’s been– yeah. I
mean, Facebook are here, but they don’t seem to value the region, so
everything tends to be Vive, with the WMI headsets sort of in the
wings somewhat. So yeah. So I did a lot of cool stuff with VR and
started becoming kind of known for the work I was doing with VR.
Couple of years ago I set up VirtualiTeach as a platform to share
best practice and share my ideas, my theory, my projects. And…
yeah. Broadly that became all I ended up speaking about at
conferences around the region and internationally. I was becoming
“the VR guy”. So–
Alan: You are the VR– You’re
the VR education guy!
Steve: So yeah. And it is
something that I’m really passionate about, and it’s something that I
see the future of education, in terms of where spatial computing will
take us. It isn’t just another gadget that schools need to consider
weighing up buying into or not. This is the evolution of computing in
general. So yeah, so I did that role — the head a digital learning
— for three years. And at this point I’d worked for the company for
11 years, and I was looking for the next step. And Mark — the
director — he was headhunted to go off to Hong Kong and work in Hong
Kong, which he now has. And so in parallel to that, I decided to
strike out on my own and set up my consultancy, which is broadly how
I’ve spent my summer. Normally in the summer, everybody leaves Dubai,
because it’s so ridiculously hot. And there’s so many expats here —
80 percent of the country is expats — so everybody leaves and
Dubai’s a ghost town in the summer. Very hot ghost town.
Alan: Oh, it’s so hot. Oh my
God. I was there this summer and oh my God…
Steve: So I was stuck here this
summer, because I had to deal with all the logistics and the
paperwork of getting my company license, my new visa, and everything
like that. And yeah, and this is me now, a couple of months in,
working for myself the first time in my life. I set up my consultancy
called Digital Inception. Anecdotally, the name refers to– it was
actually the name of a remote keynote presentation that myself and my
friend Luke Reece delivered in 2013, I believe, to the University of
Southern California from here in Dubai using the Nearpod platform.
And the presentation kind of riffed on the movie Inception and the
ideas from the movie Inception, and the idea that if professional
development is good enough that you walk away, and you feel like the
idea was there all along, that you already knew it, it isn’t
something you’ve been preached, you’ve had something unlocked that
was already within you. And I always kind of liked that title, I
thought it was cool. And it was fitting that I could use it to
encompass the work that I was going to be doing with immersive
technologies, but also broad enough to encompass some of the other
areas that I work with. I mean, I was a distinguished educator. I do
a lot with Apple technology. I’m a Microsoft master trainer, so I do
a lot of work with Microsoft technology. So I needed something that
was kind of broader to encompass all of those things. So yeah, that’s
where I’m at.
Alan: [chuckles] So you’ve done
all of these things. You’ve been a pioneer. You also do the CPD talks
where you interview people. So by all intents and purposes, you are
an expert. Well, I would say the world’s leading expert on VR and AR
in education. And you also have VirtualiTeach. What is that platform
all about? Was that just a repository, a place for you to kind of
store everything that you saw? And then it became a website. How did
that happen?
Steve: Yeah, kind of. So in
2012, when I was doing a lot of work with iPads in the classroom.
Myself and my friend Luke — who I mentioned just now — who also
works at JESS with me. And he’s still at GESS, he’s one of the deputy
heads there. We were speaking at events, and we were getting loads of
people asking us questions and stuff. So we figured it would be a
good idea to tackle a website where we could share our ideas and
direct people to. So we set up a website called iPadEducators.com,
which went on to win an award, and it led to both of us becoming
Apple distinguished educators. And VirtualiTeach is– the approach is
broadly the same, both of these websites are completely not for
profit. I turn down advertising offers on a weekly basis. It’s never
been about making money. It’s just about sharing best practice, which
outside of the education industry, some people see it’s kind of
weird. If you’ve got ideas why you’re not charging for them? But
educators as a community, we’ve always been very, very much of the
kind of pay-it-forward mentality, that you share what you’re doing,
and we grow together. To be honest, it was 2017. And my youngest
daughter had been sick. And so the summer plans were canceled that
year as well. And I– essentially I was considering writing a book.
Ironically, the book was going be called Digital Inception. I was
going to write a book about all this stuff I was doing with virtual
reality, but the more I considered the time that I would need to
spend writing a book, and the delay in this kind of gratification of
sharing the content with other people. In the end, I just thought,
you know what? Let’s just put another website together, because I can
start putting content out quickly. And I like to produce content.
Alan: Writing a book. By the
time I read the book, first of all, it’s obsolete. Second of all, you
can’t keep it up to date. So we’re struggling right now. We’re
writing a book and– two books, “XR for Business” and “XR
for Education”. And by the time they’re finished, they’ll be 100
percent obsolete.
Steve: Yeah. Yeah. You know why?
In fact, last year in the end, I did put a book proposal together and
submitted it through a colleague to a publishing house. And that’s
basically what they come back to me and said. They said, “Look,
we like your idea. But by the time when you actually write this and
we publish it, we’re conscious of the fact that it will be out of
date.” And to a degree that’s true of all books based on
technology, because everything’s moving forwards.
Alan: Yeah.
Steve: So, yeah. So that’s
basically why the website became about. Now, I will give a little
caveat to anyone that’s listening, around about the time that we’re
recording this, back end of October 2019. So with everything that was
going on, with me setting up the company and various other bits and
pieces, I took a couple of months break from the website, and
unfortunately what’s happened is the website’s designed through Wix,
and over the summer Wix have launched a new blog feature. And
essentially my website is entirely built– the entire structure of it
is built around the old blog feature. And then I use custom feeds to
pull theory content onto a page, and Vive related content onto a
page, and guest articles onto a page. And it was all done with a
tagging system, which has been made redundant with the change to
their site. And essentially, I’m now in a position where I’ve got a
load of new content ready to publish. But at the moment, I’ve got to
find the time to go back and essentially re-label everything that
I’ve ever posted, which is like 150 articles. I’ve got to manually go
into all of them and and re-tag them. So my site’s kind of sitting
dormant at the moment, but hopefully we’ll be back on its feet before
the end of the year. Because I do have a lot of new content ready to
go on. But I have to say, sadly, the old content needs to be
re-tasked first, and I need to do some some layout adjustments as
well, to fit the new style that Wix have deployed.
Alan: I’m on your site now and
you’ve got– this is like the compendium for anybody looking to do.
And it’s VirtualiTeach, but “Virtual I teach”, it could
also be.
Steve: Yeah, that tends to be
what people say to me. You know what? When I was trying to come up
with a name, I’m going backwards and forwards with different ideas.
And I’d been reading a lot around the theory of the virtuality
continuum. And I had obviously the Curiscope Virtuali-Tee, you know,
the augmented reality t-shirts where you can look inside the human
body. And I was like “Oh, VirtualiTeach! That’s a really good
play on words!” And a couple of people I said it to was like,
“No, don’t use that. It’s too confusing.” And I was
stubborn and decided to stick with it.
Alan: I like it. So Steve, on
the VirtualiTeach site at the very top, there’s an image of you
sitting with a bunch of people, but you’re in avatars. What’s that
all about? What platform is that?
Steve: So that’s one of the
images from the CBD and VR events that I host. It’s ENGAGE, by
Immersive VR Education They’re the guys that made Apollo 11 and
Titanic VR. The funny thing is people tend to think that I work for
Immersive, which I totally don’t. I’ve never worked for Immersive.
I’ve got a lot of love for these guys, they’re great guys, and I work
closely with them. But ENGAGE, it just is the platform that I chose
and continue to choose to use for my professional development events
inside VR. So I kick them off. Just before I launched the website in
mid-2017, when I first got my Vive, I’d been lucky enough to have a
sort of guided tour of the ENGAGE platform from Dave Wiehl and the
CEO of Immersive VR Education, and was blown away. It was the first
multi-user VR platform that I’d ever been inside and obviously —
especially back then — it was the only one that was dedicated to
education, and it was just stunning. And I was conscious of the fact
that schools weren’t necessarily in a position to be harnessing this
technology with a whole class full of students at that point — and
to a degree, most schools are still not right now — but a lot of
schools had begun to invest in maybe one or two Vives or Rifts or you
know, educators like me, who is a bit techie, had gone out and bought
their own one. So I decided in June 2017 to test the waters and offer
out this free professional development event inside virtual reality.
For clarity, one thing I hear quite often — just like I hear the
confusion with the VirtualiTeach name of the website — in Europe and
in the UK in particular, it tends to be referred to as CPD, for
Continuing Professional Development. And quite often I get American
educators asking me, like, what CPD stand for? Because in the States
you tend to just refer to it as Professional Development or PD. And
you know what?
Alan: Not cardiopulmonary
disease?
Steve: [chuckles] Exactly. Or
some sort of medical procedure. But at the end of the first year, I
considered rebranding it to “PD in VR”. And at that point,
even American educators like Steven Satter were saying to me “No,
don’t change it, man. Don’t change it. It’s got a legacy now.”
And it– something about it, it just didn’t sound right to change it.
So we ended up just leaving it the same. But yeah, so I did that
first session in June 2017. I actually delivered it three times in 24
hours, to three different groups of educators, some of whom accessed
via PC, and some of whom were in headsets. Obviously, there’s a
limit, and even more so back then, there were limits to the number of
avatars that we could have inside the same virtual space. And it was
tons of fun, despite all the obvious technical glitches that you face
when you’re at the tip of the spear. So I decided to start hosting it
monthly, and decided as well to start bearing them up. So I started
doing some where I would deliver presentations, normally virtual
re-enactments of presentations that I was doing locally here in the
UAE or across the Middle East region events. But I also started
hosting panel discussions and fireside chats and had some truly
phenomenal guests on, building up to the one year anniversary show we
did six hours straight last July.
Alan: Wow.
Steve: Opening with Alvin
Graylin from Vive. You know, Charlie Fink was on, and Bob Fine. And
we had the whole virtual reality podcast crew were on there.
Alan: It’s so funny you say
that. We’ve had Alvin, Charlie and Bob as guests.
Steve: Yeah.
Alan: They get around, these
guys.
Steve: They do, yeah. So, yeah.
I mean, beyond that, I’ve had guest from the BBC, I’ve had from HTC
and ClassVR, and all kinds of speakers as well — obviously — as
educators. And then after that first year, we were trying to come up
with a new idea for another kind of variation on it. So I kind of had
this idea to host a chat show. So I do this live from Dubai chat show
format under the CPD in VR banner, which I’ve hosted about half a
dozen of those. And actually, I’m in parallel to taking a little
break with the website, I also took a little break from the CBD in VR
events, but just did the kind of comeback show about two weeks ago,
which was kind of laughingly jokingly dubbed the Season 3 premiere,
because this is the start of the third year of me doing them now. And
that was the live from Dubai format. I had Steve Grubbs from Victory
XR on and–
Alan: Grubbs’ been on my show,
too.
Steve: Yeah, he’s a great guy.
Yeah. Dr. Angelina Dayton, The VR Lady, who does a lot of amazing
work with the Cherokee Nation, and now the Navajo Nation as well. And
Suzanne Lee from Pivotal Reality up in Scotland, too, who does some
stunning work with elderly people with dementia. And that kind of
encapsulates my approach when I’m doing the panels — or now the chat
show format, as well — is try and get a broad spectrum. Somebody
said to me once, “Why are you not just getting educators on?”
And to me, there needs to be a lot more dialogue between developers
and educators, and there needs to be a more open door approach to
helping developers understand what it takes to make something that’s
effective in the classroom, which is something I wrote about last
year for VR Focus. You know, my advice to to developers, if you’re
creating an educational app, how is this actually going to work in a
classroom setting? You might make the most beautiful experience of
all time, but if I can’t harness that effectively with a classroom
full of kids, then it is not something I’m going to go back to. It’s
not something I’m going to be able to integrate successfully in the
classroom.
Alan: Absolutely. It’s almost
like you have two camps, you have educators, and then you have
technologists or technology providers. And you need both to deliver
content in a way that makes sense in schools. What are you seeing, as
far as adoption of this on a broader scale. Is this something like–
if you looked at– are we’re gonna have VR in every school? Is that
in five years? Is that ten years? Is that ever?
Steve: Obviously, my viewpoint
is somewhat skewed, because I’m based in Dubai. I would underwrite
that with just because I’m in Dubai doesn’t mean everyone’s
absolutely minted. You know, schools here don’t necessarily have
endless piles of cash just because we’re based here in Dubai, like
the GESS group I worked for, was not a for-profit organization, as a
lot of the older schools are. So there are obviously somewhat limited
resources. Schools in general are very cash poor organizations and
they’re also reticent to jump on to what can be perceived as
bandwagons. That being said, I mean, at least here in the UAE, what
I’m seeing is more and more schools dipping their toes in the water.
A friend of mine works in a kind of technology director role across a
large school group in Abu Dhabi here in the UAE. And they’ve just
rolled out the ClassVR solution across all of their campuses. Two of
the schools that I’m currently working with through my consultancy
that have already deployed ClassVR as a solution as well. Obviously,
some of the schools have looked already into the higher end solutions
like Vives, and they may be invested in two or three. And another
school that I’m working in, they’ve already marked out a space
specifically to be a VR lab, painted big murals of VR headsets and
stuff on the wall, but they were conscious of the fact that they
needed to make sure that their Office365 deployment and their day to
day technology integration was firmly deployed and all accounted for,
before they then look at how they’re going to deploy, and what type
of VR they’re going to deploy. And a lot of it comes down to the
various frictions, as Alvin Graylin refers to them, frictions of VR
in education, whether it be the costs or the fear of adopting
something that is outdated within a year, or the health and safety
concerns, and various things like that are potentially holding
schools back right now. I think cost is a huge one, especially when
you know that the PCVR solutions rely on–
Alan: See, let’s lay out the
factors here, and how — in order to be kind of useful to people
listening — what are some of the major factors hindering the
adoption, and what are some of the steps that schools and school
groups can do to overcome that?
Steve: Ok. I’m going to cheat
here, and I’m going to open my website in front of me [chuckles]
Because one of the presentations that I did at the biggest education
event in the region. So the biggest education event in the Middle
East is an event called GESS Dubai, which — as you can guess — is a
global brand now, they’ve got events in South America, in Indonesia.
And I’ve got a great relationship with these guys, I’ve presented
there for probably about seven, eight years now. And this year
actually hosted a whole VR stage for three days straight. We did a
lot to Tilt Brush demos and hands-on workshops and stuff.
Alan: Cool!
Steve: But my keynote
presentation, my prestige presentation — I always try and give them
something brand new, some sort of theory content — was this session
around the five key barriers to VR adoption in schools, and how to
start breaking them down. And I’ve since published this on my site
back in March. So if you go on VirtualiTeach, you can access this.
Alan: It’s virtualiteach.com.
Scroll down about halfway the page. “The five key barriers to VR
adoption in schools, and how to start breaking them down.” Go,
Steve.
Steve: You like my wrecking ball
graphic?
Alan: I love it. It’s amazing.
Steve: So. Right. So number one,
lack of understanding, OK? The simple fact that people just don’t
really understand what virtual reality is and what it can do. And I
think paired with that, there was a… I can’t remember who published
it, but it was a really interesting article earlier in the year. I’m
sure you read it, Alan, about whether or not in the long run the
Google Cardboard did more harm to the VR industry than good. Because
it was brilliant entry-level device, but it then contained what
people’s perceptions of what virtual reality was capable of. And I’ve
had so many people that, you know, you put a VIVE on them for the
first time and the words that come out of the mouth, I’ve heard the
same sentence multiple times, “I didn’t even know this was
possible.” And that’s partly because people’s understanding of
VR is, “I can look around to 360 image. I can potentially look
around a 360 video.” And that misconception that’s been built
through the use of mobile VR, it kind of needs to be unpicked. You
cannot explain virtual reality. It’s experiential technology. You’ve
got to put people in the headsets, and particularly in schools,
you’ve got to put people in headsets who are the game-changers,
they’re the people that have the sway to actually implement change.
One of the first people that I stuck the headset on was was Mark
Steed, who I met before he was the director of the school. I mean, we
were blessed in that Mark was the director. And he’s a very, very,
very tech-savvy guy. And he’s open to new technologies. But I stuck
Mark on the plank — as I want to do for most people that come to me
who want to try VR for the first time — stuck him on the plank. This
is a guy with three master’s degrees and he couldn’t walk out on it.
Two years later, he wrote a blog article for the Tez in the U.K.
about that experience. How much that affected him and how that moment
made him realize the power of the medium in general, because he had
that visceral, emotive reaction to it, which is not something he’d
ever experienced from a different form of technology before.
Alan: It’s interesting that you
talk about that particular one, because we actually built a training
scenario where you’re in a warehouse and you have to go across a beam
and turn off a power supply. And when we built it originally, the
beam was three feet across and you were maybe 20 feet off the ground.
And it wasn’t scary at all. We decreased the beam to one foot and
increased the height to 30 feet, aAnd wow. It is terrifying. And
everybody who goes across it is just tiptoeing across. And it’s this
mind melt because you’re walking across something — you know you’re
safe because you’re in a room — your brain can’t comprehend the fact
that you’re 30 feet in the air; you can’t decipher between reality
and not. And at that point when, in our particular instance, you
actually fall — you don’t fall, you just kind of fall and then it
goes black and you start over again — and it says, “don’t
forget to put your safety gear on.” And it’s in that moment
where you’re like, “man, I will never, ever forget to put my
safety gear on again.” Terrifying people, it turns out, is very
good education!
Steve: From the psychology point
of view, the one thing that I’ve seen again and again and again as
well, which — I remember reading Jeremy Violence’s book Experience
On Demand — and in that first chapter where he’s talking about
putting Zuckerberg on the plank experience at Stanford, and he refers
to something that I’d seen in person myself; the fact that you can
have a group of people standing there watching someone doing the
plank experience and laughing at them and going like, “oh,
what’s wrong with you? You know it’s not real,” blah, blah,
blah. And then they put the headset on themself, and even though
they’ve seen it from a third-person perspective, once they are in it
from a first-person perspective, the subconscious takes over and have
their reaction to it is completely their own. It doesn’t matter that
they’ve already seen somebody else do it. One of them, in fact, one
of the early sessions that we did with the first five I got — which
if you go in, you can find it on my site. If you scroll right, right,
right back to that to the earliest articles, it’s one of the first
ones on there. But also the video is linked on that five blog that I
mentioned earlier on — so I took the plank experience into the sixth
form psychology department at GESS and along with my friend Dr.
Joseph Bell, who is one of the psychology professors there. We pull
out of 16-, 17-, 18-year-old students through the plank experience.
And we captured footage of them going through the experience as well
as their reflections afterwards, and Joe provided some commentary in
terms of what was happening from a psychology perspective. It was
absolutely fascinating. The other thing, in terms of that that I find
fascinating, is that — and I wish I was talking to Dr. Sara Jones
the other day because she’s actually writing a book about VR, and
somebody I’ve known for a number of years — I was saying to her, “I
wish I had carried through with this idea that Joe and I had to do
this study, because I probably find that more adults can’t do the
plank thing.” Kids will walk out on that plank generally with
not a fear in the world, whereas adults, there’s a good proportion of
adults that just cannot do it. You know, my dad is a builder. He does
loft conversions, so he takes people’s loft — or attics, whatever
you want to call them — and he turns them into additional rooms. He
spent his life on roofs. He’s been spending his life climbing up
ladders and walking across roofs, and he couldn’t do that plank
experience at all. Couldn’t do it.
Alan: Wow.
Steve: Which I found
fascinating.
Alan: That’s weird.
Steve: Yeah.
Alan: Makes no sense.
Steve: You’d think so. Steering
back towards what we were saying about the kind of barriers; the
second one that I covered was the cost and the ROI — the return on
investment. And we kind of touched on this already. The idea that,
you know, schools are reticent to invest in technology that they
potentially see as just another thing. And you’ve got lots of
different vendors and companies trying to hawk their wares, so to
speak. You know, STEM is massive. Now, you know that everybody’s
like, “do we need to buy robots?” Do we need to buy 3D
printers? Do we need to buy VR content?” And I think the
difference is — I kind of touched on before — is that we’re not
talking about just another gadget when we’re talking about augmented
and virtual reality. We’re talking about the evolution of computing.
We’re talking about the change in the way the technology is
interacted with, full stop. And if you look at that article, you’ll
see an image that I painstakingly sourced from Google for the
original presentation, which is a group of students from the… I
think, from the 80s. At this single PC, which, you know, this was the
experience that I had initially when I was in primary school. That
there was one computer in the school, and it was on a cart and it was
wheeled into “oooohs” and “aaaaaaahs”, because
the magical computer’s here and–
Alan: I’m not going to ask how
old you are, but I still remember the first time I went to the
library and there was three Mac computers. It was like, “Whoa!”
[laughs]
“So awesome!”
Steve: Ataris and stuff. But I
specifically chose this image because I think this is the fear of a
lot of schools, is that, “well, we’re gonna buy– we can only
afford one, we’ll buy one. And then we’ll end up with the whole class
sitting round, watching one person interact with it.” My counter
to that is, no, you won’t. Because as educators, we now understand
better. Pedagogy has evolved and we understand that that isn’t what
you should be doing with technology. You shouldn’t have a group of
kids all sitting around watching one person interact with the
technology. There are so many more ways. We have evolved in terms of
our use of technology in the classroom and our understanding of
digital learning has evolved so much, especially since the advent of
tablets and the first deployments of iPads in the classroom. We’ve
learnt new ways to integrate limited amounts of technology and
explored bunches of different ways where, I’ve maybe had half a dozen
mobile VR headsets deployed for an activity in parallel to a VIVE or
mixed reality headset. I’ve had work where students are not just
taking turns, but they are specifically paired for a reason; so that
one person’s the hands in the physical space, and the other person,
they’re immersed in the virtual experience. That was kind of my
counter to that. Paired with that is the other one that we mentioned
before, which was this rate-of-change fear, and the graphic that you
see on the site there, which I always give a shout out to my — sadly
— dearly departed friend Chris Long, who died earlier this year.
Chris originally showed me this graphic of Martec’s law. It was
during the one-year anniversary event for the CBD and VR. He
delivered a presentation and this was part of it. And it was one of
those graphics, where I saw it and I was like, “this makes
perfect sense. How have I never seen this before?” And I’ve
started using it in a lot of presentations–
Alan: I’m actually stealing this
for my presentations. Thank you, Chris!
Steve: Essentially the premise
is that, because technology changes at this rapid pace, this
exponential rate, but organizational change tends to be slower. What
happens is that the gap between the two increases over time. And
ultimately, if a company sits and procrastinates for too long, the
gap between the state of technology and the organization and their
use of technology becomes so large that an organization can actually
need a full reset. And you’ve seen this. You see it in organizations
all the time. It might mean that they have to lay off a whole bunch
of staff. It might mean that they need to source a huge chunk of
investment to catch up. And this was something that I really
learned– I mean, I learned so much working for Mark Stevens, in
terms of digital strategy and vision. He came into GESS, which was
incredibly ahead of the time in terms of technology you use in the
classroom. But he came in and was like, “well, look, you’ve got
no refresh cycles built in for your tablet or your whiteboards or
this. There’s no standardization.” That was what he ended up
bringing to GESS, was a much better strategy, and a much better
plotted roadmap in terms of technology integration at the school, and
where the school was at and where the school was going. So then the
fourth one — the fourth barrier to the VR adoption, the fourth
common thing — because just for context, again, this article, this
presentation was based on the conversations that I’d had both at GESS
and with other educators worldwide, and with people that I’ve been
using VR and their feedback, in terms of the barriers that they were
hitting. So this wasn’t me just plucking these out of thin air. This
was based on common concerns that were being thrown at educators.
People will come to me and say, “I’m being asked about this.
What do I say?” And health and safety concerns is obviously
something that was coming up quite a lot. “Is VR say for kids?
How long should they use it for? Will it hurt their eyes? What age
should kids start to interact with VR?” I wrote a piece off the
back of the Commonsense Media Report contesting some of the data that
they used, and not because they were talking about content concerns,
saying that parents had concerns about sexual content or violent
content in VR. OK, I can understand that concern.
Alan: Yeah.
Steve: Yeah. You’ve got an
infinitely higher probability of finding that type content on
YouTube, which, these same parents have got their kids sitting on
their phones with unrestricted access to YouTube because they have
got parental restrictions set up. So I thought that was somewhat
ironic. “You know, too much time with VR is bad for you.”
Well, too much time with anything is bad for you. You know? It’s like
me saying eat too many cakes is bad for you. If I read a book for six
hours straight, I’m gonna get eyestrain. Too much of anything new, an
educated person will tell you that too much of anything is bad for
you. Social isolation is another one that was coming up. And, you
know, until we get into that place where the social multi-user VR
experiences are more common, I think that, you know, there is some
credence to that. It’s just a very insular thing when you’ve got a
headset on and you know, if you’ve got a room full of people.
Alan: To be honest, my kids have
unlimited access to VR and they — I mean, we’ve got a quest, we’ve
got Vives, we’ve got Hololens, Magic Leap, we have all the toys —
and they don’t want to go in it. And when they go in it, they want to
go in with other people. They want to be in social VR, which makes
sense if you’re playing Beat Saber or something, but it’s really
great when you’re with other people and then engage the platform, or
these types of things. So I think socialize– So– that’s really hard
to say! Social isolation.
Steve: Yeah. Yeah. I mean–
Alan: Say that five times fast.
Steve: These are things that I
pulled out from there, from the Common Sense Media Report.
Alan: Now, the next one is the
health concerns of bumping into something, that is actually probably
one of the ones that I think actually could be a problem; if your dog
walks in the room when you’re in VR and you trip on the dog or hit a
table. I’ve seen people get punched in the face.
Steve: Yeah. I mean, devil’s
advocate: I approach this from an educator’s perspective. And if you
want my take on the Common Sense Media Report, that article is an
in-depth breakdown of the whole report and my response to it is there
on the site. And if you look at the theory section, you find out,
yeah, 13 percent have bumped into something. My response is, of
course they have. If you’re not watching kids and they’ve got a VR
headset on, of course they’re going to bump into stuff. I’ve had kids
using mobile VR headsets where there was no, you know, the three
degrees of freedom headsets where they can’t move anyway, and they
still stand up and start walking. You have to monitor kids using this
kind of technology. If you don’t monitor them, again, yeah, of course
they’re going to bump into things. But I’ve used VR headsets with
hundreds and hundreds and hundreds of kids. And I’ve never had a kid
bump into anything because I control the situation. You don’t just
stick them in a Vive and then walk off and make a cup of tea.
Alan: It’s interesting. We had
an event a couple of days ago, we just got — this is two months ago
— we had just got the Oculus Quest and we had an event and people
were playing the sword fighting and… it was our fault, we didn’t
put a barrier around them, but somebody walked by as they were
swinging. Got punched right in the face. And I was like, oh, jeez,
that happened. But that was our fault for not putting a personal
physical barrier around the person, even though they have a digital
one. So when the person inside didn’t know… it just was our fault
and it was only… because we use the Vives, we normally set up an
extension so that people can’t get in to the person. But because the
Quest was so new and we didn’t know, we just put them in a room and
assumed that nobody would walk near them. Somebody swinging their
arms around like an idiot. You would think people wouldn’t walk near
them. But that happened.
Steve: Yeah. I mean, one very
low-cost solution I’ve seen people start to implement is they bring
in — and you can actually point ones that are marketed as
specifically for virtual reality now — but they essentially put mats
of some kind on the floor. You get students to take the shoes off.
You’ve got very simple haptics; I can feel that I’ve stepped off of
the mat, so therefore, I need to stop. And obviously making sure that
you’ve got, whether you call it your guardian or a chaperone, you’ve
got that that set up properly. I really like the Quest, obviously,
the way that the passthrough camera kicks on once you step outside
the guardian, because it draws that full stop. I’m outside of my
space. But again, little things like if you’re marking your
chaperone, don’t take it right to the very, very limits of the space
that you’ve got. So there it goes right next to a pane of glass or
right next to a solid brick wall. Give yourself a kind of a border so
that if somebody does get lost in the moment and you happen to not be
watching them, that they’ve got that kind of leeway, that little bit
extra more… I mean, we could do a whole podcast just around this
kind of VR and health and safety and stuff with kids. But a lot of it
comes down to common sense, ironically, considering this was a
response to the Common Sense Media. It comes down to limiting the
lengths of experiences and limiting, obviously, moderating the types
of experiences that you’re using with students. One of the things
that Mark and I started to put before we both left GESS, and once the
dust settled in both moves, we do hope to pick up was we were trying
to put together a kind of white paper and look at sort of creating an
actual formal approach to this. What age would you potentially start
using a Google Cardboard with a student? And how long would that
experience maximum last? What about a Windows mixed reality headset?
Vive? If I’m working with a 13 year old, what would be the
recommended maximum length of an experience? Especially with
something like Tilt Brush, I’ve seen it happen. You stick a kid in
and they get lost in that world. Well, do you pull it out and say,
how long do you think you been in? And then say “about five
minutes” and you say, “oh, you’ve actually been in there
20.”
Alan: Time dilation is a proven
fact in VR, actually. I was reading a study on this early on and the
time dilation can be as much as 25 percent that people think they’re
in VR a lot shorter than they actually are. And one of the things
that you can actually do to completely mess with people is you can
put a virtual clock inside VR that moves slower than real time and
you can actually increase that to about 50 percent. So people think
they’re in for an hour and they’re in for two.
Steve: I’ve never heard of that
done before. I like that. That’s nice. So then the last one is, it’s
kind of the biggie, from my point of view, is that the benefits alone
and ultimately, as somebody who’s worked for a long time now with
various forms of education technology, there’s gotta be benefits to
learning. There’s gotta be some point to deploy in this technology.
When I when I first started doing this stuff, I was doing with with
the high end VR in early 2017, the kind of party line for myself and
other pioneers like Jamie Donnelly and Steven Sarto, it was like
“there hasn’t been enough studies yet. The jury’s out,”
kind of. But we’re two years on and the jury is starting to come in
now and we’re starting to see more and more evidence come in from all
corners of the world. There’s all kinds of data that you see there on
the site. From Beijing University and Warwick University and Cornell
and Stanford, there’s all these studies taking place showing that no,
VR is more engaging than than other traditional forms of media, that
it leads to significant retention of learning. The one that came out,
Alvin Graylin tagged me in a tweet last week. There was a new study
from another university in China looking at–
Alan: Yes, I saw that, too. I
asked him for the study. I didn’t get it yet.
Steve: Now, I haven’t seen the
actual study, but with statistics looking at language learning in VR,
and then from a theoretical point of view, I then start thinking
about some of the big theoretical educational models, and the one for
me over the last decade that has become very prevalent is the SAMR
model from Ruben Puentedura. This is going to become like, if you’re
playing ed-tech bingo at conferences, the first thing that you would
put down on your bingo card would be the SAMR model on a slide
because you can’t get through any sort of ed-tech presentation these
days without somebody going into the SAMR model, one talking about
the different levels of SAMR. When I was putting this presentation
together, this is the first time I’d actually looked at SAMR for a
while and the first time I ever looked at it with specifically my VR
head on. And for those that maybe don’t work in the education
industry, this is a two-phase, four-step model created by Dr. Ruben
Puentedura specifically about technology integration in the
classroom. The lower phase, the enhancement phase, has the two steps,
the lowest step is substitution, where technology acts as a direct
substitute; type something on Word rather than you hand-write it.
Then it goes augmentation, so your technology’s still a substitute,
but, you know there’ll be an additional functional improvement. So
you’re using a digital thesaurus, or you’re adding in a clipart image
or something simple. And then the second higher phase is the
transformation phase, which has the modification and ultimately
redefinition. And the idea being that you’re moving towards using
technology to create new tasks and access content in ways that was
previously impossible. You take the lowest level, use that analogy of
having to write something; the lowest level would be just, I type on
word instead of handwriting it. And highest-level might be that I
collaboratively write something on a Google doc and then we post it
to a blog or a wiki, and then we take live feedback from people from
around the world. That’s me redefining that task, using technology.
And when I looked at this through the VR goggles — pun intended —
when I looked at it with my VR kind of mindset, what I found
interesting was that it’s like a leap frog, because I can’t see any
instance where VR is being used for things like substitution, because
by its very nature, because of the experience or nature of VR
applications, you’re automatically giving students the ability to do
things that they had previously never been able to do before. Whether
that’s painting with light or defying gravity with the way that you
build a sculpture and tilt brush, or flying around the world in
Google Earth, or stepping back in time in the Titanic experience.
It’s a transcendent technology in a way that other technologies have
not offered before. And I thought that was quite telling. I’m
actually at the moment working towards finishing something I started
in early 2018. So nearly two years ago, I started looking at VR and
another very famous education theory, which is Bloom’s taxonomy and
Chris Long, who sadly no longer with us. I, myself and my friend Alex
Johnson in India and Stephen. So we were kind of informally looking
at this on a shared document and we were going backwards and forwards
with it and it kind of got put on the backburner a few times. And
then Chris Long and I dive back into it maybe six months later and we
really thought that we had something there, something interesting —
something somewhat controversial — but interesting nonetheless. And
then it went back on the backburner. But now I kind of feel like I
need to finish that. I need to get that published, not least because
it was kind of the last project that I was working on with Chris. And
I’d like to see that through to completion. So 2020, I think at this
point, will be when I’m looking to publish that, because as I
mentioned earlier, I’ve got to rebuild my entire website before I can
publish anything.
Alan: Well, Steve, I mean,
there’s so much we can talk about. It’s been really amazing to listen
to these. You are one of the world leaders in this. So thank you so
much for joining us today.
Steve: Thank you, Alan. It’s
always a pleasure.
51 min
February 18, 2020 Finding the Tangible Value of XR in Business, with Nestlé’s Richard Hess
You may not immediately think of XR technologies when you think of Nestlé, who are more likely to conjure the idea of milk chocolate and bottled water. But their immersive technology lead Richard Hess drops by to explain how even a food company like Nestlé can benefit from embracing emerging tech, on the 100th episode of the XR for Business podcast.
Alan: Hey, everyone. Alan
Smithson here with the XR for Business Podcast. Today, we’re speaking
with Richard Hess, the immersive experience lead at the massive
multi-national Nestlé, making a billion products a year. A day, he
said, but I don’t know, a lot of products. You have them on your
shelf, you have them in your fridge. We’re gonna be speaking with
Rich about Nestlé’s VR and AR efforts in marketing, sales,
enterprise solutions, and training. All that and more on the XR for
Business Podcast. Rich, welcome to the show, my friend.
Richard: Hey, Alan. Thanks for
having me.
Alan: It’s my absolute pleasure.
You and I have spent a lot of time kind of talking over the phone,
but also spending some time on a panel at AWB.
Richard: Yeah, that’s right.
Yeah, we’ve crossed paths a few times. I’m just happy to be here, to
talk a little bit about what we’re doing at Nestlé.
Alan: We’re super excited. Why
don’t you make an introduction to you, and what you’re doing in XR
with Nestlé?
Richard: Sure. So for myself,
I’ve been with Nestlé for about 10 years now. First based out of the
US, working for our waters division there. Mostly supporting digital
marketing on the technology side. If you go back 10 years ago, the
mobile phone was becoming big, people were starting looking at social
media as a way to communicate. Through that journey around three
years ago, spent about a year in San Francisco, starting with our
innovation outpost that we have out there, looking at emerging
technologies. And that’s kind of where I gained a passion for
augmented/virtual reality, mixed reality, extended reality, whatever
kind of acronym comes in the space there.
Alan: Realities? All the Rs?
Richard: All the realities.
Yeah. [laughs]
Alan: I actually wrote an
article — you can find it on LinkedIn — it’s called “The ABCs
of R”.
Richard: Oh yeah, there we go.
That’s good, I got to take a look at that. But yeah, that’s when I
started getting a bit more hands on from my under the organization of
getting “Okay, I’ve seen a lot of tangible use cases.” And
around a little more than two years ago I came over here to
Barcelona, where Nestlé has set up this global digital hub, that was
mostly — at the time — looking more marketing and sales focused on
how do we build centralized global platforms, and products, and
services that can serve all of our markets and brands across the
world, but now is more holistic across all different use cases,
whether it’s in the factories, supply chain, HR etc. It’s kind of
looking across the whole spectrum. So the past two to three years or
so, I’ve been looking at augmented/virtual reality in that way of how
do we take all these little different one-off experiences that we’ve
done. And when we see a lot of tangible value in leveraging these
technologies, we try to bring them to scale in something that is a
full industrialized product, that the rest of the organization can
take advantage of.
Alan: Let’s give an example of
one that you did a pilot, you realized the success of it and now
you’re going–
Richard: Well, I think a really
good example — and when you think about, it’s a very simple one —
but what we used for augmented reality within a sales organization
was using AR as a tool in the sales person’s toolbox. So we have a
brand called Nestlé Professional and they mostly sell to people like
hotels, restaurants, big convention centers, cafeterias. And they’re
selling often these big industrial coffee machines, these big Nescafe
machines. You need two people to carry, they’re very heavy. And often
when they were trying to make a sale, in order to do this, they had a
paper cut-out, 2D of the machine. They put it down on a table and
they say, “OK, imagine there’s a coffee machine here,” or
they would take out a tape measure and start showing them the
dimensions, giving them a flyer, things like this. Or usually what
would have to happen is we would have to actually ship a machine out
before a sale. And more often than not, you have to then ship that
machine back without closing the sale. It costs time, it costs money,
it’s ineffective. So just a simple use case of say, hey, why don’t we
digitize this and use augmented reality to display what the machine
would look like, surround it with, say, certain hotspots of different
sales material, promotions, switch in different colors or different
models, go through the entire catalog of different machines. That was
something that is not the, say, sexiest use case of AR. To your
peers, it seems very simple, something that’s been around a while,
but being able to do that at scale and say hey, we’re gonna take all
of our sales reps in the US, test and learn this out, get some
feedback. The feedback right away was this is actually providing
value. My competitor doesn’t have this or if they do, it’s not at
this scale. It’s going to provide a lot of value back to us. And even
reducing the number of visits, going to a customer from 3 to 2 in
order to make a sale. And each one of those costs couple hundred
dollars. You start to look at bringing that to the entire sales
organization in that region. You start to see tangible dollars back.
You get to a point where you can go to a senior leadership or at a
executive level, and say augmented reality is not something that’s
just a cool thing that’s going to get ourselves people excited. We
can show it’s actually affecting bottom line or it’s reducing costs
to sales people that are trying to do their jobs. So that was just
kind of one example where we started really small, maybe half a dozen
sales reps in the US. And as we built it along, as the technology
moves along, we move with it. Starting from, say, you had to have
marketed your kit around with you to display that machine, to then
world tracking, to then now the advancements of more WebAR that’s
coming, and quick look functions in the phone. We’ve been able to
take that and then scale it to all the what we call Zone Americas,
which would be both North and South America. Looking to next year,
we’re going to try to take that more globally into different markets
in Europe and places in Asia, Africa. That’s the type of scale that I
think we’re looking for, for augmented reality, where it’s providing
a lot of utility and a lot of value added back is something that
can– it’s easy to prove or there’s tangible ROI there.
Alan: So how are you measuring
ROI in that? If you have a measurement like decreasing sales visits
from 3 to 2, that’s demonstrable value immediately. But how else were
you measuring KPIs and ROI, in order to go and prove that business
case to your senior execs?
Richard: Well, definitely the
one around cost avoidance is a big one. And that helps speed things
along. But also when you’re getting, say, feedback directly from the
customers themselves or their surveys, whether it’s repeated
purchases, people that want to get more from that. So, for example,
last year we revealed a purchase for the Starbucks out-of-home
business and saying now instead of just the machine’s pairing that
with different products that could be on the Starbucks line, whether
it’s different coffee corners, and the coffee corner would be this
big cabinet that has a coffee machine there that people go
self-serve. You would see it probably in places like the US and
Canada, like in a gas station or something like this. Or even just a
different point sales material to help sell different packets and
stuff like that. Adding these tangible things over time to make it as
much of a sales tool for the Nestlé sales rep, but also as a tool
for the boutique owner or somebody that’s looking to sell this to
customers. Being able to see a swing in an uptick in net sales,
certainly that’s the golden goose, if you will, that that’s the one
that everyone’s chasing. And we’re starting now to see that over time
a little bit more. But for sure, what gets the lightning rod started
is to say we can reduce costs right away and provide a more enriched
tool. Let’s get that to get the ball rolling and then we’ll see the
sales you brought over time. And we’ve been able to measure also
metrics not related to finance or numbers. But if we’re able to say
this many people are scanning different environments or scanning that
leaflet or using world tracking, this is the amount of unique people
are doing it versus the number of times they’re scanning. You can
measure how many times people are going into stores, how many times
they have to scan these things, how many people are doing it versus
how many times they are doing it. You could draw those numbers back
to “OK. Did this lead to a sale or not?” And so that’s the
kind of metrics that we’re starting to collect across the board.
Alan: How are you tracking that?
Are you doing it through your sales force or whatever your CRM is?
Richard: Not so much CRM. So I
think CRM integration is more of a long term play for us. And it was
kind of important to, I guess, have it in a siloed approach, at least
initially, just because we wanted to be fast moving and get the ball
rolling and helping develop the overall business case. So when we
look at, say, a big CRM integration, especially with a company like
ours, where you have a CRM tool that supporting tens of thousands of
employees, pushing AR integration from the get-go to that, that’s
probably going to be a project that’s going to be 18 months long. So
where we were more starting first is, let’s prove the concept around
augmented reality first and see if that provides value. We can
probably do that project or that first test and learn in around eight
to ten weeks and then build upon that. And then we’ll have this kind
of convergence down the road to say, “OK, everyone’s agreeing
that this provides a lot of tangible value. Now it’s time to
integrate this as just a function or feature of the CRM tool.” I
would more say looking at augmented reality that way, is not the AR
tool that we’re adding the CRM functionality to. It would be more
long term. How do we add the AR feature to our CRM tool? I think
that’s the more long term play. But in terms of adding value now,
it’s certainly the AR providing that as a standalone.
Alan: Yeah, I think that– this
has come up a few times, where making AR for the sake of AR is
interesting, but when you make it as a subset of the bigger sales
tool and say, “OK, well, it’s just a feature within the sales
tool.” You’re still gonna have printed materials, you’re still
gonna have all these other things. It’s just another amazing tool in
a sales person’s arsenal.
Richard: Exactly. It will never,
I don’t think — at least for the foreseeable future — it’s never
going to be the end all, be all and replace everything. It will
certainly always be — at least initially — one enhancer to
everything else we have going on, and more competitive advantage
where other companies are not using and you are, you will have the
ability to differentiate yourself against your competitors, for sure.
Alan: So I have a couple of
technical questions. How many products did you– did you do the whole
catalogue of these devices, or was it something that you rolled out
with the five models and then slowly kind of iterated on? And so
that’s question one. Question two is, where did you get the models
from? Was this from your manufacturing partners? And then the last
question I have is, what was the platform you used to make this
happen?
Richard: Yeah. So for the first
one, we started off small, of course. So we did two models to start
saying, “OK, let’s do a coffee machine and a juice machine.”
So that was what we started off first. And with that initial tests,
with those half a dozen sales reps, as I said, it was just around
these two models. Let’s get it out there. Let’s see if there’s
tangible value added back. Let’s get that feedback and see if we
should go from there. Because we were also in a place like, “OK,
we’re assuming this is going to work, but we need to actually put it
out there before we make a large investment to justify us making a
bigger investment.” When we did see it was adding value, we
said, “OK, let’s start to roll this out to different regions.”
So starting in the US, we started then to roll it out into Latin
America, countries such as Chile, Brazil, Peru. And we’re looking at
saying, “OK, let’s add a couple more models, see how this goes.
Maybe there’s different makes from market that we had to take a look
at.” And actually, one of the challenges that we did find at the
time was when you look at the maturity of smartphones around the
world, it is getting up there, where a lot of phones will have ARKit,
of course the new iPhones, ARCore for a lot of newer Android models.
When you go to certain markets, like Brazil, we’re discovering a lot
of sales reps are being issued phones that are 5, 6 years old.
Sometimes we didn’t even have gyroscopes in them. So we had to kind
of take a step back and say, “Well, OK, we’re making maybe some
assumptions that this is going to work out the box. We’re going have
the best experience. Throw in some world tracking through ARCore on
Android.” And we quickly had to be like, “Whoa, OK. We have
to maybe take a step back and think about what’s the right way to do
this.”
Alan: Hold your horses, man.
There’s still people on texting phones.
Richard: Of course, of course. It was quite a lesson learned for us to say, OK, let’s make sure that we’re identifying this up front. We’re building a solution that’s holistic for everyone. So now we have a bit of a mix of different, say, tracking mechanisms or triggers that would enable the experience. But from there, after we did have that fallback option for Latin America, that’s when the brand was more convinced to say, “All right, let’s start to do this for this entire region with all the different machines and coffee corners.” And you look at a total volume for that, you’re looking at around two dozen different models as well. For different use cases outside of, say, this Nestlé professional who sells things. If I take a look at Nespresso, for example, this is something when we look to say, OK, how do we enable quick look or WebAR functions on our e-commerce sites, we are looking at doing the entire range of espresso models and that’s up to around high 80s, 90s, different models that we have with different accessories, different colors, things like this. So we’re starting to kind of get to a point where a lot of people are seeing the value for it and they’re kind of going all in and saying, let’s throw it on our website. Let’s try it out. Let’s get a lot of different feedback and change it as needed. When it comes to the models themselves, it’s a very interesting question and another lessons learned that we had internally for that, because, as you know, there’s almost hundreds of ways to create the more raw CAD file for complex machines, say like coffee machines like Nespresso or things like this, but even point of sales all on this line. So we were dealing with a scenario, where we were getting them in any different type of format in 3D quality that you can imagine, often something as big as multiple gigabytes, something you can’t say put into an AR experience that you expect the consumer is going to deleverage. It wouldn’t work that way. So we had to do a lot of what we called 3D model curation. So we would have to either take the existing model or know what the model that they were trying to bring to life in AR have a lot of the 2D high-res photos of them. Because often you’re selling something like a coffee machine, you have an e-commerce site that’s also selling this. And there’s a lot of photos that have from different sides of the machine. Those have been dumbed down for more high quality ones that have been taken in a product shoot somewhere. So we would have to take either that existing 3D model or 2D photos of those machines, the different dimensions of them and then go through a process of recreating that model that’s kind of AR ready, good to go, something that’s 2 to 3 megabytes, tops that is in kind of glTF, GLB format, something that we can plug and play really easily into an AR or VR experience. So I think for us I think the industry will eventually move towards something where there’ll be some type of automation for that, and an ingestion engine, probably, for taking certain models from different formats and then spitting them out into something that’s AR/VR friendly. And you look at companies like Autodesk have platforms that they’re developing, that say that the goal is to do that. For us, we were more partnering with different providers that had capabilities in that space, to be able to help us bring that to life. People that we’re building their startups or business models around that specific use case, that there’s going to be a content– well, there is; there is a content problem of trying to develop a lot of these things at scale. And they’re gonna help for the first start of that, providing different resources that can create these from scratch, with a more of a long term goal to how do we automate that in the future.
Alan: That’s everybody’s goal. I
actually just saw an article yesterday that Nvidia is working on
creating 3D models from 2D photographs.
Richard: Yeah. So something like
that. When we see stuff like that, it definitely gets us excited of
looking at scale, because you said in the beginning Nestlé, this
huge conglomerate, we have 2,000 brands, we operate around 100, 80
countries around the world. In order to do this at a massive scale,
you’re looking for all the tools that you can get in order to reduce
the burden of creating a lot of this stuff. So being able to do what
you said with Nvidia, and create a new model from a 2D image, that’s
something that would definitely help along those lines.
Alan: Yeah, that’s like the
penultimate if we can get to that. *When* we get to that. It’s not
even an if anymore. It’s like there’s so much research being done in
this. I have a whole folder on 3D from 2D images, so there’s lots of
people working on it. You talked about kind of working with startups
to help get you where you need to be. What is the platform you’re
using to distribute this? Because you talked about quick look in AR.
Are you just using kind of Apple’s framework for that? And then what
about people on Android? What is the actual nuts and bolts of it?
Richard: We do look at
partnering on it with different startups or platforms that are either
emerging or established in the space. Every company has to be a
digital technology company today. But again, at our core, we’re a
food company. We don’t have, say, teams of developers that are doing
this space. We’re not developing our own proprietary AR tracking
technology or things like that. So we are leveraging what’s out
there. So whether we’re working with, say, I would call them like AR
content platforms. For example, we do some work with Zappar as one of
them. There are some more emerging platforms they’re looking at from
an e-commerce perspective, which is great from our end, because
obviously there’s a lot that needs to go into educating different
marketers and people in Nestlé. People get excited about this. But
again, you have to bring it back to that tangible use case, of where
it’s going to provide value.
Alan: So easy for us to chase
the shiny object.
Richard: It really is. It really
is.
Alan: And it’s interesting. Five
years ago we were doing 360 videos and then we said, “OK, well,
this is easy. Move on to something more difficult.” We then did
videogrammetry and volumetric capture, and then we did incredibly
complex 3D models. And then VR where you get right in. And it turns
out the 360 video stuff we were doing years ago is what people want.
Richard: [laughs]
Alan: They’re like, “Oh,
that’s perfect for what I need!” And I’m like “Oh man…”
Richard: Yeah. There’s still a
lot of tangible value in doing that.
Alan: Yeah. One of the things
that I think is valid for Nestlé — and for every company — is a
simple 360 tour of the facility for new employees. Just simple,
here’s what our facility is, here’s where the kitchen is. You can
kind of navigate around. Even just a Matterport tour of the facility
before you even get on on site, because a lot of the first few days
of new employees roll is wandering around, trying to find out where
to go. So if you could shorten that, put in VR and let them do that
at home, then you’ve just saved a couple of days of running around.
Richard: Exactly. And that’s
actually something we’ve done a bit with Purina in the US– that’s
Purina, the pet food company. They’ve done some– a lot of work
around. They’re called VR factory tours. So putting employees or
customers into their different factories, having them navigate
through it, interact with some things, learn a lot about them. From
an employees side, sometimes it’s people that support these guys,
maybe from the IT perspective or HR perspective day-to-day. The
factory employees that you have to have access to the different tools
and things like this. They never actually been to the factory
sometimes. It sounds kind of crazy, but when you’re working company
as big as ours, that does happen. So just being able to put them in a
VR experience — and for this they use the Quest in order to do that,
something that they can buy a couple of, carry around, bring them
around the office, things like this — and put them to that tour, it
gets a lot of tangible feedback. But at the end of the day, it’s a
high quality 360 video that has some interactivity that’s been laced
throughout it. But it is really effective content, it’s really good
and it incites emotion and it gets the point across. So–
Alan: We don’t need to
overcomplicate things. Actually, you’ve said that twice now. The 360
video stuff works, and it’s simple, and it doesn’t need to be crazy
complex. And then also with the stuff in the AR quick look stuff, it
doesn’t need to be super complicated. You don’t need to have a multi
exploded view of the product. You just need to see it on the shelf in
the right size and say, “OK, that doesn’t fit.”
Richard: Exactly. I mean,
there’s always a time and a place for the more enriched complex
experiences for sure. But yeah, often what consumers want to see,
they want to see, “Hey, buying a new coffee machine. Is it going
to fit in my counter? Am I going to like how it looks?” Things
like this. It’s a very quick five to six second decision, whether or
not they’re going to move forward.
Alan: How are you guys using
this kind of on the enterprise side and in your factories? Are you
using AR at all on the factory floor?
Richard: Absolutely. And where
we look at it a bit differently from what I was discussing before
is– so, if we take a step back, look at Nestlé factories, we have,
I think, around 400 — don’t hold me to this number — I think it’s
418 factories around the world, 300,000 employees. Whether or not my
team exists here in Barcelona, somebody has used augmented/virtual
reality in this organization before. So a lot of what we’re doing
when it was coming to the enterprise side initially was trying to
understand who’s been doing what, what ongoing experiences have been
happening, who’re the evangelists in this company that have actually
tangible hands-on experience of having it at the factory level. And
when we did this kind of analysis, we did see over the past half a
decade or so, we’ve had over 30 different experiments with different
form factors like headsets, whether they’re mixed reality or
smartglasses, different vendors out there. I sometimes laugh. If I go
to AWE and I meet all the different enterprise vendors there,
everyone’s going to tell me they’ve worked with us before, even
though it’s the first time I’ve met them. And I say, “I’m sure
you have.” Because there’s a lot of people out there that are
interested in this, whether they’re testing it on a small scale or
they want to expand. So we had to get a good look at that first and
what we–
Alan: How did you collect that?
Richard: So the way Nestlé
works, as well as we have people on the factory floor, that are a
part of what we call our technical production organization. They have
some kind of co-pilots in these factory areas, part of our R&D
organization, specifically an organization called PTC, which is
Product Technology Center. So they often try out new digital
technologies in these spaces to see if they’re viable for factory
floors. So what we’re finding is often, yeah, they were testing a lot
of this stuff out. We had to do a lot of internal networking. And we
actually also implemented across the organization, not just
augmented/virtual reality, it could be AI, digital twin, robotics,
things like this. We’ve kind of implemented the internal what we call
an innovation repository, a space where people can submit the
experiments they’re working on, what they’re trying to get out of it,
where they are in their experiment. Just to give exposure to other
parts of the organization. Because often where you’re seeing, say,
you take remote assistance in a factory, we’re probably repeating the
same experiment 15 different times in 15 different locations around
the world. But these teams weren’t talking to each other. So just
getting an exposure to different teams to say, hey, you want to do
this? These five teams also did this. This is their success. This is
their failures. At least build off of that.
Alan: Yeah, of course, that
makes way more sense than being stuck in pilot and POC purgatory
forever in every division. I actually have a question about the
innovation repository. How do you standardize it, so that it’s easy
for somebody to look at the results? Like do you have a standard form
or–?
Richard: Yeah. So we use a
platform called Startup Flow for that, that allows us to kind of have
a good view–
Alan: What’s it called?
Richard: Startup Flow. One we
started using last year. But it gives us a good kind of holistic view
over a lot of different things that are going on. And there was a
campaign to say, “Hey guys, put your stuff in here. Let’s not
reinvent the wheel.” Sometimes it’s a difficult conversation
because like I said before, this is an inherently cool technology.
And sometimes you don’t want to be told you can’t do it because
somebody else has done it before. We did get a lot of people to
understand the inherent value of that, and it’s not to say that
“We’ve done your use because before we’re not going to do it
again.” More often, I think the conversation going into this
year and next year is once you start to scale this across all the
organization of people that raise their hands, where again, you can
find a lot of tangible value added back. This just proves the use
cases there and that it’s going to provide us a lot of different
value.
Alan: I was just looking at the
Startup Flow webpage. What a great tool. Wow, super cool. Measuring
the ecosystem of startups and new technologies and then measuring the
KPI as metrics on one single dashboard. That’s awesome! That’s value.
Anybody who’s listening, startupflow.io. I mean, that’s worth the
value of listening to this podcast in itself.
Richard: Yeah, it’s a good tool
or something. Yeah. We’re using a lot in our organization.
Alan: What other ways are you
using this technology? Are you using it for training?
Richard: Absolutely for
training. So one example being a couple of weeks ago in a factory
here in Spain, in Girona, and they’re kind of mended them in a lot of
different factories around the world to follow what is kind of TPM
certification, which– it’s not my forte, this area, but I believe
it’s for Total Production Maintenance. Or Management. It’s
essentially certification for being kind of a best in practice
factory. We have zero defects across the board. And one of the areas,
one of the steps into getting to there is that they have to have an
on-site, say, training center where employees can go and self-train
themselves in different areas, whether it’s safety areas for the
here’s what you do in a confined quarters. Here’s what you do when
there is a explosive situation like a gas leak. If you’re in a height
situation, where you’re up somewhere high, it’s not secure. They had
to have this kind of facility onsite, that people can go to and learn
as much about that use case as they can, and train themselves up.
Now, the factory that we’re talking about is one of the more advanced
ones in the world, it’s coffee factory. It produces a lot of the same
Nescafe, Nescafe Dolce Gusto for good parts of the world. But going
back to we have a ton of factories, right over 400. Not every factory
has the ability to build a huge training center on-site. They need to
find creative ways to essentially get this type of material out
there. So using something like virtual reality for that, where to put
them in these scenarios now where the form factors are. We look at,
say, stuff like the Oculus Quest, where you can just have the
headset, controllers, the certain amount of space around you. You can
pop into a training session and it’s something that you can have a
lot of. You don’t have to set up a lot of different gaming PCs in
order to enable that. That’s something that brings a lot of value.
You don’t need as much space in order to do that, and to have that
type of trainings on-site. That and going through a height training
or explosive training in VR, probably it looks much more of a
emotional and personal impact to you, like a heightened sense of
receiving that content than watching a video or reading a PowerPoint
or something like this. It’s certainly something that you could take
a look and say it’s a much more effective way of learning that
content or material, retaining that information over time.
Alan: Are you guys measuring
that? Do you have– is it anecdotal? Are you measuring KPIs against
that?
Richard: Right now, we’re still
starting out that use case as well. It’s gonna be something that
we’re building Q1, Q2 next year. We have done some other safety
training in the US with their waters division. That’s also looking at
it from that scenario, like pointing out– different scenarios where
on the factory floor you have to point out different safety
violations, things like this. I believe they are making the KPIs more
on how much has it reduced the number of safety infractions or
reported infractions in the factory over time, of the amount of
people they have through the training versus the amount of reports
that they get. So it’s something that it’s still emerging for us a
bit, but we see a lot of big potential. And I think 2020 is where
we’re going to be looking at it a lot from scale.
Alan: And now, is this something
that– again, are you working with vendors there or are you trying to
do it in-house?
Richard: Mostly, again, with
vendors. And for us, we’re testing and learning with a few different
ones, but certainly it’s something that we’re keeping an eye on going
into next year. And I think just from the point of vendors, too, I
should make a note. As for a lot of these scenarios, we do– let’s
say we have global vendors that we love to work with, but we’re
always constantly working with different startups in the different
communities, because for technologies like this, everything is
changing every three to six months. Companies are rising, falling,
pivoting, being bought, picked up. So we–
Alan: Yep.
Richard: Yeah. Certainly, we’re
in a scenario where we try to work with as many people as possible.
The procurement team probably won’t like me saying that. But we do
try to. [chuckles]
Alan: One of the problems we’ve
been trying to solve is, how do you disrupt an industry constantly
and consistently disrupting itself? And so what we’ve come up with is
we’re working on a platform marketplace for training and learning
technologies. Because — like you realized — companies are coming
and going. Technologies are being invented and displaced all the
time. So rather than try to invent all of the different technologies
and try to keep on top of it, we’re building the platform that
synchronizes with the CMS, it synchronizes with the elements that
walks the customers through that, and keeps them always on the most
cutting edge platform across whatever device happens to be the
hottest thing of the year. And so that’s what we’re doing on a per
user basis. Then you can just say, OK, we’re gonna use VR and AR
across our whole enterprise. And we now have x number of employees
doing it, x number of headsets. And then it’s an ongoing thing and
you have access to all of the startups that have been pre-vetted,
rather than try to go through this selection period. As you know,
it’s challenging, right?
Richard: Absolutely, no. That’s
something I would love to take a look at that. [laughs] Later, after
when it comes out. It’s certainly–
Alan: We’ll talk offline.
Richard: There you go. There you
go.
Alan: I didn’t want to highjack
this podcast. Like, man, this is exactly the problem we’ve been
trying to solve is, how do you how do you overcome the point
of–technology is changing, and companies are being bought and sold,
and everything. And look, from our standpoint, we’ve started our own
accelerator. You know, you’re part of it.
Richard: Mm-hm.
Alan: And having this
accelerator to find and identify this talent, make sure that they are
able to deliver at the level of the customer’s requirements. Like,
you guys have 400 factories. That is a different conversation than
one company that has one factory.
Richard: Exactly.
Alan: And maybe wants one
training exposure. And then the other side of it is being a managed
marketplace. What are the things that you guys are creating in
safety? Let’s say, for example, a factory or warehouse for fire
safety, or height safety, or driving. What can then be reused and
resold to other companies? Nestlé is different because you have 400
factories. What can be then packaged and sold to all the factories?
But also what can be generisized and resold, so that everybody can be
safer at work?
Richard: I think, yeah, for at
least my perspective coming up with a lot of this content, it should
be something that at the end of the day, looking at proprietary I
don’t think is the right way to do it. If it’s for something that’s
safety, that’s going to address scenarios that you’ll find in a lot
of different factors — majority of factories — it’s something that
if we could help build the ecosystem of understanding on that and
it’s something that other, CPGs or fast moving consumer good
companies can leverage as well, I think we’re OK helping pave the way
for that. I am passionate about this area, and I think all boats need
to rise.
Alan: I couldn’t agree with you
more. And the interesting thing about this industry right now — and
it may change, because as the industry goes from nascent to prolific,
the gloves come off, so to speak — but so far, every single person
in this industry has been exactly that mindset. How do we all work
together? Because this is not a net sum game. More wealth will be
created in the world in the next 10 years than all of previous human
history. So we’re not scrambling to try to– “I get 10 bucks and
you have to lose 10 bucks.” This is not the world we live in
anymore.
Richard: Yeah, agreed.
Alan: You have a unique view
that most people will never have. You are kind of running VR and AR
for one of the largest companies in the planet, across all sorts of
different ways, from marketing and sales, to enterprise and training.
You really are the poster board for VR and AR in an enterprise, in a
company. So what problem in the world do you want to see solved using
XR technologies?
Richard: Thank you, no pressure. [chuckles] There’s two that come to mind. So one that may be more unique to us is when it comes to the food industry or the TP industry as well. Nestlé has made a 2025 commitment to make all their packaging sustainable, 100 per cent recyclable and things like this. If XR — through augmented reality or virtual reality to help sell a story, things like this — can help around helping people recycle. Scanning more about, say, a plastic bottle. Getting more information about it, directing into the nearest recycling center, gamifying it somehow to reward people for doing that. I think that’s something as powerful. And using the reach that we have, we have a billion individual packages that we generate every day, to get the communication out on that and tackle a problem that, frankly, we help generate.
Alan: The Nestlé blockchain
gamification recycling game. Ooh!
Richard: There you go. Yes. This
is something along those lines. Of course, yes. So I think having XR
tackle that or be a piece in the puzzle for doing that, I think would
be something that may be more uniquely to us and more powerful more
across the industry, I think. Helping with the massive upskilling of
the workforce, whether it’s robotics, removing repeatable tasks. That
example of the factory that we’re there in Girona. They’re using
cobots, they’re using different automotive forklifts, things like
this. And often the feedback is what happens to the work versus this.
And you get mixed tags sometimes, but they’re like, this frees them
up for doing other tasks that we need done or we freed themselves up
to doing more advanced things that we want to do. I think using
AR/VR/XR across the board to help skill current workers or train new
workers on next level skills that they need to do, I think, is also
something that’s going to be unique to this medium, this industry,
where this will provide a lot of tangible value as a kind of
learning, training or computing platform of the future.
Alan: Well, I’d have to say that
hats off to you for those amazing opportunities, because telling
people– well, first of all, Nestlé– hats off to Nestlé for
recognizing that they’re causing the problem — or part of it — and
saying by 2025, we’re going to solve the problem, and at least fix
it. Because this is a global problem, we need to fix it. And we all
need to come together. And it starts with big companies who can do
it. And Nestlé’s in a position to do that as well. So hats off to
you guys for for taking those initiatives, and really excited to see
where that goes. And if anything, if we can help in any way, I’m
happy to do so.
Richard: Absolutely.
Alan: Thank you so much for
joining me, man. This has been amazing.
40 min
February 18, 2020 Finding the Tangible Value of XR in Business, with Nestlé’s Richard Hess
You may not immediately think of XR technologies when you think of Nestlé, who are more likely to conjure the idea of milk chocolate and bottled water. But their immersive technology lead Richard Hess drops by to explain how even a food company like Nestlé can benefit from embracing emerging tech, on the 100th episode of the XR for Business podcast.
Alan: Hey, everyone. Alan
Smithson here with the XR for Business Podcast. Today, we’re speaking
with Richard Hess, the immersive experience lead at the massive
multi-national Nestlé, making a billion products a year. A day, he
said, but I don’t know, a lot of products. You have them on your
shelf, you have them in your fridge. We’re gonna be speaking with
Rich about Nestlé’s VR and AR efforts in marketing, sales,
enterprise solutions, and training. All that and more on the XR for
Business Podcast. Rich, welcome to the show, my friend.
Richard: Hey, Alan. Thanks for
having me.
Alan: It’s my absolute pleasure.
You and I have spent a lot of time kind of talking over the phone,
but also spending some time on a panel at AWB.
Richard: Yeah, that’s right.
Yeah, we’ve crossed paths a few times. I’m just happy to be here, to
talk a little bit about what we’re doing at Nestlé.
Alan: We’re super excited. Why
don’t you make an introduction to you, and what you’re doing in XR
with Nestlé?
Richard: Sure. So for myself,
I’ve been with Nestlé for about 10 years now. First based out of the
US, working for our waters division there. Mostly supporting digital
marketing on the technology side. If you go back 10 years ago, the
mobile phone was becoming big, people were starting looking at social
media as a way to communicate. Through that journey around three
years ago, spent about a year in San Francisco, starting with our
innovation outpost that we have out there, looking at emerging
technologies. And that’s kind of where I gained a passion for
augmented/virtual reality, mixed reality, extended reality, whatever
kind of acronym comes in the space there.
Alan: Realities? All the Rs?
Richard: All the realities.
Yeah. [laughs]
Alan: I actually wrote an
article — you can find it on LinkedIn — it’s called “The ABCs
of R”.
Richard: Oh yeah, there we go.
That’s good, I got to take a look at that. But yeah, that’s when I
started getting a bit more hands on from my under the organization of
getting “Okay, I’ve seen a lot of tangible use cases.” And
around a little more than two years ago I came over here to
Barcelona, where Nestlé has set up this global digital hub, that was
mostly — at the time — looking more marketing and sales focused on
how do we build centralized global platforms, and products, and
services that can serve all of our markets and brands across the
world, but now is more holistic across all different use cases,
whether it’s in the factories, supply chain, HR etc. It’s kind of
looking across the whole spectrum. So the past two to three years or
so, I’ve been looking at augmented/virtual reality in that way of how
do we take all these little different one-off experiences that we’ve
done. And when we see a lot of tangible value in leveraging these
technologies, we try to bring them to scale in something that is a
full industrialized product, that the rest of the organization can
take advantage of.
Alan: Let’s give an example of
one that you did a pilot, you realized the success of it and now
you’re going–
Richard: Well, I think a really
good example — and when you think about, it’s a very simple one —
but what we used for augmented reality within a sales organization
was using AR as a tool in the sales person’s toolbox. So we have a
brand called Nestlé Professional and they mostly sell to people like
hotels, restaurants, big convention centers, cafeterias. And they’re
selling often these big industrial coffee machines, these big Nescafe
machines. You need two people to carry, they’re very heavy. And often
when they were trying to make a sale, in order to do this, they had a
paper cut-out, 2D of the machine. They put it down on a table and
they say, “OK, imagine there’s a coffee machine here,” or
they would take out a tape measure and start showing them the
dimensions, giving them a flyer, things like this. Or usually what
would have to happen is we would have to actually ship a machine out
before a sale. And more often than not, you have to then ship that
machine back without closing the sale. It costs time, it costs money,
it’s ineffective. So just a simple use case of say, hey, why don’t we
digitize this and use augmented reality to display what the machine
would look like, surround it with, say, certain hotspots of different
sales material, promotions, switch in different colors or different
models, go through the entire catalog of different machines. That was
something that is not the, say, sexiest use case of AR. To your
peers, it seems very simple, something that’s been around a while,
but being able to do that at scale and say hey, we’re gonna take all
of our sales reps in the US, test and learn this out, get some
feedback. The feedback right away was this is actually providing
value. My competitor doesn’t have this or if they do, it’s not at
this scale. It’s going to provide a lot of value back to us. And even
reducing the number of visits, going to a customer from 3 to 2 in
order to make a sale. And each one of those costs couple hundred
dollars. You start to look at bringing that to the entire sales
organization in that region. You start to see tangible dollars back.
You get to a point where you can go to a senior leadership or at a
executive level, and say augmented reality is not something that’s
just a cool thing that’s going to get ourselves people excited. We
can show it’s actually affecting bottom line or it’s reducing costs
to sales people that are trying to do their jobs. So that was just
kind of one example where we started really small, maybe half a dozen
sales reps in the US. And as we built it along, as the technology
moves along, we move with it. Starting from, say, you had to have
marketed your kit around with you to display that machine, to then
world tracking, to then now the advancements of more WebAR that’s
coming, and quick look functions in the phone. We’ve been able to
take that and then scale it to all the what we call Zone Americas,
which would be both North and South America. Looking to next year,
we’re going to try to take that more globally into different markets
in Europe and places in Asia, Africa. That’s the type of scale that I
think we’re looking for, for augmented reality, where it’s providing
a lot of utility and a lot of value added back is something that
can– it’s easy to prove or there’s tangible ROI there.
Alan: So how are you measuring
ROI in that? If you have a measurement like decreasing sales visits
from 3 to 2, that’s demonstrable value immediately. But how else were
you measuring KPIs and ROI, in order to go and prove that business
case to your senior execs?
Richard: Well, definitely the
one around cost avoidance is a big one. And that helps speed things
along. But also when you’re getting, say, feedback directly from the
customers themselves or their surveys, whether it’s repeated
purchases, people that want to get more from that. So, for example,
last year we revealed a purchase for the Starbucks out-of-home
business and saying now instead of just the machine’s pairing that
with different products that could be on the Starbucks line, whether
it’s different coffee corners, and the coffee corner would be this
big cabinet that has a coffee machine there that people go
self-serve. You would see it probably in places like the US and
Canada, like in a gas station or something like this. Or even just a
different point sales material to help sell different packets and
stuff like that. Adding these tangible things over time to make it as
much of a sales tool for the Nestlé sales rep, but also as a tool
for the boutique owner or somebody that’s looking to sell this to
customers. Being able to see a swing in an uptick in net sales,
certainly that’s the golden goose, if you will, that that’s the one
that everyone’s chasing. And we’re starting now to see that over time
a little bit more. But for sure, what gets the lightning rod started
is to say we can reduce costs right away and provide a more enriched
tool. Let’s get that to get the ball rolling and then we’ll see the
sales you brought over time. And we’ve been able to measure also
metrics not related to finance or numbers. But if we’re able to say
this many people are scanning different environments or scanning that
leaflet or using world tracking, this is the amount of unique people
are doing it versus the number of times they’re scanning. You can
measure how many times people are going into stores, how many times
they have to scan these things, how many people are doing it versus
how many times they are doing it. You could draw those numbers back
to “OK. Did this lead to a sale or not?” And so that’s the
kind of metrics that we’re starting to collect across the board.
Alan: How are you tracking that?
Are you doing it through your sales force or whatever your CRM is?
Richard: Not so much CRM. So I
think CRM integration is more of a long term play for us. And it was
kind of important to, I guess, have it in a siloed approach, at least
initially, just because we wanted to be fast moving and get the ball
rolling and helping develop the overall business case. So when we
look at, say, a big CRM integration, especially with a company like
ours, where you have a CRM tool that supporting tens of thousands of
employees, pushing AR integration from the get-go to that, that’s
probably going to be a project that’s going to be 18 months long. So
where we were more starting first is, let’s prove the concept around
augmented reality first and see if that provides value. We can
probably do that project or that first test and learn in around eight
to ten weeks and then build upon that. And then we’ll have this kind
of convergence down the road to say, “OK, everyone’s agreeing
that this provides a lot of tangible value. Now it’s time to
integrate this as just a function or feature of the CRM tool.” I
would more say looking at augmented reality that way, is not the AR
tool that we’re adding the CRM functionality to. It would be more
long term. How do we add the AR feature to our CRM tool? I think
that’s the more long term play. But in terms of adding value now,
it’s certainly the AR providing that as a standalone.
Alan: Yeah, I think that– this
has come up a few times, where making AR for the sake of AR is
interesting, but when you make it as a subset of the bigger sales
tool and say, “OK, well, it’s just a feature within the sales
tool.” You’re still gonna have printed materials, you’re still
gonna have all these other things. It’s just another amazing tool in
a sales person’s arsenal.
Richard: Exactly. It will never,
I don’t think — at least for the foreseeable future — it’s never
going to be the end all, be all and replace everything. It will
certainly always be — at least initially — one enhancer to
everything else we have going on, and more competitive advantage
where other companies are not using and you are, you will have the
ability to differentiate yourself against your competitors, for sure.
Alan: So I have a couple of
technical questions. How many products did you– did you do the whole
catalogue of these devices, or was it something that you rolled out
with the five models and then slowly kind of iterated on? And so
that’s question one. Question two is, where did you get the models
from? Was this from your manufacturing partners? And then the last
question I have is, what was the platform you used to make this
happen?
Richard: Yeah. So for the first
one, we started off small, of course. So we did two models to start
saying, “OK, let’s do a coffee machine and a juice machine.”
So that was what we started off first. And with that initial tests,
with those half a dozen sales reps, as I said, it was just around
these two models. Let’s get it out there. Let’s see if there’s
tangible value added back. Let’s get that feedback and see if we
should go from there. Because we were also in a place like, “OK,
we’re assuming this is going to work, but we need to actually put it
out there before we make a large investment to justify us making a
bigger investment.” When we did see it was adding value, we
said, “OK, let’s start to roll this out to different regions.”
So starting in the US, we started then to roll it out into Latin
America, countries such as Chile, Brazil, Peru. And we’re looking at
saying, “OK, let’s add a couple more models, see how this goes.
Maybe there’s different makes from market that we had to take a look
at.” And actually, one of the challenges that we did find at the
time was when you look at the maturity of smartphones around the
world, it is getting up there, where a lot of phones will have ARKit,
of course the new iPhones, ARCore for a lot of newer Android models.
When you go to certain markets, like Brazil, we’re discovering a lot
of sales reps are being issued phones that are 5, 6 years old.
Sometimes we didn’t even have gyroscopes in them. So we had to kind
of take a step back and say, “Well, OK, we’re making maybe some
assumptions that this is going to work out the box. We’re going have
the best experience. Throw in some world tracking through ARCore on
Android.” And we quickly had to be like, “Whoa, OK. We have
to maybe take a step back and think about what’s the right way to do
this.”
Alan: Hold your horses, man.
There’s still people on texting phones.
Richard: Of course, of course. It was quite a lesson learned for us to say, OK, let’s make sure that we’re identifying this up front. We’re building a solution that’s holistic for everyone. So now we have a bit of a mix of different, say, tracking mechanisms or triggers that would enable the experience. But from there, after we did have that fallback option for Latin America, that’s when the brand was more convinced to say, “All right, let’s start to do this for this entire region with all the different machines and coffee corners.” And you look at a total volume for that, you’re looking at around two dozen different models as well. For different use cases outside of, say, this Nestlé professional who sells things. If I take a look at Nespresso, for example, this is something when we look to say, OK, how do we enable quick look or WebAR functions on our e-commerce sites, we are looking at doing the entire range of espresso models and that’s up to around high 80s, 90s, different models that we have with different accessories, different colors, things like this. So we’re starting to kind of get to a point where a lot of people are seeing the value for it and they’re kind of going all in and saying, let’s throw it on our website. Let’s try it out. Let’s get a lot of different feedback and change it as needed. When it comes to the models themselves, it’s a very interesting question and another lessons learned that we had internally for that, because, as you know, there’s almost hundreds of ways to create the more raw CAD file for complex machines, say like coffee machines like Nespresso or things like this, but even point of sales all on this line. So we were dealing with a scenario, where we were getting them in any different type of format in 3D quality that you can imagine, often something as big as multiple gigabytes, something you can’t say put into an AR experience that you expect the consumer is going to deleverage. It wouldn’t work that way. So we had to do a lot of what we called 3D model curation. So we would have to either take the existing model or know what the model that they were trying to bring to life in AR have a lot of the 2D high-res photos of them. Because often you’re selling something like a coffee machine, you have an e-commerce site that’s also selling this. And there’s a lot of photos that have from different sides of the machine. Those have been dumbed down for more high quality ones that have been taken in a product shoot somewhere. So we would have to take either that existing 3D model or 2D photos of those machines, the different dimensions of them and then go through a process of recreating that model that’s kind of AR ready, good to go, something that’s 2 to 3 megabytes, tops that is in kind of glTF, GLB format, something that we can plug and play really easily into an AR or VR experience. So I think for us I think the industry will eventually move towards something where there’ll be some type of automation for that, and an ingestion engine, probably, for taking certain models from different formats and then spitting them out into something that’s AR/VR friendly. And you look at companies like Autodesk have platforms that they’re developing, that say that the goal is to do that. For us, we were more partnering with different providers that had capabilities in that space, to be able to help us bring that to life. People that we’re building their startups or business models around that specific use case, that there’s going to be a content– well, there is; there is a content problem of trying to develop a lot of these things at scale. And they’re gonna help for the first start of that, providing different resources that can create these from scratch, with a more of a long term goal to how do we automate that in the future.
Alan: That’s everybody’s goal. I
actually just saw an article yesterday that Nvidia is working on
creating 3D models from 2D photographs.
Richard: Yeah. So something like
that. When we see stuff like that, it definitely gets us excited of
looking at scale, because you said in the beginning Nestlé, this
huge conglomerate, we have 2,000 brands, we operate around 100, 80
countries around the world. In order to do this at a massive scale,
you’re looking for all the tools that you can get in order to reduce
the burden of creating a lot of this stuff. So being able to do what
you said with Nvidia, and create a new model from a 2D image, that’s
something that would definitely help along those lines.
Alan: Yeah, that’s like the
penultimate if we can get to that. *When* we get to that. It’s not
even an if anymore. It’s like there’s so much research being done in
this. I have a whole folder on 3D from 2D images, so there’s lots of
people working on it. You talked about kind of working with startups
to help get you where you need to be. What is the platform you’re
using to distribute this? Because you talked about quick look in AR.
Are you just using kind of Apple’s framework for that? And then what
about people on Android? What is the actual nuts and bolts of it?
Richard: We do look at
partnering on it with different startups or platforms that are either
emerging or established in the space. Every company has to be a
digital technology company today. But again, at our core, we’re a
food company. We don’t have, say, teams of developers that are doing
this space. We’re not developing our own proprietary AR tracking
technology or things like that. So we are leveraging what’s out
there. So whether we’re working with, say, I would call them like AR
content platforms. For example, we do some work with Zappar as one of
them. There are some more emerging platforms they’re looking at from
an e-commerce perspective, which is great from our end, because
obviously there’s a lot that needs to go into educating different
marketers and people in Nestlé. People get excited about this. But
again, you have to bring it back to that tangible use case, of where
it’s going to provide value.
Alan: So easy for us to chase
the shiny object.
Richard: It really is. It really
is.
Alan: And it’s interesting. Five
years ago we were doing 360 videos and then we said, “OK, well,
this is easy. Move on to something more difficult.” We then did
videogrammetry and volumetric capture, and then we did incredibly
complex 3D models. And then VR where you get right in. And it turns
out the 360 video stuff we were doing years ago is what people want.
Richard: [laughs]
Alan: They’re like, “Oh,
that’s perfect for what I need!” And I’m like “Oh man…”
Richard: Yeah. There’s still a
lot of tangible value in doing that.
Alan: Yeah. One of the things
that I think is valid for Nestlé — and for every company — is a
simple 360 tour of the facility for new employees. Just simple,
here’s what our facility is, here’s where the kitchen is. You can
kind of navigate around. Even just a Matterport tour of the facility
before you even get on on site, because a lot of the first few days
of new employees roll is wandering around, trying to find out where
to go. So if you could shorten that, put in VR and let them do that
at home, then you’ve just saved a couple of days of running around.
Richard: Exactly. And that’s
actually something we’ve done a bit with Purina in the US– that’s
Purina, the pet food company. They’ve done some– a lot of work
around. They’re called VR factory tours. So putting employees or
customers into their different factories, having them navigate
through it, interact with some things, learn a lot about them. From
an employees side, sometimes it’s people that support these guys,
maybe from the IT perspective or HR perspective day-to-day. The
factory employees that you have to have access to the different tools
and things like this. They never actually been to the factory
sometimes. It sounds kind of crazy, but when you’re working company
as big as ours, that does happen. So just being able to put them in a
VR experience — and for this they use the Quest in order to do that,
something that they can buy a couple of, carry around, bring them
around the office, things like this — and put them to that tour, it
gets a lot of tangible feedback. But at the end of the day, it’s a
high quality 360 video that has some interactivity that’s been laced
throughout it. But it is really effective content, it’s really good
and it incites emotion and it gets the point across. So–
Alan: We don’t need to
overcomplicate things. Actually, you’ve said that twice now. The 360
video stuff works, and it’s simple, and it doesn’t need to be crazy
complex. And then also with the stuff in the AR quick look stuff, it
doesn’t need to be super complicated. You don’t need to have a multi
exploded view of the product. You just need to see it on the shelf in
the right size and say, “OK, that doesn’t fit.”
Richard: Exactly. I mean,
there’s always a time and a place for the more enriched complex
experiences for sure. But yeah, often what consumers want to see,
they want to see, “Hey, buying a new coffee machine. Is it going
to fit in my counter? Am I going to like how it looks?” Things
like this. It’s a very quick five to six second decision, whether or
not they’re going to move forward.
Alan: How are you guys using
this kind of on the enterprise side and in your factories? Are you
using AR at all on the factory floor?
Richard: Absolutely. And where
we look at it a bit differently from what I was discussing before
is– so, if we take a step back, look at Nestlé factories, we have,
I think, around 400 — don’t hold me to this number — I think it’s
418 factories around the world, 300,000 employees. Whether or not my
team exists here in Barcelona, somebody has used augmented/virtual
reality in this organization before. So a lot of what we’re doing
when it was coming to the enterprise side initially was trying to
understand who’s been doing what, what ongoing experiences have been
happening, who’re the evangelists in this company that have actually
tangible hands-on experience of having it at the factory level. And
when we did this kind of analysis, we did see over the past half a
decade or so, we’ve had over 30 different experiments with different
form factors like headsets, whether they’re mixed reality or
smartglasses, different vendors out there. I sometimes laugh. If I go
to AWE and I meet all the different enterprise vendors there,
everyone’s going to tell me they’ve worked with us before, even
though it’s the first time I’ve met them. And I say, “I’m sure
you have.” Because there’s a lot of people out there that are
interested in this, whether they’re testing it on a small scale or
they want to expand. So we had to get a good look at that first and
what we–
Alan: How did you collect that?
Richard: So the way Nestlé
works, as well as we have people on the factory floor, that are a
part of what we call our technical production organization. They have
some kind of co-pilots in these factory areas, part of our R&D
organization, specifically an organization called PTC, which is
Product Technology Center. So they often try out new digital
technologies in these spaces to see if they’re viable for factory
floors. So what we’re finding is often, yeah, they were testing a lot
of this stuff out. We had to do a lot of internal networking. And we
actually also implemented across the organization, not just
augmented/virtual reality, it could be AI, digital twin, robotics,
things like this. We’ve kind of implemented the internal what we call
an innovation repository, a space where people can submit the
experiments they’re working on, what they’re trying to get out of it,
where they are in their experiment. Just to give exposure to other
parts of the organization. Because often where you’re seeing, say,
you take remote assistance in a factory, we’re probably repeating the
same experiment 15 different times in 15 different locations around
the world. But these teams weren’t talking to each other. So just
getting an exposure to different teams to say, hey, you want to do
this? These five teams also did this. This is their success. This is
their failures. At least build off of that.
Alan: Yeah, of course, that
makes way more sense than being stuck in pilot and POC purgatory
forever in every division. I actually have a question about the
innovation repository. How do you standardize it, so that it’s easy
for somebody to look at the results? Like do you have a standard form
or–?
Richard: Yeah. So we use a
platform called Startup Flow for that, that allows us to kind of have
a good view–
Alan: What’s it called?
Richard: Startup Flow. One we
started using last year. But it gives us a good kind of holistic view
over a lot of different things that are going on. And there was a
campaign to say, “Hey guys, put your stuff in here. Let’s not
reinvent the wheel.” Sometimes it’s a difficult conversation
because like I said before, this is an inherently cool technology.
And sometimes you don’t want to be told you can’t do it because
somebody else has done it before. We did get a lot of people to
understand the inherent value of that, and it’s not to say that
“We’ve done your use because before we’re not going to do it
again.” More often, I think the conversation going into this
year and next year is once you start to scale this across all the
organization of people that raise their hands, where again, you can
find a lot of tangible value added back. This just proves the use
cases there and that it’s going to provide us a lot of different
value.
Alan: I was just looking at the
Startup Flow webpage. What a great tool. Wow, super cool. Measuring
the ecosystem of startups and new technologies and then measuring the
KPI as metrics on one single dashboard. That’s awesome! That’s value.
Anybody who’s listening, startupflow.io. I mean, that’s worth the
value of listening to this podcast in itself.
Richard: Yeah, it’s a good tool
or something. Yeah. We’re using a lot in our organization.
Alan: What other ways are you
using this technology? Are you using it for training?
Richard: Absolutely for
training. So one example being a couple of weeks ago in a factory
here in Spain, in Girona, and they’re kind of mended them in a lot of
different factories around the world to follow what is kind of TPM
certification, which– it’s not my forte, this area, but I believe
it’s for Total Production Maintenance. Or Management. It’s
essentially certification for being kind of a best in practice
factory. We have zero defects across the board. And one of the areas,
one of the steps into getting to there is that they have to have an
on-site, say, training center where employees can go and self-train
themselves in different areas, whether it’s safety areas for the
here’s what you do in a confined quarters. Here’s what you do when
there is a explosive situation like a gas leak. If you’re in a height
situation, where you’re up somewhere high, it’s not secure. They had
to have this kind of facility onsite, that people can go to and learn
as much about that use case as they can, and train themselves up.
Now, the factory that we’re talking about is one of the more advanced
ones in the world, it’s coffee factory. It produces a lot of the same
Nescafe, Nescafe Dolce Gusto for good parts of the world. But going
back to we have a ton of factories, right over 400. Not every factory
has the ability to build a huge training center on-site. They need to
find creative ways to essentially get this type of material out
there. So using something like virtual reality for that, where to put
them in these scenarios now where the form factors are. We look at,
say, stuff like the Oculus Quest, where you can just have the
headset, controllers, the certain amount of space around you. You can
pop into a training session and it’s something that you can have a
lot of. You don’t have to set up a lot of different gaming PCs in
order to enable that. That’s something that brings a lot of value.
You don’t need as much space in order to do that, and to have that
type of trainings on-site. That and going through a height training
or explosive training in VR, probably it looks much more of a
emotional and personal impact to you, like a heightened sense of
receiving that content than watching a video or reading a PowerPoint
or something like this. It’s certainly something that you could take
a look and say it’s a much more effective way of learning that
content or material, retaining that information over time.
Alan: Are you guys measuring
that? Do you have– is it anecdotal? Are you measuring KPIs against
that?
Richard: Right now, we’re still
starting out that use case as well. It’s gonna be something that
we’re building Q1, Q2 next year. We have done some other safety
training in the US with their waters division. That’s also looking at
it from that scenario, like pointing out– different scenarios where
on the factory floor you have to point out different safety
violations, things like this. I believe they are making the KPIs more
on how much has it reduced the number of safety infractions or
reported infractions in the factory over time, of the amount of
people they have through the training versus the amount of reports
that they get. So it’s something that it’s still emerging for us a
bit, but we see a lot of big potential. And I think 2020 is where
we’re going to be looking at it a lot from scale.
Alan: And now, is this something
that– again, are you working with vendors there or are you trying to
do it in-house?
Richard: Mostly, again, with
vendors. And for us, we’re testing and learning with a few different
ones, but certainly it’s something that we’re keeping an eye on going
into next year. And I think just from the point of vendors, too, I
should make a note. As for a lot of these scenarios, we do– let’s
say we have global vendors that we love to work with, but we’re
always constantly working with different startups in the different
communities, because for technologies like this, everything is
changing every three to six months. Companies are rising, falling,
pivoting, being bought, picked up. So we–
Alan: Yep.
Richard: Yeah. Certainly, we’re
in a scenario where we try to work with as many people as possible.
The procurement team probably won’t like me saying that. But we do
try to. [chuckles]
Alan: One of the problems we’ve
been trying to solve is, how do you disrupt an industry constantly
and consistently disrupting itself? And so what we’ve come up with is
we’re working on a platform marketplace for training and learning
technologies. Because — like you realized — companies are coming
and going. Technologies are being invented and displaced all the
time. So rather than try to invent all of the different technologies
and try to keep on top of it, we’re building the platform that
synchronizes with the CMS, it synchronizes with the elements that
walks the customers through that, and keeps them always on the most
cutting edge platform across whatever device happens to be the
hottest thing of the year. And so that’s what we’re doing on a per
user basis. Then you can just say, OK, we’re gonna use VR and AR
across our whole enterprise. And we now have x number of employees
doing it, x number of headsets. And then it’s an ongoing thing and
you have access to all of the startups that have been pre-vetted,
rather than try to go through this selection period. As you know,
it’s challenging, right?
Richard: Absolutely, no. That’s
something I would love to take a look at that. [laughs] Later, after
when it comes out. It’s certainly–
Alan: We’ll talk offline.
Richard: There you go. There you
go.
Alan: I didn’t want to highjack
this podcast. Like, man, this is exactly the problem we’ve been
trying to solve is, how do you how do you overcome the point
of–technology is changing, and companies are being bought and sold,
and everything. And look, from our standpoint, we’ve started our own
accelerator. You know, you’re part of it.
Richard: Mm-hm.
Alan: And having this
accelerator to find and identify this talent, make sure that they are
able to deliver at the level of the customer’s requirements. Like,
you guys have 400 factories. That is a different conversation than
one company that has one factory.
Richard: Exactly.
Alan: And maybe wants one
training exposure. And then the other side of it is being a managed
marketplace. What are the things that you guys are creating in
safety? Let’s say, for example, a factory or warehouse for fire
safety, or height safety, or driving. What can then be reused and
resold to other companies? Nestlé is different because you have 400
factories. What can be then packaged and sold to all the factories?
But also what can be generisized and resold, so that everybody can be
safer at work?
Richard: I think, yeah, for at
least my perspective coming up with a lot of this content, it should
be something that at the end of the day, looking at proprietary I
don’t think is the right way to do it. If it’s for something that’s
safety, that’s going to address scenarios that you’ll find in a lot
of different factors — majority of factories — it’s something that
if we could help build the ecosystem of understanding on that and
it’s something that other, CPGs or fast moving consumer good
companies can leverage as well, I think we’re OK helping pave the way
for that. I am passionate about this area, and I think all boats need
to rise.
Alan: I couldn’t agree with you
more. And the interesting thing about this industry right now — and
it may change, because as the industry goes from nascent to prolific,
the gloves come off, so to speak — but so far, every single person
in this industry has been exactly that mindset. How do we all work
together? Because this is not a net sum game. More wealth will be
created in the world in the next 10 years than all of previous human
history. So we’re not scrambling to try to– “I get 10 bucks and
you have to lose 10 bucks.” This is not the world we live in
anymore.
Richard: Yeah, agreed.
Alan: You have a unique view
that most people will never have. You are kind of running VR and AR
for one of the largest companies in the planet, across all sorts of
different ways, from marketing and sales, to enterprise and training.
You really are the poster board for VR and AR in an enterprise, in a
company. So what problem in the world do you want to see solved using
XR technologies?
Richard: Thank you, no pressure. [chuckles] There’s two that come to mind. So one that may be more unique to us is when it comes to the food industry or the TP industry as well. Nestlé has made a 2025 commitment to make all their packaging sustainable, 100 per cent recyclable and things like this. If XR — through augmented reality or virtual reality to help sell a story, things like this — can help around helping people recycle. Scanning more about, say, a plastic bottle. Getting more information about it, directing into the nearest recycling center, gamifying it somehow to reward people for doing that. I think that’s something as powerful. And using the reach that we have, we have a billion individual packages that we generate every day, to get the communication out on that and tackle a problem that, frankly, we help generate.
Alan: The Nestlé blockchain
gamification recycling game. Ooh!
Richard: There you go. Yes. This
is something along those lines. Of course, yes. So I think having XR
tackle that or be a piece in the puzzle for doing that, I think would
be something that may be more uniquely to us and more powerful more
across the industry, I think. Helping with the massive upskilling of
the workforce, whether it’s robotics, removing repeatable tasks. That
example of the factory that we’re there in Girona. They’re using
cobots, they’re using different automotive forklifts, things like
this. And often the feedback is what happens to the work versus this.
And you get mixed tags sometimes, but they’re like, this frees them
up for doing other tasks that we need done or we freed themselves up
to doing more advanced things that we want to do. I think using
AR/VR/XR across the board to help skill current workers or train new
workers on next level skills that they need to do, I think, is also
something that’s going to be unique to this medium, this industry,
where this will provide a lot of tangible value as a kind of
learning, training or computing platform of the future.
Alan: Well, I’d have to say that
hats off to you for those amazing opportunities, because telling
people– well, first of all, Nestlé– hats off to Nestlé for
recognizing that they’re causing the problem — or part of it — and
saying by 2025, we’re going to solve the problem, and at least fix
it. Because this is a global problem, we need to fix it. And we all
need to come together. And it starts with big companies who can do
it. And Nestlé’s in a position to do that as well. So hats off to
you guys for for taking those initiatives, and really excited to see
where that goes. And if anything, if we can help in any way, I’m
happy to do so.
Richard: Absolutely.
Alan: Thank you so much for
joining me, man. This has been amazing.
40 min
February 11, 2020 Making a Pair of Ray-Bans Act Like a HoloLens x50 with Edge Computing, with Verizon’s TJ Vitolo
Verizon’s XR development lead, TJ Vitolo, dreams of a day where he can download an entire TV series in an instant, or visualize info about the entire world with AR glasses, even living in a connectivity dead zone by the beach. In his position, he’s able to work to make that dream a forthcoming reality by developing the technology that will make 5G possible.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today, I’ve got an
amazing guest, TJ Vitolo. He is the director and head of XR
Technology Development at Verizon. Today, he leads the commercial
strategy and product execution behind Verizon’s VR, AR and 360
organization environment. Recently, TJ and his team launched AR
Designer, the world’s first streaming-based AR tool kit that allows
brands and developers to quickly and easily create augmented reality
experiences, with no technical expertise. You can visit Verizon.com
or envrmnt.com. I want to
welcome TJ to the show. Welcome.
TJ: Hey, thanks for having me,
Alan.
Alan: Oh, it’s my absolute
pleasure. I’m so excited to have you on the show. This is like– all
the things you guys are doing, from working with the accessibility
team at Cornell Tech, to your acquisition of Riot, to working with
the Sacramento Kings, Yahoo! News. There is so much going on at
Verizon. You want to just give us a high level summary of what you
do, and what the plan is at Verizon for introducing 5G and XR?
TJ: It’s quite dynamic here. You
know, the VR space is ever evolving. Teams that do a number of things
within VR here. But specifically you mentioned Riot. Between our team
and Riot, we manage both of the content and creative end of XR, and
that’s Riot. And our team manages the technical– technology side of
virtual reality. So really, my team is focused on building tools and
enablers, systems, platforms on the 5G network, sort of the
underlying side of XR, to help accelerate and grow the adoption of
the technology. On the other side, Riot’s all about the product and
the creative storytelling around VR, which really brings these things
to life for people.
Alan: So you’ve got both the
technical side and then the creative. And this is something that I’ve
been harping on with customers as well, and just the industry at
large: that this industry is no longer about just making products.
And you look at the VC investments and they’re investing in platforms
and products, but you still need people to create the content. And I
think you guys have found that balance with Riot. What do you see as
kind of the future of how we create this content, is it going to be
user generated versus studio content, or a mixture of both?
TJ: It’s going to be a mixture
of both. User generation is quite difficult today. One of the
products you mentioned, we launched was AR Designer. And really the
foundation for that was to put the power of augmented reality and
virtual reality into the hands of even the most common user of
technology. We built this platform initially with the mindset that
schoolteachers– and not by any means that they’re simpletons, but
the fact of the matter is they’re teaching students, young children,
and they’ve got to have a very effective way to do that, efficient
way to do that. And so when we were building this tool, we baseline
on children as the audience, schoolteachers as the user of the tool,
to produce something that’s really effective. So I think you’re going
to see as VR/AR becomes more ubiquitous, access is going to be much
greater, and more in the hands of users. At the end of the day,
there’s always going to be the community outside of the content or
the UGC community producing content. And I think those are the folks
who are going to synthesize really compelling, powerful stories to
users to grow that adoption. So I think you’re seeing a lot with UGC,
where it sort of leads the way to broader, more institutional
creation of content, but it could very much see a [inaudible].
So walk me through your platform that
you guys have built. Was it in the market already or– walk us
through that.
It was in the market. So we pulled back
on that platform specifically because we had changed the strategy of
our team. Initially, I was brought into the organization around
commercialization for Envrmnt — which is Verizon’s XR organization
— and we wanted to generate revenue off of the XR ecosystem. And
there’s a fair amount of money out there to be made. But at the end
of the day is when we started to launch our commercial products, we
started to build up and prepare for our 5G launch strategy. And the
task of my engineering team was to go down a few different levels in
the technology stack, and start building platform enablers into the
5G network that will drive the adoption acceleration growth of AR/VR.
So the tool we still actually use today, we’ve got over 10,000 users
internally in Verizon that use it across our training organizations,
our HRO organizations, our network operations organizations. So it’s
been very successful. There is still a plan to commercialize that in
the future, but the idea was that we wanted to pin it against our 5G
launch, to show what 5G can do for the XR space. That’s where I’m
super excited, it’s about what 5G specifically does for XR technology
moving forward.
Alan: Absolutely, one of the
videos that I watched of you was a retail demo, where you took a
phone — just a regular phone with 5G — and you pointed at some
products on the shelf, and it not only recognized one product and
gave you like that standard AR image recognition and showed some
overlay information, but it recognized all the products at once. And
I thought that was really a great way of showing how 5G will enable
so much more than just simple AR that we’re used to now using our
phones. And then as that moves to glasses, you’ll be able to walk
down and say, “I’m on a keto diet.” and walk down the
aisles and anything that’s keto will show up in green. I think that’s
where it’s going. And that demo was really incredible.
TJ: Thank you. Yeah, I think AR,
from a mobile standpoint, has been put in this bubble, because of 4G.
And that’s one of the examples of what 5G is going to do to AR. It’s
going to make it highly functional, highly useful, and a lot more
entertaining in that space. Computer vision, graphics rendering.
Those are the two sort of fundamental underlying technologies between
virtual reality and augmented reality. And what we did there is that
we expanded the capability of computer vision by offloading what
typically is done on a mobile device over the 5G network — so
extreme amount of bandwidth, extremely low latency — to a network
node that sits within our network, that is very high powered from a
processing standpoint. It allows us to offload all of that computer
vision information and provide a response back in real time. This is
something only possible over 5G and only possible with our net edge
network. Fundamentally, what this does is, it ends the current
limitations of augmented reality, blows them out of the water, in
terms of their limitations.
Alan: You know, you’re taking
all the compute power off the device, and putting it into the edge.
You’ve been recognized at the Edge Awards already, winning Best
Contribution to Edge Computing for R&D and then Greatest
Commercial Potential for Edge Concept. So you guys are clearly
leading the way for this. One of the things I saw last week in Wired
was startups building a new chip. It’s an artificial intelligence
chip, and it’s the size of an iPad. Rather than everybody’s trying to
make them smaller and smaller, these guys went the opposite way and
made a huge chip, and it can do trillions and trillions of
calculations. But obviously you can’t put a chip the size of an iPad
in your phone. But having the ability to offload that to the cloud
and have the processing power when you need it, where you need it,
but only offloaded into the cloud was really incredibly powerful for
not only rendering, but also capturing the data that’s around you. A
lot of people don’t realize that as much data as you’re pushing from
the cloud down to the graphics processing and all that, you’re also
capturing data from point cloud data using RGB cameras, or all the
phones will start to have infrared camera sensors now. So being able
to capture that data, send it to cloud, make sense of it, all within
milliseconds, I think is really going to be a game changer for VR and
AR.
TJ: It’s a massive amount of
data, too. If you look at all those different sensors on those
devices, it’s crazy if you look at the future of volumetric video.
Well informed on Microsoft came out and said, hey, their studio does
two terabytes a minute of data capture.
Alan: Yeah, the Metastage.
TJ: With a handful of cameras
and that sort of tech, texture, and depth, and other sensors. But
you’re right, the thing that’s going to close the gap between really
powerful technology in your hands is extremely low latent,
high-bandwidth network connected to computer– very high power,
scalable computer network. And it’s not just AR/VR, right? It’s a lot
of things, although I’m focused on AR/VR. We are now going to be
putting supercomputers in everybody’s hands.
Alan: So, what 5G XR use case–
your focus is in XR. If you take 5G to the nth degree and 5G and edge
computing, you’ve got autonomous vehicles, you’ve got drones, you’ve
got– there’s all sorts of ways. But let’s focus on 5G and XR for a
second. What use cases do you guys as Verizon see as happening first?
I mean, we’re already seeing it in enterprise, where they’re using
heads-up displays to help field and service workers, and machine
workers, factory workers repair things, and see-what-I-see, and all
of these types of things. But what do you guys– what’s your roadmap
for the next 10 years, let’s say?
TJ: Yeah, it’s an interesting
question. So our organization fundamentally is working on the
platform and services that will enable very thin, lightweight
augmented reality or mixed reality glasses. So I think that’s one big
step, is to move away from the clunky form factor to something that’s
super sleek, and super powerful. So how can I have a pair of standard
Ray-Bans look and act like a Hololens times 50?
Alan: [laughs] Oh my god, that’s
a huge quote. [laughs] Think about that. “How do I make a pair
of Ray-Bans look and act like a Hololens times 50?” Oh man.
TJ: That’s the platform that we’re building, right? That’s the vision. Now, on top of that, once you do that, the world sort of your oyster in terms of what the use cases are. And enterprise is definitely the first entry point into that, because we will go through this evolutionary process with hardware for glasses, that it’s not just the compute– and we’re solving compute problems, but you do have to solve the display problem, and you have to solve a couple other things. But ultimately, at the end of the day, reduce battery power on that device, reduce battery size, reduce battery power by reducing compute on that device. And then ultimately, at the end of the day, through that step by step process, you get something in. But in the meantime, you’re going to get that adoption in the enterprise space. And so we look at — from a use case standpoint within our enterprise organization — things like worker safety, and obviously things around efficiency and improvements of workers within environments. And specifically in the industrial space right now, which seems to be where a lot of the opportunity sits, at least from companies that have been coming to us, interested in the space. There’s only so much that you can do to [inaudible] certain verticals within a market to adopt a technology. And a lot of them are a lot more forward thinking than others. So we start there.
Alan: It’s interesting that you
say that, because some of the industries that you actually think
would be the least technical — mining, for example, they haven’t
changed in 100 years — they were one of the first people to jump on
this technology, because they can use it so quickly and so easily in
manufacturing. Old school businesses that you wouldn’t think would be
technologically advanced are just making these leaps and bounds now,
it’s amazing to watch.
TJ: It’s amazing and it’s
amazing cultural thing to watch, in my view. It’s like you’d think
that these– some of these industries are so advanced and there’s so
much money, but they have old school practices. And then you look at
other ones who have been forced to innovate and change their culture
and adopt in these nascent spaces. And you scratch your head and say,
“Wow, that’s really interesting.” And so it kind of throws
you off-guard. But, you know, that’s where you have to go. You have
to go where the people have a sense of urgency and demand around it.
And then you make it happen on that front.
Alan: Interesting. I guess what
I’m trying to get at is, what are the 5G XR use cases that Verizon
thinks — or you think — will make the best use of the new networks,
of the new 5G capabilities?
TJ: We’re looking at a few with the underlying premise that you’re trying to merge the physical and digital worlds together. And so retail was– is a very big area for us, both front office and back office, or consumer focus and then back office. So if we’re looking at the consumer front for a second, we’re looking at the really interesting and I think most people can relate to use case. Here’s where I go into a retail store, and then I’m always on my phone looking at ratings, reviews, pricing information, and other things with respect to those physical products that are on the shelf. And I spend a lot more time on my phone than I actually do perusing and browsing the stuff on the shelf. And so really what we want to do is merge that physical and digital divide, by having a pair of mixed reality glasses that as you’re walking down that store — using a 5G powered headset — you’re literally taking in all the information within your field of view about a set of products and services. So now I’m standing in front of a set of consumer electronics devices, and I want to know which ones are the best rated, which one has the best value. All the stuff that’s typically online, now I can have all that stuff instantly overlaid on top of those products, whether it be makeup or electronics, or even clothing. And then take that off the rack and go and purchase it. So that’s one of the retail experiences. Other has a safety component to it, the one that you saw us demo. I have a family. I spend an inordinate amount of time at the grocery store looking at the backs of boxes, if they contain certain allergens for my family. Now I can tell my glasses to filter for products that are gluten-free, [inaudible], or kosher, whatever might be the case, then instantly everything within my field of view will light up based on those requirements.
Alan: Do you think that will be
driven by computer vision picking up the boxes, or do you think
they’ll be driven by companies, like the grocery stores — like Whole
Foods, for example — submitting planograms, so it knows what store
you’re in, the planogram knows where the boxes are, will it be
combination of both?
TJ: I think it’ll be a
combination of both, because you’ve seen it in the QR space. You’ve
seen stores do it themselves, you’ve seen third parties do it
themselves. Most of that information is actually publicly available,
those databases. And so third parties can actually easily construct
that software, but I think it’ll be dependent on obviously the
training and the learning of that object — a lot of those imagery is
publicly available — and then blend it with the publicly available
information for those products.
Alan: I know Google and Amazon
and pretty much everybody’s working on computer vision for products.
I want to point my phone at a pair of shoes and say, “What are
those shoes?” And right now we’ve got a device in our hands that
is pretty powerful and can do a lot of things right now. What are
some of the things that we can do right now with our phones, that
you’re seeing are emerging as killer use cases in this technology?
TJ: That’s a great question. Like I said, I think that going back to sort of the chokehold that current existing networks placed on augmented reality, there’s a couple areas where I see a big amount of potential for a mobile device. And I think a lot of that fits around potentially markets where you have growing economies. You look at APAC and other areas around the world, where they don’t have access to certain types of [inaudible] medical facilities. So one of the things that I saw that was really interesting, is how you could use a mobile phone and computer vision help diagnose patients, by using computer vision and artificial intelligence to look for signs that you wouldn’t necessarily be trained or have access to. One of the interesting use cases I saw was to help support a potential phlebotomist out in the field, where they’re using a phone to detect veins, so they don’t mispuncture a vein in the arm. They could do it right the first time, and completely limit the opportunity for infection.
Alan: I think that’s the
AccuVein system, isn’t it?
TJ: Mm-hm, mm-hm.
Alan: Yeah, it’s really great.
TJ: And I think that’s
transformative on a world basis, is that anytime you use technology
to add intelligence to give access to underserved or underprivileged
or markets that just don’t have the ability to. So we look at that
space, too. We look at– we talked about accessibility, I think, a
little bit. We look very much into that space also, as a way to just
improve in general. And I think we share a very similar feeling. It’s
about improving quality of life. It’s not about introducing one more
thing, one more piece of noise into the environment. How can we help
each other sort through daily life, whether it be from a medical
standpoint or being inundated with information? How do we make those
things simple?
Alan: Absolutely. And as more
and more people move into urbanized areas, there’s that culture shift
from living in the country to living in a city and– offline, we were
talking about taking our kids camping and stuff. My 11-year-old
daughter made this huge billboard poster and put it by the fire. It
said “No cell phones by the campfire.” And I think we’re
really getting to the point where the technology is pervasive. It’s
everywhere we are. My kids sit on the couch and watch TV with their
phone in their hands. And sometimes they’ve got an iPad and a phone.
It’s nuts. I don’t even know how they focus. And the other day, my
daughter was watching a show, and she had the show in a small window
and she had a game that was related to the show in the big window. So
she had like picture-in-picture, but the show wasn’t the dominant
part of it. And I thought that was really interesting, how youth are
starting to use these technologies. And we’ve done a lot of work on
delivering people entertainment content. Netflix is using AI
algorithms to give people better movies to watch. Amazon’s giving you
better algorithms to help you purchase better. What I think we need
to do is harness those technologies and give kids better ways to
learn. And I think these technologies can really catapult that. What
are your thoughts around VR, AR, AI in education and training?
TJ: Yeah, we we touched on this
a little bit. And I think the impact is absolutely [inaudible] that
space, specifically with convergence of people either domestically or
internationally from an education standpoint. I think one of the
things that you might have seen, our team produces a virtual reality
platform called Operation Convergent Response. And the idea behind
that was to aggregate or bring together a number of people with
different backgrounds and skillsets into a single virtual environment
— or a war room, essentially — so that they can help support a
natural disaster. So if there was an earthquake, bring an earthquake
expert, someone who’s an expert in fire, someone who’s an expert in
weather — or whatever might be the case — to help quickly triage
sort of a situation, without bringing them into a physical location.
That’s immense also. You can have highly custom education armaments,
where you’re bringing in specialists in different areas, that hold
all under sort of one umbrella. Let’s take VR/AR. Someone interested
in VR/AR, you could bring in an expert in virtual reality, expert in
augmented reality, expert in computer vision, and then you can
coalesce and bring all together for the students. Very specific
interests in that space, into that area, into that arena. And with
virtual reality, you can create a 100,000 square foot space in a
1,000 square space. You put 75 TV’s on the wall, you put one TV on
the wall–
Alan: Yeah, it’s great.
TJ: –you can literally create
the most dynamic environment that works for those students. And
touching on artificial intelligence for a sec, one of the most
amazing things I think that comes out of virtual reality training —
and also in the medical space and other areas — is the ability for
analytics platforms to look at every single piece of interaction
that’s going on in that space. What does that yield at the end of the
day, it yields a very efficient and effective way to help you
understand where you’re improving, where you’re falling behind. And
it’s amazing because in essence, in a typical education environment,
you’ve got to rely on a teacher — or a boss, as might be the case —
to provide that feedback, and you can’t do that for 30 students in a
classroom. But if you’ve got an analytics platform that’s in each one
of those individual students on a case-by-case basis, you can then
custom produce areas in a report where they should be improving and
moving forward. And the amount of advancement we make is just because
of that little improvement. I think it’s absolutely massive.
Alan: You nailed it. By changing
the way we teach and using exponential technologies. It’s not even a
10x improvement. It’s a 100x improvement, a 1,000x improvement
because we’re not even scratching the surface of things like how
people learn or when they learn. I learn better, maybe from 10 AM to
11 AM. That’s my maximum capacity, so all the hard stuff maybe I
learned then, and maybe somebody else learns in the evening. Maybe by
using galvanic response or measuring your heart rate or your
biometrics, you can then deliver content that is highly personalized,
highly contextualized and delivers it at the time of maximum
absorption or retention. And we’re already seeing across the board 20
to 100 percent improvements in retention rates in training. One of
the things that you guys did was use VR for hostage and robbery
training. How did that come about?
TJ: It was an interesting one.
It was actually during our early commercial stages, we were looking
at different verticals in areas where we can use technology to
obviously improve awareness and other things that happen in our
retail environments. It wasn’t specifically around hostages, more
like store robberies, which don’t happen often. But obviously, when
they do, how do you train someone in a traditional environment to
address a burglary situation, especially with someone being held at
gunpoint? So we use 360 video. We use actors and we pull together a
very immersive experience around both an armed robbery and also
identifying risks for theft by immersing them in a 360 world. We did
monitor different bio signs. We also use an analytics platform that
allows us to do gaze tracking and other things. So you saw exactly
where they were looking. What they were interested in. Were they
looking in the right spots? Were they looking in the wrong spots? And
we were able to produce a relatively detailed report based on that,
and what that allowed us to do is it allowed us to understand, as
much as you could in a replicated environment, what is the normal
response for a retail rep when these situations occur? And then it
allowed us to cater a training program around that, that helped
better prepare them for a situation. So we use it as a way to gather
analytics, which then yielded a ton of information for us that we
could never capture in a — even if you did this in a real-life
scenario and you brought in some actors and you brought in a rep and
you them in– you wouldn’t be able to understand where they were
looking, how they were acting, how they were responding, without this
type of analytics in this virtual environment. So because of that, we
felt like we were able to compose a training module that was much
more advanced than what was in the market today.
Alan: How do you measure the
success of that? What are the KPIs that you guys were looking for?
TJ: That’s a great question. And
unfortunately, we provided a lot of the technical support and
background for that while our learning development team managed the
KPIs. But I think some of the key KPIs that we used was sort of this
post mortem experience where we then place them back into into that
experience with their training to see where there are improvements.
And specifically, we use the gaze data, the biometric data, to then
see how they respond in that situation. And I don’t know what the
specific performance improvements were, but I know they were
relatively more significant. That’s sort of what’s baseline for
typical training. Unfortunately, I don’t have that data, but the
process, the methodology, was to re-immerse them back into that
situation with the tools that they now have. Again, because it’s
case-by-case and see where they were looking, how they are acting,
how they’re responding, so on and so forth.
Alan: I want to shift away from
training, because there’s so much to unpack here in the long term of
this technology. 5G is really going to enable a lot. And one of the
things that we have to overcome is some of the challenges. What are
some of the challenges that you see — or Verizon as a company sees
— as standing in the way of the broad adoption of 5G-powered XR,
beyond cheaper headsets and 5G coverage? But what do you see as the
broader challenges around adoption?
TJ: I think adoption is
dependent on establishing a community that understands the
technology, in order to build on that. And that is the other half of
our 5G labs. I own the development side. We also have a 5G lab–
multiple 5G labs, spread across the country which provide developers
and enterprises access to this technology, so they can understand it.
So education is hugely critical. We are very immersed in the
technical side of things. So we’ve won those Edge awards, very
technical. And the work we do, while deeply technical, has to be
translated to a common level. And I think our biggest hurdle and
challenge is making that consumable to the end user. And we are
solving that through our 5G labs. We are providing developers with
training and access to 5G networks, providing enterprises with a full
view into the capability. So as much as we’re having this
conversation here, any CTO or CIO or even CEO listening, visit our 5G
labs. You’ll get a full view of what we’re doing and how it can apply
to your business. There are so many companies across so many
verticals that it really helps them understand and inspire what they
what they could do with the technology.
Alan: It’s interesting that you
guys created these 5G labs with the purpose of showing what the
technology can do. One of the reasons we actually started XR Ignite
was the same thing. We kept seeing all these amazing startups coming
up with crazy, amazing ideas on how to use XR and AI, but the they
were missing the business acumen to be able to take those
technologies and bring them to a commercialized state. We formed XR
Ignite to kind of help bridge that gap between corporations and
startups. And it sounds like you guys are doing the same. Sounds like
there’s an interesting fit there. We’ll talk offline and see if we
can collaborate.
TJ: Yeah, no, absolutely. I
think there’s there’s great opportunity for anyone looking and
interested in it.
Alan: So what is the most
important thing that businesses can start to do now to leverage the
power of XR, and AI, and 5G? What can they start doing immediately to
reap the benefits?
TJ: Yeah, I think any company
should — just like we do here — is to assign someone from the
strategy side of the business to either bring someone in or to do
their own objective (and to some extent, subjective) view of
collection of information. There’s so much data out there. There’s so
much information out there. You know, the stuff that I’m talking
about is very topical in the sense that this is stuff that you and I
have access to. And we’re very immersed in this space. Obviously, we
know relatively well, but anyone in any space could find the same
amount of information. I work at a very deep technical level, which
obviously makes not much sense to businesses until we translate that
into a business function at the end of the day. But one or two people
in an organization can compile some really compelling amount of data
and information that exists out there, to really help transform
businesses with this technology. So I think it’s just that first step
that you need to take to say AR, VR; what is this about? Right? What
does this mean to my business? And I think they’ll be surprised.
Alan: We actually started
MetaVRse as a consulting firm to help businesses understand how to
use this technology, because our long-term vision was always
education. And so we’ve kind of morphed into consulting on how to use
this technology specifically for education, training, learning
modules and stuff like that. But with XR Ignite, it’s pretty broad.
There’s so many companies out there building such great tech and
everybody is chomping at the bit to do start leveraging 5G. And I had
a meeting a couple weeks ago and I got to see the new Samsung s10 5G
phone, and it was the first time I’d ever held a 5G phone in my hand
and it kind of felt like this new renaissance of technology is
coming, and it’s going to come as a tidal wave, because it’s been
years that people have been working on it. I don’t know, Verizon’s
probably working on this for a decade. So it’s coming and it’s going
to come really fast for people. What can consumers do to prepare for
that? Because I think by having a 5G phone is great. It’s faster,
whatever. But really, what can consumers expect from that?
TJ: If you look at where we were
with 4G and sort of this scale and ubiquity of 4G were in that same
phase. — and not just in this scale, but also in sort of the
advancement of technology — I think one of the things… so just
sort of prefacing this, one of the things that is interesting is that
the iPhone didn’t release a 4G iPhone until about a year and a half
later. And I think once the network launched, I think part of that is
because there are ramp ups to this technology as it gets exposed and
becomes more accessible to developers, they could start building
experiences on top of that, that really validates the value
proposition of 5G. As it stands right now, the biggest value
proposition of 5G is just the ability to download an immense amount
of content in a quick amount of time. Right? So I want to download
entire season The Sopranos through HBO Go, or whatever might be the
case, before I’m about to hop on a plane or I’m going to make a split
second decision. You know, I don’t have to wait 10, 15, 20 minutes to
do that. I can literally do that within 30 seconds or less.
Alan: That’s insane.
TJ: It really makes on demand
“on demand,” right? It used to be that you had to plan some
stuff, or– one of the great things, I have got music service and
unfortunately where I am down at the shore on the beach, I don’t have
access to — because I’m pretty remote — to that network. And so I
want to be able to download an entire alternative 90s collection of
music of 10,000 songs in…a minute? Less than that?
Alan: OK. I got to just… what
the hell, man? I used to be a DJ for 20 years and I remember going to
the record store and you’d buy a record. Then I had a collection of
books of CDs, and I remember when my daughter was about 10 and I gave
her my CD collection, and she looked at me, she’s like, “what do
I need that for? It all fits in my iPod.” And now we’re talking
about downloading entire genre of music in seconds.
TJ: It’s quite crazy. Again,
that’s just scratching the surface of 5G. Beyond that, there are a
lot of other things that we foresee with the implementation of the
Edge network that we talked about and providing access to developers.
One of the things that I did wanted to touch on is sort of our last
piece a minute ago was that developers, the XR space is still nascent
enough that if you’re a business, their may [be]– or likely isn’t —
a solution for your business today. And so I’d encourage businesses
to go out there, understand the technology and build their own
solutions, because half of our practice, going back to our commercial
period, was a pro services organization and we were just all ears. We
would listen to what companies would come to us with challenges and
then we would solve those problems for those businesses. And one of
the benefits of being Verizon, a massive company that we are, is that
we’ve got so much diversity. We’ve got retail. We have network
operations, sales. Right? We can hear these challenges from all these
organizations and build solutions that no developer would ever really
think of to build. So I’d encourage businesses to go out there and
not just look for answers to their problems, but work with folks that
can solve those problems.
Alan: That was our entire
business model, was to create some solutions, sit and listen to
customers, figure what they wanted and build it for them and then
develop those into products. Literally. That was our our entire
business model, was just, listen! And that would create products,
which it has.
TJ: This is an SAP, right? I
know I need some sort of platform. I go to SAP and they’re like, I
get inside a box software and just customize it. Then that’s answer
for my business. That’s not the state of XR. And if you can jump on
that as a business, I think you’ll have a major upper hand like we do
here. So I’ll give you one example. We have a team that’s responsible
for going out and working with community. You have them understand
where we’re going to be putting 5G cell sites. And one of the things
they do is they hire a graphic artist, a number of graphic artists,
to go out and draw what a cell site would look like in a community
and they would do it in Photoshop and other tools right there. And
then they would bring it to the community board meetings. San
Francisco would be the case and say, hey, these are the sites that we
want to produce. Right? And this is what the tower is going to look
like on this building or on this light pole, whatever might be the
case. And it costs them on average just in this one section of the
US, $5-million a year contract, graphic artists. So what we did, and
what they asked us to do, is how do we solve this problem with
augmented reality? Can we just take the poles that they would draw,
import them as 3D models and then just drop them into the
environment, capture that photo with our camera and submit that to
them to the town planning meeting? We said, absolutely. The Verizon
rep goes out there, they go to a corner, they pick the pole that they
want. They change the color of the pole to match the environment.
Right? And then they set it right on the plane of the ground. And
then they capture a picture and they upload that picture for the
board planning meeting. And literally, that application probably took
a month to develop and saved in saving our business $5-million
dollars a year just in this one area. That was not something that
we’d ever imagined building. But we took that cost out of the
business.
Alan: Amazing. Some of the cost
savings that I’ve heard on different interviews on this podcast have
been insane.
TJ: It’s an insane amount of
money, for relatively simple implementation of the technology.
Alan: So with that, last
question; what problem in the world do you want to see solved using
XR technologies?
TJ: I want to see people’s lives
become simpler. Everything that we build, from a technology
standpoint, very complex in nature. I think the fundamental idea is
to make people’s lives simpler and to make people’s lives safer. I
think we’ve come to a point at technology just earns us so much that
you companies out there that are now creating artificial digital
clones of ourselves to manage the information overload that we have
coming in, so we provide them with a set of rules and
responsibilities that they could take on our behalf. And I don’t
think that’s necessary, if you would, in a way that filters out the
noise and makes everything [inaudible]. And I think what that is, is
that that’s blending the digital and physical by… I don’t want to
be constantly distracted by my cell phone when I’m camping. Right?
Because there’s something interesting that I just saw, maybe out in
the wild. Was that a deer? Was that whatever might be the case? With
glasses, I think that information is so passive to you that we no
longer have to be distracted by our device and we can tune in and
tune out things as needed. And my view at the end of the day is, we
strive for simplicity in our lives. And I think that’s what we want
to go with this technology.
Alan: Oh, I love that. Striving
for simplicity with our lives, using the most advanced technologies
in the world. I don’t know what better way to wrap that up, but thank
you so much. Thank you, everyone, for listening. If you want to learn
more about the work that T.J. and his team are doing at Verizon, you
can visit verizon.com, envrmnt.com, or verizon5glabs.com. I think
those are the three places where everybody can get as much
information as they want. Thank you again, TJ. This has been amazing.
TJ: Thanks, Alan. I look forward
to talking to you soon.
41 min
February 11, 2020 Making a Pair of Ray-Bans Act Like a HoloLens x50 with Edge Computing, with Verizon’s TJ Vitolo
Verizon’s XR development lead, TJ Vitolo, dreams of a day where he can download an entire TV series in an instant, or visualize info about the entire world with AR glasses, even living in a connectivity dead zone by the beach. In his position, he’s able to work to make that dream a forthcoming reality by developing the technology that will make 5G possible.
Alan: Welcome to the XR for
Business Podcast with your host, Alan Smithson. Today, I’ve got an
amazing guest, TJ Vitolo. He is the director and head of XR
Technology Development at Verizon. Today, he leads the commercial
strategy and product execution behind Verizon’s VR, AR and 360
organization environment. Recently, TJ and his team launched AR
Designer, the world’s first streaming-based AR tool kit that allows
brands and developers to quickly and easily create augmented reality
experiences, with no technical expertise. You can visit Verizon.com
or envrmnt.com. I want to
welcome TJ to the show. Welcome.
TJ: Hey, thanks for having me,
Alan.
Alan: Oh, it’s my absolute
pleasure. I’m so excited to have you on the show. This is like– all
the things you guys are doing, from working with the accessibility
team at Cornell Tech, to your acquisition of Riot, to working with
the Sacramento Kings, Yahoo! News. There is so much going on at
Verizon. You want to just give us a high level summary of what you
do, and what the plan is at Verizon for introducing 5G and XR?
TJ: It’s quite dynamic here. You
know, the VR space is ever evolving. Teams that do a number of things
within VR here. But specifically you mentioned Riot. Between our team
and Riot, we manage both of the content and creative end of XR, and
that’s Riot. And our team manages the technical– technology side of
virtual reality. So really, my team is focused on building tools and
enablers, systems, platforms on the 5G network, sort of the
underlying side of XR, to help accelerate and grow the adoption of
the technology. On the other side, Riot’s all about the product and
the creative storytelling around VR, which really brings these things
to life for people.
Alan: So you’ve got both the
technical side and then the creative. And this is something that I’ve
been harping on with customers as well, and just the industry at
large: that this industry is no longer about just making products.
And you look at the VC investments and they’re investing in platforms
and products, but you still need people to create the content. And I
think you guys have found that balance with Riot. What do you see as
kind of the future of how we create this content, is it going to be
user generated versus studio content, or a mixture of both?
TJ: It’s going to be a mixture
of both. User generation is quite difficult today. One of the
products you mentioned, we launched was AR Designer. And really the
foundation for that was to put the power of augmented reality and
virtual reality into the hands of even the most common user of
technology. We built this platform initially with the mindset that
schoolteachers– and not by any means that they’re simpletons, but
the fact of the matter is they’re teaching students, young children,
and they’ve got to have a very effective way to do that, efficient
way to do that. And so when we were building this tool, we baseline
on children as the audience, schoolteachers as the user of the tool,
to produce something that’s really effective. So I think you’re going
to see as VR/AR becomes more ubiquitous, access is going to be much
greater, and more in the hands of users. At the end of the day,
there’s always going to be the community outside of the content or
the UGC community producing content. And I think those are the folks
who are going to synthesize really compelling, powerful stories to
users to grow that adoption. So I think you’re seeing a lot with UGC,
where it sort of leads the way to broader, more institutional
creation of content, but it could very much see a [inaudible].
So walk me through your platform that
you guys have built. Was it in the market already or– walk us
through that.
It was in the market. So we pulled back
on that platform specifically because we had changed the strategy of
our team. Initially, I was brought into the organization around
commercialization for Envrmnt — which is Verizon’s XR organization
— and we wanted to generate revenue off of the XR ecosystem. And
there’s a fair amount of money out there to be made. But at the end
of the day is when we started to launch our commercial products, we
started to build up and prepare for our 5G launch strategy. And the
task of my engineering team was to go down a few different levels in
the technology stack, and start building platform enablers into the
5G network that will drive the adoption acceleration growth of AR/VR.
So the tool we still actually use today, we’ve got over 10,000 users
internally in Verizon that use it across our training organizations,
our HRO organizations, our network operations organizations. So it’s
been very successful. There is still a plan to commercialize that in
the future, but the idea was that we wanted to pin it against our 5G
launch, to show what 5G can do for the XR space. That’s where I’m
super excited, it’s about what 5G specifically does for XR technology
moving forward.
Alan: Absolutely, one of the
videos that I watched of you was a retail demo, where you took a
phone — just a regular phone with 5G — and you pointed at some
products on the shelf, and it not only recognized one product and
gave you like that standard AR image recognition and showed some
overlay information, but it recognized all the products at once. And
I thought that was really a great way of showing how 5G will enable
so much more than just simple AR that we’re used to now using our
phones. And then as that moves to glasses, you’ll be able to walk
down and say, “I’m on a keto diet.” and walk down the
aisles and anything that’s keto will show up in green. I think that’s
where it’s going. And that demo was really incredible.
TJ: Thank you. Yeah, I think AR,
from a mobile standpoint, has been put in this bubble, because of 4G.
And that’s one of the examples of what 5G is going to do to AR. It’s
going to make it highly functional, highly useful, and a lot more
entertaining in that space. Computer vision, graphics rendering.
Those are the two sort of fundamental underlying technologies between
virtual reality and augmented reality. And what we did there is that
we expanded the capability of computer vision by offloading what
typically is done on a mobile device over the 5G network — so
extreme amount of bandwidth, extremely low latency — to a network
node that sits within our network, that is very high powered from a
processing standpoint. It allows us to offload all of that computer
vision information and provide a response back in real time. This is
something only possible over 5G and only possible with our net edge
network. Fundamentally, what this does is, it ends the current
limitations of augmented reality, blows them out of the water, in
terms of their limitations.
Alan: You know, you’re taking
all the compute power off the device, and putting it into the edge.
You’ve been recognized at the Edge Awards already, winning Best
Contribution to Edge Computing for R&D and then Greatest
Commercial Potential for Edge Concept. So you guys are clearly
leading the way for this. One of the things I saw last week in Wired
was startups building a new chip. It’s an artificial intelligence
chip, and it’s the size of an iPad. Rather than everybody’s trying to
make them smaller and smaller, these guys went the opposite way and
made a huge chip, and it can do trillions and trillions of
calculations. But obviously you can’t put a chip the size of an iPad
in your phone. But having the ability to offload that to the cloud
and have the processing power when you need it, where you need it,
but only offloaded into the cloud was really incredibly powerful for
not only rendering, but also capturing the data that’s around you. A
lot of people don’t realize that as much data as you’re pushing from
the cloud down to the graphics processing and all that, you’re also
capturing data from point cloud data using RGB cameras, or all the
phones will start to have infrared camera sensors now. So being able
to capture that data, send it to cloud, make sense of it, all within
milliseconds, I think is really going to be a game changer for VR and
AR.
TJ: It’s a massive amount of
data, too. If you look at all those different sensors on those
devices, it’s crazy if you look at the future of volumetric video.
Well informed on Microsoft came out and said, hey, their studio does
two terabytes a minute of data capture.
Alan: Yeah, the Metastage.
TJ: With a handful of cameras
and that sort of tech, texture, and depth, and other sensors. But
you’re right, the thing that’s going to close the gap between really
powerful technology in your hands is extremely low latent,
high-bandwidth network connected to computer– very high power,
scalable computer network. And it’s not just AR/VR, right? It’s a lot
of things, although I’m focused on AR/VR. We are now going to be
putting supercomputers in everybody’s hands.
Alan: So, what 5G XR use case–
your focus is in XR. If you take 5G to the nth degree and 5G and edge
computing, you’ve got autonomous vehicles, you’ve got drones, you’ve
got– there’s all sorts of ways. But let’s focus on 5G and XR for a
second. What use cases do you guys as Verizon see as happening first?
I mean, we’re already seeing it in enterprise, where they’re using
heads-up displays to help field and service workers, and machine
workers, factory workers repair things, and see-what-I-see, and all
of these types of things. But what do you guys– what’s your roadmap
for the next 10 years, let’s say?
TJ: Yeah, it’s an interesting
question. So our organization fundamentally is working on the
platform and services that will enable very thin, lightweight
augmented reality or mixed reality glasses. So I think that’s one big
step, is to move away from the clunky form factor to something that’s
super sleek, and super powerful. So how can I have a pair of standard
Ray-Bans look and act like a Hololens times 50?
Alan: [laughs] Oh my god, that’s
a huge quote. [laughs] Think about that. “How do I make a pair
of Ray-Bans look and act like a Hololens times 50?” Oh man.
TJ: That’s the platform that we’re building, right? That’s the vision. Now, on top of that, once you do that, the world sort of your oyster in terms of what the use cases are. And enterprise is definitely the first entry point into that, because we will go through this evolutionary process with hardware for glasses, that it’s not just the compute– and we’re solving compute problems, but you do have to solve the display problem, and you have to solve a couple other things. But ultimately, at the end of the day, reduce battery power on that device, reduce battery size, reduce battery power by reducing compute on that device. And then ultimately, at the end of the day, through that step by step process, you get something in. But in the meantime, you’re going to get that adoption in the enterprise space. And so we look at — from a use case standpoint within our enterprise organization — things like worker safety, and obviously things around efficiency and improvements of workers within environments. And specifically in the industrial space right now, which seems to be where a lot of the opportunity sits, at least from companies that have been coming to us, interested in the space. There’s only so much that you can do to [inaudible] certain verticals within a market to adopt a technology. And a lot of them are a lot more forward thinking than others. So we start there.
Alan: It’s interesting that you
say that, because some of the industries that you actually think
would be the least technical — mining, for example, they haven’t
changed in 100 years — they were one of the first people to jump on
this technology, because they can use it so quickly and so easily in
manufacturing. Old school businesses that you wouldn’t think would be
technologically advanced are just making these leaps and bounds now,
it’s amazing to watch.
TJ: It’s amazing and it’s
amazing cultural thing to watch, in my view. It’s like you’d think
that these– some of these industries are so advanced and there’s so
much money, but they have old school practices. And then you look at
other ones who have been forced to innovate and change their culture
and adopt in these nascent spaces. And you scratch your head and say,
“Wow, that’s really interesting.” And so it kind of throws
you off-guard. But, you know, that’s where you have to go. You have
to go where the people have a sense of urgency and demand around it.
And then you make it happen on that front.
Alan: Interesting. I guess what
I’m trying to get at is, what are the 5G XR use cases that Verizon
thinks — or you think — will make the best use of the new networks,
of the new 5G capabilities?
TJ: We’re looking at a few with the underlying premise that you’re trying to merge the physical and digital worlds together. And so retail was– is a very big area for us, both front office and back office, or consumer focus and then back office. So if we’re looking at the consumer front for a second, we’re looking at the really interesting and I think most people can relate to use case. Here’s where I go into a retail store, and then I’m always on my phone looking at ratings, reviews, pricing information, and other things with respect to those physical products that are on the shelf. And I spend a lot more time on my phone than I actually do perusing and browsing the stuff on the shelf. And so really what we want to do is merge that physical and digital divide, by having a pair of mixed reality glasses that as you’re walking down that store — using a 5G powered headset — you’re literally taking in all the information within your field of view about a set of products and services. So now I’m standing in front of a set of consumer electronics devices, and I want to know which ones are the best rated, which one has the best value. All the stuff that’s typically online, now I can have all that stuff instantly overlaid on top of those products, whether it be makeup or electronics, or even clothing. And then take that off the rack and go and purchase it. So that’s one of the retail experiences. Other has a safety component to it, the one that you saw us demo. I have a family. I spend an inordinate amount of time at the grocery store looking at the backs of boxes, if they contain certain allergens for my family. Now I can tell my glasses to filter for products that are gluten-free, [inaudible], or kosher, whatever might be the case, then instantly everything within my field of view will light up based on those requirements.
Alan: Do you think that will be
driven by computer vision picking up the boxes, or do you think
they’ll be driven by companies, like the grocery stores — like Whole
Foods, for example — submitting planograms, so it knows what store
you’re in, the planogram knows where the boxes are, will it be
combination of both?
TJ: I think it’ll be a
combination of both, because you’ve seen it in the QR space. You’ve
seen stores do it themselves, you’ve seen third parties do it
themselves. Most of that information is actually publicly available,
those databases. And so third parties can actually easily construct
that software, but I think it’ll be dependent on obviously the
training and the learning of that object — a lot of those imagery is
publicly available — and then blend it with the publicly available
information for those products.
Alan: I know Google and Amazon
and pretty much everybody’s working on computer vision for products.
I want to point my phone at a pair of shoes and say, “What are
those shoes?” And right now we’ve got a device in our hands that
is pretty powerful and can do a lot of things right now. What are
some of the things that we can do right now with our phones, that
you’re seeing are emerging as killer use cases in this technology?
TJ: That’s a great question. Like I said, I think that going back to sort of the chokehold that current existing networks placed on augmented reality, there’s a couple areas where I see a big amount of potential for a mobile device. And I think a lot of that fits around potentially markets where you have growing economies. You look at APAC and other areas around the world, where they don’t have access to certain types of [inaudible] medical facilities. So one of the things that I saw that was really interesting, is how you could use a mobile phone and computer vision help diagnose patients, by using computer vision and artificial intelligence to look for signs that you wouldn’t necessarily be trained or have access to. One of the interesting use cases I saw was to help support a potential phlebotomist out in the field, where they’re using a phone to detect veins, so they don’t mispuncture a vein in the arm. They could do it right the first time, and completely limit the opportunity for infection.
Alan: I think that’s the
AccuVein system, isn’t it?
TJ: Mm-hm, mm-hm.
Alan: Yeah, it’s really great.
TJ: And I think that’s
transformative on a world basis, is that anytime you use technology
to add intelligence to give access to underserved or underprivileged
or markets that just don’t have the ability to. So we look at that
space, too. We look at– we talked about accessibility, I think, a
little bit. We look very much into that space also, as a way to just
improve in general. And I think we share a very similar feeling. It’s
about improving quality of life. It’s not about introducing one more
thing, one more piece of noise into the environment. How can we help
each other sort through daily life, whether it be from a medical
standpoint or being inundated with information? How do we make those
things simple?
Alan: Absolutely. And as more
and more people move into urbanized areas, there’s that culture shift
from living in the country to living in a city and– offline, we were
talking about taking our kids camping and stuff. My 11-year-old
daughter made this huge billboard poster and put it by the fire. It
said “No cell phones by the campfire.” And I think we’re
really getting to the point where the technology is pervasive. It’s
everywhere we are. My kids sit on the couch and watch TV with their
phone in their hands. And sometimes they’ve got an iPad and a phone.
It’s nuts. I don’t even know how they focus. And the other day, my
daughter was watching a show, and she had the show in a small window
and she had a game that was related to the show in the big window. So
she had like picture-in-picture, but the show wasn’t the dominant
part of it. And I thought that was really interesting, how youth are
starting to use these technologies. And we’ve done a lot of work on
delivering people entertainment content. Netflix is using AI
algorithms to give people better movies to watch. Amazon’s giving you
better algorithms to help you purchase better. What I think we need
to do is harness those technologies and give kids better ways to
learn. And I think these technologies can really catapult that. What
are your thoughts around VR, AR, AI in education and training?
TJ: Yeah, we we touched on this
a little bit. And I think the impact is absolutely [inaudible] that
space, specifically with convergence of people either domestically or
internationally from an education standpoint. I think one of the
things that you might have seen, our team produces a virtual reality
platform called Operation Convergent Response. And the idea behind
that was to aggregate or bring together a number of people with
different backgrounds and skillsets into a single virtual environment
— or a war room, essentially — so that they can help support a
natural disaster. So if there was an earthquake, bring an earthquake
expert, someone who’s an expert in fire, someone who’s an expert in
weather — or whatever might be the case — to help quickly triage
sort of a situation, without bringing them into a physical location.
That’s immense also. You can have highly custom education armaments,
where you’re bringing in specialists in different areas, that hold
all under sort of one umbrella. Let’s take VR/AR. Someone interested
in VR/AR, you could bring in an expert in virtual reality, expert in
augmented reality, expert in computer vision, and then you can
coalesce and bring all together for the students. Very specific
interests in that space, into that area, into that arena. And with
virtual reality, you can create a 100,000 square foot space in a
1,000 square space. You put 75 TV’s on the wall, you put one TV on
the wall–
Alan: Yeah, it’s great.
TJ: –you can literally create
the most dynamic environment that works for those students. And
touching on artificial intelligence for a sec, one of the most
amazing things I think that comes out of virtual reality training —
and also in the medical space and other areas — is the ability for
analytics platforms to look at every single piece of interaction
that’s going on in that space. What does that yield at the end of the
day, it yields a very efficient and effective way to help you
understand where you’re improving, where you’re falling behind. And
it’s amazing because in essence, in a typical education environment,
you’ve got to rely on a teacher — or a boss, as might be the case —
to provide that feedback, and you can’t do that for 30 students in a
classroom. But if you’ve got an analytics platform that’s in each one
of those individual students on a case-by-case basis, you can then
custom produce areas in a report where they should be improving and
moving forward. And the amount of advancement we make is just because
of that little improvement. I think it’s absolutely massive.
Alan: You nailed it. By changing
the way we teach and using exponential technologies. It’s not even a
10x improvement. It’s a 100x improvement, a 1,000x improvement
because we’re not even scratching the surface of things like how
people learn or when they learn. I learn better, maybe from 10 AM to
11 AM. That’s my maximum capacity, so all the hard stuff maybe I
learned then, and maybe somebody else learns in the evening. Maybe by
using galvanic response or measuring your heart rate or your
biometrics, you can then deliver content that is highly personalized,
highly contextualized and delivers it at the time of maximum
absorption or retention. And we’re already seeing across the board 20
to 100 percent improvements in retention rates in training. One of
the things that you guys did was use VR for hostage and robbery
training. How did that come about?
TJ: It was an interesting one.
It was actually during our early commercial stages, we were looking
at different verticals in areas where we can use technology to
obviously improve awareness and other things that happen in our
retail environments. It wasn’t specifically around hostages, more
like store robberies, which don’t happen often. But obviously, when
they do, how do you train someone in a traditional environment to
address a burglary situation, especially with someone being held at
gunpoint? So we use 360 video. We use actors and we pull together a
very immersive experience around both an armed robbery and also
identifying risks for theft by immersing them in a 360 world. We did
monitor different bio signs. We also use an analytics platform that
allows us to do gaze tracking and other things. So you saw exactly
where they were looking. What they were interested in. Were they
looking in the right spots? Were they looking in the wrong spots? And
we were able to produce a relatively detailed report based on that,
and what that allowed us to do is it allowed us to understand, as
much as you could in a replicated environment, what is the normal
response for a retail rep when these situations occur? And then it
allowed us to cater a training program around that, that helped
better prepare them for a situation. So we use it as a way to gather
analytics, which then yielded a ton of information for us that we
could never capture in a — even if you did this in a real-life
scenario and you brought in some actors and you brought in a rep and
you them in– you wouldn’t be able to understand where they were
looking, how they were acting, how they were responding, without this
type of analytics in this virtual environment. So because of that, we
felt like we were able to compose a training module that was much
more advanced than what was in the market today.
Alan: How do you measure the
success of that? What are the KPIs that you guys were looking for?
TJ: That’s a great question. And
unfortunately, we provided a lot of the technical support and
background for that while our learning development team managed the
KPIs. But I think some of the key KPIs that we used was sort of this
post mortem experience where we then place them back into into that
experience with their training to see where there are improvements.
And specifically, we use the gaze data, the biometric data, to then
see how they respond in that situation. And I don’t know what the
specific performance improvements were, but I know they were
relatively more significant. That’s sort of what’s baseline for
typical training. Unfortunately, I don’t have that data, but the
process, the methodology, was to re-immerse them back into that
situation with the tools that they now have. Again, because it’s
case-by-case and see where they were looking, how they are acting,
how they’re responding, so on and so forth.
Alan: I want to shift away from
training, because there’s so much to unpack here in the long term of
this technology. 5G is really going to enable a lot. And one of the
things that we have to overcome is some of the challenges. What are
some of the challenges that you see — or Verizon as a company sees
— as standing in the way of the broad adoption of 5G-powered XR,
beyond cheaper headsets and 5G coverage? But what do you see as the
broader challenges around adoption?
TJ: I think adoption is
dependent on establishing a community that understands the
technology, in order to build on that. And that is the other half of
our 5G labs. I own the development side. We also have a 5G lab–
multiple 5G labs, spread across the country which provide developers
and enterprises access to this technology, so they can understand it.
So education is hugely critical. We are very immersed in the
technical side of things. So we’ve won those Edge awards, very
technical. And the work we do, while deeply technical, has to be
translated to a common level. And I think our biggest hurdle and
challenge is making that consumable to the end user. And we are
solving that through our 5G labs. We are providing developers with
training and access to 5G networks, providing enterprises with a full
view into the capability. So as much as we’re having this
conversation here, any CTO or CIO or even CEO listening, visit our 5G
labs. You’ll get a full view of what we’re doing and how it can apply
to your business. There are so many companies across so many
verticals that it really helps them understand and inspire what they
what they could do with the technology.
Alan: It’s interesting that you
guys created these 5G labs with the purpose of showing what the
technology can do. One of the reasons we actually started XR Ignite
was the same thing. We kept seeing all these amazing startups coming
up with crazy, amazing ideas on how to use XR and AI, but the they
were missing the business acumen to be able to take those
technologies and bring them to a commercialized state. We formed XR
Ignite to kind of help bridge that gap between corporations and
startups. And it sounds like you guys are doing the same. Sounds like
there’s an interesting fit there. We’ll talk offline and see if we
can collaborate.
TJ: Yeah, no, absolutely. I
think there’s there’s great opportunity for anyone looking and
interested in it.
Alan: So what is the most
important thing that businesses can start to do now to leverage the
power of XR, and AI, and 5G? What can they start doing immediately to
reap the benefits?
TJ: Yeah, I think any company
should — just like we do here — is to assign someone from the
strategy side of the business to either bring someone in or to do
their own objective (and to some extent, subjective) view of
collection of information. There’s so much data out there. There’s so
much information out there. You know, the stuff that I’m talking
about is very topical in the sense that this is stuff that you and I
have access to. And we’re very immersed in this space. Obviously, we
know relatively well, but anyone in any space could find the same
amount of information. I work at a very deep technical level, which
obviously makes not much sense to businesses until we translate that
into a business function at the end of the day. But one or two people
in an organization can compile some really compelling amount of data
and information that exists out there, to really help transform
businesses with this technology. So I think it’s just that first step
that you need to take to say AR, VR; what is this about? Right? What
does this mean to my business? And I think they’ll be surprised.
Alan: We actually started
MetaVRse as a consulting firm to help businesses understand how to
use this technology, because our long-term vision was always
education. And so we’ve kind of morphed into consulting on how to use
this technology specifically for education, training, learning
modules and stuff like that. But with XR Ignite, it’s pretty broad.
There’s so many companies out there building such great tech and
everybody is chomping at the bit to do start leveraging 5G. And I had
a meeting a couple weeks ago and I got to see the new Samsung s10 5G
phone, and it was the first time I’d ever held a 5G phone in my hand
and it kind of felt like this new renaissance of technology is
coming, and it’s going to come as a tidal wave, because it’s been
years that people have been working on it. I don’t know, Verizon’s
probably working on this for a decade. So it’s coming and it’s going
to come really fast for people. What can consumers do to prepare for
that? Because I think by having a 5G phone is great. It’s faster,
whatever. But really, what can consumers expect from that?
TJ: If you look at where we were
with 4G and sort of this scale and ubiquity of 4G were in that same
phase. — and not just in this scale, but also in sort of the
advancement of technology — I think one of the things… so just
sort of prefacing this, one of the things that is interesting is that
the iPhone didn’t release a 4G iPhone until about a year and a half
later. And I think once the network launched, I think part of that is
because there are ramp ups to this technology as it gets exposed and
becomes more accessible to developers, they could start building
experiences on top of that, that really validates the value
proposition of 5G. As it stands right now, the biggest value
proposition of 5G is just the ability to download an immense amount
of content in a quick amount of time. Right? So I want to download
entire season The Sopranos through HBO Go, or whatever might be the
case, before I’m about to hop on a plane or I’m going to make a split
second decision. You know, I don’t have to wait 10, 15, 20 minutes to
do that. I can literally do that within 30 seconds or less.
Alan: That’s insane.
TJ: It really makes on demand
“on demand,” right? It used to be that you had to plan some
stuff, or– one of the great things, I have got music service and
unfortunately where I am down at the shore on the beach, I don’t have
access to — because I’m pretty remote — to that network. And so I
want to be able to download an entire alternative 90s collection of
music of 10,000 songs in…a minute? Less than that?
Alan: OK. I got to just… what
the hell, man? I used to be a DJ for 20 years and I remember going to
the record store and you’d buy a record. Then I had a collection of
books of CDs, and I remember when my daughter was about 10 and I gave
her my CD collection, and she looked at me, she’s like, “what do
I need that for? It all fits in my iPod.” And now we’re talking
about downloading entire genre of music in seconds.
TJ: It’s quite crazy. Again,
that’s just scratching the surface of 5G. Beyond that, there are a
lot of other things that we foresee with the implementation of the
Edge network that we talked about and providing access to developers.
One of the things that I did wanted to touch on is sort of our last
piece a minute ago was that developers, the XR space is still nascent
enough that if you’re a business, their may [be]– or likely isn’t —
a solution for your business today. And so I’d encourage businesses
to go out there, understand the technology and build their own
solutions, because half of our practice, going back to our commercial
period, was a pro services organization and we were just all ears. We
would listen to what companies would come to us with challenges and
then we would solve those problems for those businesses. And one of
the benefits of being Verizon, a massive company that we are, is that
we’ve got so much diversity. We’ve got retail. We have network
operations, sales. Right? We can hear these challenges from all these
organizations and build solutions that no developer would ever really
think of to build. So I’d encourage businesses to go out there and
not just look for answers to their problems, but work with folks that
can solve those problems.
Alan: That was our entire
business model, was to create some solutions, sit and listen to
customers, figure what they wanted and build it for them and then
develop those into products. Literally. That was our our entire
business model, was just, listen! And that would create products,
which it has.
TJ: This is an SAP, right? I
know I need some sort of platform. I go to SAP and they’re like, I
get inside a box software and just customize it. Then that’s answer
for my business. That’s not the state of XR. And if you can jump on
that as a business, I think you’ll have a major upper hand like we do
here. So I’ll give you one example. We have a team that’s responsible
for going out and working with community. You have them understand
where we’re going to be putting 5G cell sites. And one of the things
they do is they hire a graphic artist, a number of graphic artists,
to go out and draw what a cell site would look like in a community
and they would do it in Photoshop and other tools right there. And
then they would bring it to the community board meetings. San
Francisco would be the case and say, hey, these are the sites that we
want to produce. Right? And this is what the tower is going to look
like on this building or on this light pole, whatever might be the
case. And it costs them on average just in this one section of the
US, $5-million a year contract, graphic artists. So what we did, and
what they asked us to do, is how do we solve this problem with
augmented reality? Can we just take the poles that they would draw,
import them as 3D models and then just drop them into the
environment, capture that photo with our camera and submit that to
them to the town planning meeting? We said, absolutely. The Verizon
rep goes out there, they go to a corner, they pick the pole that they
want. They change the color of the pole to match the environment.
Right? And then they set it right on the plane of the ground. And
then they capture a picture and they upload that picture for the
board planning meeting. And literally, that application probably took
a month to develop and saved in saving our business $5-million
dollars a year just in this one area. That was not something that
we’d ever imagined building. But we took that cost out of the
business.
Alan: Amazing. Some of the cost
savings that I’ve heard on different interviews on this podcast have
been insane.
TJ: It’s an insane amount of
money, for relatively simple implementation of the technology.
Alan: So with that, last
question; what problem in the world do you want to see solved using
XR technologies?
TJ: I want to see people’s lives
become simpler. Everything that we build, from a technology
standpoint, very complex in nature. I think the fundamental idea is
to make people’s lives simpler and to make people’s lives safer. I
think we’ve come to a point at technology just earns us so much that
you companies out there that are now creating artificial digital
clones of ourselves to manage the information overload that we have
coming in, so we provide them with a set of rules and
responsibilities that they could take on our behalf. And I don’t
think that’s necessary, if you would, in a way that filters out the
noise and makes everything [inaudible]. And I think what that is, is
that that’s blending the digital and physical by… I don’t want to
be constantly distracted by my cell phone when I’m camping. Right?
Because there’s something interesting that I just saw, maybe out in
the wild. Was that a deer? Was that whatever might be the case? With
glasses, I think that information is so passive to you that we no
longer have to be distracted by our device and we can tune in and
tune out things as needed. And my view at the end of the day is, we
strive for simplicity in our lives. And I think that’s what we want
to go with this technology.
Alan: Oh, I love that. Striving
for simplicity with our lives, using the most advanced technologies
in the world. I don’t know what better way to wrap that up, but thank
you so much. Thank you, everyone, for listening. If you want to learn
more about the work that T.J. and his team are doing at Verizon, you
can visit verizon.com, envrmnt.com, or verizon5glabs.com. I think
those are the three places where everybody can get as much
information as they want. Thank you again, TJ. This has been amazing.
TJ: Thanks, Alan. I look forward
to talking to you soon.
41 min
February 08, 2020 The Future of Retail is Virtual, with Macy’s Mohamed Rajani
Macy’s has been in the news a lot this week, and many are worried about what the latest round of store closures mean for the long-running retailer, and the future of in-person shopping in general. But Macy’s resident XR guru Mohamed Rajani came by our podcast a little while back to suggest that the future of retail exists in the virtual world.
Alan: Welcome to the XR for Business Podcast with your host, Alan Smithson. Today’s guest is Mohamed Rajani, responsible for VR and AR initiatives at Macy’s. Mohamed is part of the new Business Development and Innovation team at Macy’s, and is responsible for driving change through the development of new retail concepts and partnerships amidst an evolving retail landscape. “Mo” also leads Macy’s immersive technology initiatives, including VR and AR in furniture, which is removing key friction points for the customer, enabling an AR view in-room capabilities on the Macy’s mobile app. To learn more about the work he’s doing, you can visit macys.com.
Mohamed, welcome to the show.
Mohamed: Thank you. Thanks for
having me, Alan. Happy to be here.
Alan: We had the opportunity to
to have a few calls prior to Augmented World Expo. We were on a panel
together, and you were talking about the amazing work that you’re
doing at Macy’s. So let’s start unpacking that. Mo, tell us about
what you guys are doing at Macy’s.
Mohamed: So just a little bit of
context that our team does. Our team’s about two and half years old.
I’ve been with the company for over eight years, across a variety of
different functions. But about two and a half years ago, as a
company, we decided to establish a dedicated team that’s purely
focused on what’s new, what’s next. That’s focused on the emerging
consumer landscape, the emerging technology landscape, and making
sure that the Macy’s brand continues to be relevant not only today,
but 10, 20, 30 years from now. So as a team, we’re purely focused on
looking at new business models, new concepts, emerging technologies,
but then really tying those to our strategic businesses. We want to
make sure that any new innovation that we bring into the organization
has a lasting impact. But more importantly, a meaningful impact that
is actually moving the needle.
So if we think about from that context
of how we ended up playing in virtual reality and augmented reality,
in our business we have a strategic business fillers, and furniture
is one of them. It is a business that is high touch, a high margin
business, so it’s margin accretive, more profitable to the company,
and it’s destination business for us. We’re top of mind for the
customer when they’re thinking about furniture. And if you’ve had any
experience in buying furniture, it is not a very easy process. It’s
one of the few businesses that’s still overwhelmingly physical
purchases. More business happens in-store than online, and by a
higher margin. And part of it is just the friction involved in it.
You don’t know how it’s going to look, how it’s going to fit. And
it’s a business that we, as a company, need to fortify. It’s a
business that if we want to remain relevant for the next five, 20
years, we want to make sure that we’re not only fortifying the
business, but growing and capturing market share. So is that context,
whereas we came across emerging technologies as part of our job, we
were navigating the landscape and looking at what’s coming out.
This was 2016, into 2017. We started
seeing virtual reality technologies, especially in the furniture
space, and we started exploring and we wanted to make sure that there
was a practical application to a business there. A couple of things
that align. One was here is a technology that had a practical
application and here is a business that was a strategic priority for
the company. So our job was then to go out and figure out how to
integrate the two. And so we did that first by launching a pilot in
two stores in August of 2017 with a company called Marxent. And
really the goal was to understand how this technology can apply to
our business, but with two ultimate objectives. One was to drive
employee adoption so our Macy’s colleagues and making sure they’re
adopting the technology, and two is consumer adoption on the other
end of it, making sure that our consumers are adopting an innovative
but emerging technology. And we felt if we tackle these two goals,
everything else will come together.
Alan: You’re talking about
employee engagement or adoption. That is one of the most important
things when you bring a new technology into a company. If people
don’t use it, then it’s kind of useless. And I’ve heard some stories
where a company will buy VR headsets and they just sit on the shelf.
So how did you get your employees to use this? What were some of the
techniques that you did, and what is the adoption rate now?
Mohamed: The ultimate goal–
you’re right. In the past, we may have been guilty — just like a lot
of other retailers — in bringing the technology either without a
clear practical application or the execution is lacking for some of
these basic parameters. So for us, it was critical that we had a
comprehensive training program. We worked with our Marxent partners
to really build a robust end-to-end training program that clearly
outlined and communicated the benefits of the technology, the
practical application of it, and what it means for our colleagues. If
you’re in the furniture business, your ultimate goal is to drive
sales. That’s how you’re evaluated. That’s how you’re commissioned.
So it’s really to drive sales. And the idea for us was that if we
were able to jump the hurdle and get our colleagues to adapt to this
new technology, we knew that the upside was higher basket sizes and
lower returns.
So the ultimate goal for us is to
communicate with the colleagues, to say “Here is a technology
that can not only help you drive higher meaningful paychecks or
higher meaningful basket sizes, but also reduce returns,” which
if you’re in the furniture business, you know the percentage of
returns is not very high, but the cost of maintaining and managing
those returns are as significant, as very burdensome as well. It was
clearly outlining the benefits of the technology and the fact that we
piloted in two stores and saw the results for ourselves. That allowed
us to then take it to the balance of the organization and tell them,
“Here are the benefits. We’ve done this before. This is what it
looked like.” And that allowed us to drive really employee
adoption. And that’s critical, because the consumer adoption cannot
come if our colleagues don’t adopt the technology. So if we’re able
to cross one hurdle, the other one becomes a lot easier, too.
Alan: Let’s talk numbers. You’ve
seen some incredible uptake with this technology. You started with
two stores. Talk to us about what you saw or what your original
findings were in those two stores, because they’re almost
unbelievable.
Mohamed: The first two stores
eventually allowed us to make a decision that typically retailers of
our size did not make: to scale something very, very quickly. So in
the first two stores we launched in August of 2017 in Paramus, New
Jersey, and in Miami, Dadeland. And we added our third location —
our flagship location — Herald Square, a few months after that. But
what we saw was staggering: in the initial pilot we saw our basket
sizes increase by 60 percent. That’s 60.
Alan: Hold on. What?
Mohamed: Yeah.
Alan: Sorry. The basket size.
Now, what does basket size mean?
Mohamed: The dollars for a
transaction. So that includes number of items in the basket, and the
average unit retail of each product. So the combination of that
obviously is the basket size. And we saw the average dollar per
transaction increased 60 percent for every sale that went through VR.
Now, because–.
Alan: I don’t believe you.
Mohamed: [chuckles] Well, that’s
exactly what everyone else said, but they had to see it for
themselves. And when you go through the experience, you realize it is
not anecdotal information. When you go through the experience, you
realize that, yes, I can see how this would increase basket size,
because I’m more confident in my purchase, so I’m less likely to
return. But more importantly, I have now more confidence in how this
is going to look. So you’re seeing more expensive items being
purchased, because you’re more confident in your purchase.
Alan: Can you walk us through
the experience?
Mohamed: Well, the experience
starts way before you put on the headset. You walk into a furniture
gallery, you’re met by our colleagues, our season trained colleagues
who know the furniture business in and out. They’re probably one of
the best in the industry. And when you walk into the stores, you’ll
typically engage with a Macy’s colleague, who’ll walk you through our
assortment, will tell you how it’s merchandised, and really ask
questions around what you’re looking for. And typically, as someone
who’s coming in to browse or exploring furniture, you have a sense of
idea of what you’re looking for, and you’re tying that to now with
either you’re trying to add extra pieces to your living space or
you’re totally furnishing a new space. You have a sense of what
you’re looking for, but you really need someone to guide you through
what the best options are. Around that same process, and they’re
typically used to identifying that hesitation around fit, “I’m
not sure how this is going to look in my space. I don’t know exactly
how big my living space is, or what the relation to the furniture
space is.” And that’s where the colleagues now try to solve that
problem for our customers. And they’ll bring them to our VR design
experience in the store, and through a 3D design application on an
iPad, they’ll be able to design. You will provide your general
dimensions of your living space. They’ll put input it into the iPad
application, and design it with Macy’s 3D content. So we’ve
rendered– when we started, we had rendered about a small section of
our furniture assortment in 3D. Our library now is ten times bigger
than what it was when we first started two years ago. You’re able to
design in the 3D application, and then you put on the headset and
that’s where the magic happens. With the right dimensions, the exact
sizing, you’re able to move things around, you’re able to add new
things, edit things if they don’t look right. You’re able to move in
closer, look at the texture of the furniture, and really brings the
experience to life.
Alan: Can you add windows when
you’re designing?
Mohamed: You can at windows,
telephones, fireplaces. You can add plants to the experience. You can
really mimic your living space.
Alan: Can you change the wall
colors and stuff?
Mohamed: You can change the wall
colors, floors. You can add wooden floors. You can change different
kinds of colors. Everything they can imagine in a design application,
you can do it here. And the best part is you can visualize it and do
it on the fly. And really, when you go through the experience, that’s
when you can really start understanding the numbers and saying,
“Okay, these numbers are real, because this experience is so
much better. And I have a much better idea of how this is going to
look, how this is going to fit. So I’m more confident in my
purchases.”
Alan: One of the metrics that I
found almost more impressive than your basket size increase of 60
percent was the return rates. What is a typical return rate before
VR?
Mohamed: So typically furniture
is in a mid-single digit returns. Anywhere between 5 to 10 percent in
returns. Just generally in the industry average. And what we were
seeing through the pilot — and that’s held through as we scaled, as
well — is about a 25 percent reduction in returns. When you take
that number and put it at the scale that we have, that’s a monumental
impact on the returns line for the company and eventually to the
bottom line.
Alan: So 60 percent increase in
sales and 25 percent reduction in returns. These are not immaterial.
Mohamed: They’re not. They’re
three stores, but not immaterial.
Alan: Yeah, and that’s what I
was going to say, is OK, so now you’ve presented this in two stores.
You got some data. You go to your executives and say, “Hey,
guys, we have something here.” What was the response met like,
and how did it go from two stores to 100 stores? Was this like a long
term rollout? How did this work?
Mohamed: If you asked me this
maybe three years ago, maybe that would be the trajectory. We would
do it in three stores and say “This works, let’s take it to five
stores. Let’s take it to 10 more stores.” What we met with the
reaction from our executives– and they’d seen this, so our CEO was
on the ground in stores. Our leadership had been in stores, and
they’d seen this in real time and seen how it was working. They’d
seen the feedback — from our colleagues in-store, from the
leadership in-store — that this was a game changer. It was a
differentiator against our competitors. We were the only retailer
that had this in VR form inside our stores. And it could be a
differentiating factor for us and a competitive advantage in the
business. So the feedback from the leadership was “How can we go
faster, and how can we go bigger?”
And then it was up to my team to start
building the plan to roll out. And we went back a week or two later
with our plan to roll it out to 50 stores initially, actually. And
that was tying to our strategic priority at the time of our Growth 50
initiative, which was fortifying our 50 top stores across the country
with new investments. So we went with the plan to take 50 more stores
and add VR to it. The response we got was “No, we meant bigger.
How do we go bigger?” And that was 100 stores, which at the time
was 100 that ended up being 110 stores. So within a space of actually
six months, we added 100 new stores, so by the end of January of 2019
we were in one 110 stores.
Alan: Wow. OK. So there’s one
other thing that I think– it would be an incredible addition to any
retail location to get an increase of 60 percent and a decrease of 25
percent returns. But let’s talk about the cost that Macy’s spends to
bring a furniture store into a typical Macy’s. The cost savings alone
of bringing this in versus a traditional furniture store.
Mohamed: That’s the beauty of
the technology, right? There’s multiple formats, models that we can
really try out. And so the format that we’ve gone with today is to
augment the furniture buying experience in our furniture galleries,
in our furniture stores. And we’ve got about 250+ of those stores and
we’re in about half of them. And the hope is to expand that to all of
them. But what the technology now allows us to figure out is there
are markets where there is demand for furniture. There is demand for
Macy’s to play in. But it’s cost-prohibitive. The ROI is in there, in
context of what it cost to build a furniture store there. And that’s
where we think this technology could play a critical role in us
taking furniture across the nation. We’re in markets where we can not
justify a fully fledged investment in building a furniture gallery.
Can VR play that virtual furniture gallery with an endless island?
The key there is that if we can expand
our 3D content — and we’re on the path to doing that — it’s still a
cost-prohibitive process in driving 3D content. But we’ve come a long
way to where we were two and half years ago, where the cost continues
to go downward. But the idea is that if we’ve been able to build a 3D
library that allows us to now take furniture to stores where we don’t
have to invest in the inventory in that space, we don’t have to make
a 50,000 square foot or even a 10,000 square foot furniture store.
Alan: What is the footprint of
the VR area?
Mohamed: The VR space today is–
the largest we have is about 500 square feet, but that’s in our
flagship stores. You can do it in up to 100, 150 square feet. So
that’s– a typical furniture gallery for us is 50,000 square feet, a
furniture store inside of Macy’s is about 10 to 20 thousand square
feet.
Alan: So 60 percent increase in
sales. 25 percent reduction in returns. And a hundred times decrease
in the space required, which in retail is dollars.
Mohamed: Exactly right. And so
the goal becomes, remember, the 60 percent increase is on any
transactions that’s going through VR. Some of that is a function, a
result of how much your 3D content library is, or how big your
library is. Today that’s a small percentage of our library. But the
goal is to continue to expand that. And as I said, we’re increasing
in multiples. We’re about 10x where we were two years ago. And the
idea is to continue to expand that library, because we know the
upside is higher basket sizes, lower returns, and smaller footprints.
Alan: Let me ask you a personal
question. If you were any other company in the world, why are you not
doing this?
Mohamed: That’s a good question.
And if I wasn’t in the role that I am, I probably wouldn’t know that
either. I think at the heart of it is, you need a champion, you need
an internal champion. You need someone who is– one exposed to the
technology. But you also need someone at the leadership level to say,
“OK, I’ve seen enough in this. This is big. This is a
differentiating factor.” And the companies that get it, by the
way, are doing this. The beauty about the Macy’s business is that we
have– we’re not a pure play retailer. We’ve got a massive store
footprint, and we’ve got some of the best real estate across the
country. And then you’ve got a massive digital footprint. So we’re
able to leverage this technology across both channels, online and
in-store. I think when you talk about investments and when you talk
about ROI, sometimes that’s where the hurdle becomes, where you don’t
have enough use cases to leverage the technology.

Where the beauty for us is, our
singular investment in 3D content goes across our mobile app, goes to
our digital properties, goes to our store. So we’re able to spread
the investment as well and move faster. I think if businesses really
hunkered down and looked at the practical application of this
technology, it’s not right for everyone. It’s not even for us. Not
all categories have the same practical application. So we’re still
thinking about today, how do we take this technology and where else
can we use it? Furniture is an important part of a business, our
business, but it’s a smaller part of our business.
Alan: Clothing, for example, it
would be great to see your own avatar with clothing on, but the
technology just isn’t really there yet.
Mohamed: It’s not there yet.
Exactly. And that would be the holy grail for us, right? That’s the
bread and butter of our business. Apparel.
Alan: If we figure this out,
will you sign a contract with us? No, I’m kidding.
Mohamed: I mean, you’ve piqued
my interest. Trust me, I spent a lot of time looking at what other
technology companies are doing in the non-furniture categories. In
fact, we’ve made some investments this year in starting to build our
3D content library in non-furniture, as well. So we’re doing it
across apparel, across accessories, footwear. In a small scale, but
really trying to understand where also can we leverage this
technology? Having spent the last two and half, three years in this
space, I truly believe the consumer of the future is going to
interact in 3D. We’re one of the earlier movers, and we’re trying to
set the foundation. But if businesses are not making those
investments today, they’re going to be left behind five years from
now.
Alan: Yeah, absolutely.
Mohamed: Macy’s as a brand as
you, in spite of everything that you hear from the reporting in the
media that department stores are dead, the reality is that Macy’s has
always been one of the pioneers. We launched our e-commerce business
in 1998, way before any of the other retailers had even started. And
that’s a core reason why today we’re one of the largest Internet
retailers in the country. And so it’s always been a leadership that
has been excited about change, and been investing in it, and having
the courage to invest in change. And that’s what we’re seeing here as
well. By leadership that we’ve got, their support to really explore
how do we evolve our e-commerce business, evolve our stores business,
and invest in emerging technologies that really have a practical
application to our strategic priorities.
Alan: It’s true. And being able
to take the knowledge and experience that you guys have gained. And
then thank you, first of all, thank you for joining this
conversation, because I think people listening, especially those of
you listening in retail, this punctuates the value that virtual and
augmented reality brings. Talk about your AR component, because I
know you’ve built these furniture visualizers in VR, so you can go
into a store. But I think equally as important is something like what
you guys are doing in AR, being able to see these same 3D models, you
don’t have to create anything new, just the same 3D model. Maybe you
decimated a bit or make it smaller for the AR to use on a mobile
phone, but you can now see that couch, or that hutch, or whatever in
your own house.
Mohamed: Absolutely. And that’s
the beauty of this. I was talking about diversifying the ROI. And
that’s where an investment in 3D content allows you multiple use
cases. Today we’re absolutely scaled in VR in-store. But since last
year, we initially started as a test in piloting the augmented
reality component in our furniture business on our mobile app. Now
mobile is a core part of our strategic priority. It’s our flagship
location. Our best customers are omni-channel customers. They shop on
our mobile app. They shop on our mobile properties. They shop on our
website. They shop in our stores. And really, the goal for us is to
continue to drive the mobile app experience.
We’ve got one of the highest-rated
retail apps in the app store. Last year, we grew our mobile business
by about 50 percent. We did nearly a billion dollars through our
mobile app last year across Macy’s and Bloomingdale’s. So you can see
it’s a top priority for the company. And the goal is to continue
growing that. The way to grow that is when you pack value inside the
mobile application, you drive different experiences. And that’s where
we’ve made sure that we’re able to extend the AR experience into
mobile apps. So we tested it last year. We launched it this year
across Android and iOS. So all our app users are able to do that. And
really the experience is if you’re on the furniture product page,
you’re able to click a button that says “View in my room”
and you’re able to– it will use your camera to superimpose the 3D
content in your actual living space. And the beauty is you’re able to
change colors and try different things. What I’m excited about is
with where that technology heads today, so single scene experience.
You can put one product and view that in your room. We’re not far
away from doing a multi-scene environ where you’ll be able to bring
in multiple products and look at how that. And I think that’s going
to change.
I think if you think about the beauty
of VR, where I like it is, it’s a location-based experience. I have
600 locations across the country in about 250 to 300 furniture
locations. So I’m able to drive consumers to our store through VR.
Really, the scale is in AR when the consumer through their
smartphones will be able to interact with the technology. So that
gets us excited. We’re seeing some really good early results, that
I’m not able to share yet, we haven’t made them public yet. But
they’re really meaningful results in terms of driving conversion,
when a consumer interacts with AR in our furniture business. I’m
excited about where that’s going to go.
Alan: There’s so many
opportunities. Now, something that comes up on almost every one of
these podcasts is training. Are you guys using this technology in
that capacity, as well?
Mohamed: So– we’re not. Not
yet, I would say. I think it’s one of the core opportunities for us,
especially as a retailer with a lot of stores and a lot of retail.
We’re 130,000 strong organizations, so clearly there’s a lot of
requirements for training. And it’s something that we’ve started
exploring. We’ve had some conversations, and it’s something that
we’re deeply looking into. I think there’s– if I look at that three
core uses for immersive technology in general, clearly the need for
practical application to specific business is one of them. Training
is the second one, where I really think it can add value and
meaningfully drive costs, but also increase engagement, and as a
result, retention of the training material. And then the third piece
is obviously as a branding and entertainment avenue. And as you know,
Macy’s is a powerhouse in that sense. We run some of the most sought
after events, from the July 4th fireworks, to the big annual
Thanksgiving Day parade, to the flower show. So there’s a lot of
opportunity for us to leverage immersive technology in these three
streams.
Alan: I have one for you. So on
the Fourth of July, I posted a couple of AR apps that you could do AR
fireworks, and all of them were meh. So let’s make the Macy’s Fourth
of July fireworks AR app.
Mohamed: I mean, all year. I
think the idea is you want to make sure the experience is authentic.
You want to make sure that one, the experience is authentic and you
really get immersed into it. So I’m not sure if the fireworks in AR
is the right– it’s something that we want to explore. We want to
look into it. But we want to make sure that the experience that
someone who’s live in that space is experiencing, you’re able to
replicate that in an immersive technology setting. So we’re going to
look at that space closely. We’ve spent some time with our leadership
in that space to talk about it. And I’m excited about what could come
down the line.
Alan: There’s so many great
things that you guys are doing. And I think one thing that’s really
interesting about the work that you’re doing at Macy’s in general, is
that you got bit by the VR bug. You found an early use case that
showed real ROI. And now you realize, “Wow, if it shows this
kind of ROI in one section of our company, what else can it do?”
Mohamed: And that’s the key,
finding a practical application. Because that’s the catalyst. If
you’re able to find a practical application — when I mentioned
earlier about core goals are on driving employee and consumer
adoption — because there is a practical application, our job got a
little bit easier in driving the employee adoption and the consumer
adoption. But then — as you just alluded to — it’s exactly that.
Like as soon as– if it worked here, where else can it work? Where
else can I take it? And that’s been the mentality on our team. That’s
been the mentality with our leadership to say, “OK, this was
great. Now figure out how do we spread it to other parts of the
company.”
Alan: Absolutely. What advice
would you give a new company that’s new into VR? What would the first
steps be for somebody that is looking to invest in this technology?
Mohamed: Technology for
technology’s sake is not going to work. So there is always that shiny
object that’s out there. But I think really focus on what you were
trying to solve. Identify a problem that you’re trying to solve, and
figure out a practical application. That’s exactly the approach that
we’ve used. We had a business that was a strategically important
business to us. It was full of friction in the consumer buying
process. And here was a technology that could alleviate that problem.
If you’re in that situation, then constantly focus on the end user
experience — whether that’s the employee who’s going to use that
technology or drive the adoption of the technology, or the customer
— make sure the end-to-end process is fully embedded out and fully
thought through. From training to execution, all of those need to be
really focused on how the end user is going to use the technology.
And if you’re able to do that– and not all of those, even if you
follow that to the tee, things may not work out, but you will learn a
whole lot about what are the aspects that worked and what didn’t, and
that allows you to pivot elsewhere. But really, the core is to find a
problem you’re trying to solve, versus something that’s good to have.
So get the basics right, then try to identify real problems that
either your employees are having or your customers are experiencing,
and then go about finding a technology that’s truly solving that
problem.
Alan: Mo, this has been really,
really intriguing, and I think it’s really amazing that you’re
sharing this information. And I think I want to applaud you and the
Macy’s executive team for sharing this, because without these early
case studies and these early wins, this technology doesn’t go
anywhere. If you guys had just said, “Yeah, we’re seeing these
great results, don’t tell anybody,” there would be no reason for
other companies to invest in it. And I think what we’re going to see
is a much, much larger pie be created from all of this, rather than
the scarcity mindset. So thank you for being one of those people that
champions this technology.
Mohamed: No, I’m happy to. And I
think there is a benefit for us in ensuring that. One, obviously, it
highlights some of the great work our teams are doing. Obviously I’m
the face of it, but it’s a massive team behind me that’s actually
executing this on the ground every day. But more importantly, even
our investments today are early investments. So we’re typically–
because the market’s not– it’s still not mainstream yet. Our
investments are a lot higher than what it would usually be. So the
hope in return is that as we evangelize the technology and continue
to share, and more players enter the fray as that demand increases,
we’ll be able to see some of the costs come down as well, and as a
result improved ROI for later players as well. But more importantly,
some of our– some of the early movers, such as us. So I’m excited to
share. It’s something that I’m deeply passionate about. I think
there’s a lot of opportunity in this technology and a lot of
retailers — the ones that are truly serious in this space — are
actually making meaningful investments alongside us. And the hope is
that more retailers come on. I think it’ll be good for the industry
in general if more players join the fray.
Alan: It only serves to decrease
the costs for everyone because, for example, three years ago, VR
developer wasn’t really a thing. You know, you had to kind of go
hunting for people that were making video games, and bring them over
to the dark side of enterprise. But as more and more people start to
see this technology as it expands, I think we’re gonna see what we’ve
already seen, a dramatic decrease in the cost to produce this type of
content. 3D objects or 3D assets for retail. Three years ago,
building a shoe in 3D would be a 1,000 bucks to build one shoe. And
now there’s lots of people that will do it around the world. And then
there’s photogrammetry techniques, there’s different ways of doing
it.
Mohamed: If you think about it,
just on our mobile channels we’ve got over a million and half units
or products. So if you think about it, the scale that we require, the
content price to be able to scale 3D content is monumental. And so
we’re seeing it move in the right direction. Our costs are
significantly lower than where we were two years ago, but we’re still
not at the rate where we need to be to scale. But the hope is that as
more players come in, and to your point is there is higher demand for
designers, and we’ll be able to continue to see that moving in the
right direction. I’m excited about where it’s going, but there’s a
lot of work to be done to get to scale.
Alan: Well, it sounds like you
and your team are leading the way, so thank you again. I can ask you
one last question. Mo, what problem in the world would you like to
see solved using XR technologies?
Mohamed: So I will be biased,
because I am still– I’m a retailer at heart. So it’s the consumer
experience. If we think about where the consumer five years from now
is going to engage with retailers, I think XR is the space where that
can bring that experience closer to life. If you see today why
physical retail continues to be strong, you hear about store closures
everywhere, but you’re also hearing about a lot of new store
openings. And part of it is because the physical experience going to
a store, the online experience can’t replicate that. And if you think
about it five, 10, 20 years from now, and if you want the online
business to continue to grow and become an equal share of global
commerce revenue, XR is, I think, an avenue to help solve that
problem, to create a digital experience that mimics as closely to a
physical experience. And until we get there, you’ll continue to see
physical retail thrive. So that’s one I’m really curious. I mean,
obviously I’m at the center of it for some parts of it, but I’m
really, really interested in seeing how that comes to life and how–
not only does it solve consumer pain points, but also enhances and
augments that experience.
Alan: Thank you again for taking
the time to join us on this podcast. And I think on behalf of
everybody listening, this has been a fantastic opportunity to learn
about Macy’s and the technological advancements of a 100 year old
retailer remaining relevant in 2019 and beyond. So thank you.
Mohamed: My pleasure. Thanks for
having me.
35 min
February 08, 2020 The Future of Retail is Virtual, with Macy’s Mohamed Rajani
Macy’s has been in the news a lot this week, and many are worried about what the latest round of store closures mean for the long-running retailer, and the future of in-person shopping in general. But Macy’s resident XR guru Mohamed Rajani came by our podcast a little while back to suggest that the future of retail exists in the virtual world.
Alan: Welcome to the XR for Business Podcast with your host, Alan Smithson. Today's guest is Mohamed Rajani, responsible for VR and AR initiatives at Macy's. Mohamed is part of the new Business Development and Innovation team at Macy's, and is responsible for driving change through the development of new retail concepts and partnerships amidst an evolving retail landscape. "Mo" also leads Macy's immersive technology initiatives, including VR and AR in furniture, which is removing key friction points for the customer, enabling an AR view in-room capabilities on the Macy's mobile app. To learn more about the work he's doing, you can visit macys.com.
Mohamed, welcome to the show.
Mohamed: Thank you. Thanks for
having me, Alan. Happy to be here.
Alan: We had the opportunity to
to have a few calls prior to Augmented World Expo. We were on a panel
together, and you were talking about the amazing work that you're
doing at Macy's. So let's start unpacking that. Mo, tell us about
what you guys are doing at Macy's.
Mohamed: So just a little bit of
context that our team does. Our team's about two and half years old.
I've been with the company for over eight years, across a variety of
different functions. But about two and a half years ago, as a
company, we decided to establish a dedicated team that's purely
focused on what's new, what's next. That's focused on the emerging
consumer landscape, the emerging technology landscape, and making
sure that the Macy's brand continues to be relevant not only today,
but 10, 20, 30 years from now. So as a team, we're purely focused on
looking at new business models, new concepts, emerging technologies,
but then really tying those to our strategic businesses. We want to
make sure that any new innovation that we bring into the organization
has a lasting impact. But more importantly, a meaningful impact that
is actually moving the needle.
So if we think about from that context
of how we ended up playing in virtual reality and augmented reality,
in our business we have a strategic business fillers, and furniture
is one of them. It is a business that is high touch, a high margin
business, so it's margin accretive, more profitable to the company,
and it's destination business for us. We're top of mind for the
customer when they're thinking about furniture. And if you've had any
experience in buying furniture, it is not a very easy process. It's
one of the few businesses that's still overwhelmingly physical
purchases. More business happens in-store than online, and by a
higher margin. And part of it is just the friction involved in it.
You don't know how it's going to look, how it's going to fit. And
it's a business that we, as a company, need to fortify. It's a
business that if we want to remain relevant for the next five, 20
years, we want to make sure that we're not only fortifying the
business, but growing and capturing market share. So is that context,
whereas we came across emerging technologies as part of our job, we
were navigating the landscape and looking at what's coming out.
This was 2016, into 2017. We started
seeing virtual reality technologies, especially in the furniture
space, and we started exploring and we wanted to make sure that there
was a practical application to a business there. A couple of things
that align. One was here is a technology that had a practical
application and here is a business that was a strategic priority for
the company. So our jo
35 min
February 04, 2020 Talking AI and Future of Work in XR — In a Truck — with Timoni West and Cole Crawford
This week’s episode goes all the way back to last year’s Curiosity Camp, when Alan shared a ride with Unity Lab’s Timoni West and Vapor IO CEO Cole Crawford, recording a podcast along the way. The three discuss the challenges that will arise as AI begins to replace human workers.
Alan: In a very special episode
of the XR for Business Podcast, we’re driving in a car with Timoni
West, head of XR… Research?
Timoni: Director of XR in Unity
Labs.
Alan: Director of XR at Unity
Labs, and Cole Crawford, CEO of Vapor IO. So we’re driving on our way
up to Curiosity Camp through these beautiful winding roads, and we
decided that we would record a podcast, because Cole, in his
incredible company building the infrastructure of cloud computing,
they built an AR app to help service that. And I thought, what a cool
way to use this technology and this time on this beautiful drive.
Wow. Look at the size of those trees.
Timoni: They are enormous.
Alan: Oh, my God. Wow. Well,
anyway. Timoni, how are you doing?
Timoni: Excellently. And I’m
also enjoying the view. Yeah. Yeah, actually, Cole, I’m really
interested to hear more about why you chose to go with that, and what
the process was like. My team is working on tools for mixed reality.
So for Unity itself, that’s used to make, I think, 90 percent of all
Hololens applications right now. Century is using Unity for that. But
the tools that we’re making today are allowing, I think, for you to
more easily make robust, distributed applications that can work
across various devices and for various users.
Cole: And that’s very needed.
First off, Alan, I just want to say, you sound like you should be a
podcast DJ.
Timoni: So it’s cool that you
are.
Cole: Well done. But yeah, I
mean, the issue for us when we started down this journey was very
much a question of, how robust can we make an experience, about how
widely could we make that experience? And the vertical integrated
solutions that you had to choose from in the early days of AR/VR, I
think, are primed for disruption. I’m super glad to hear that Unity
is working on the open APIs, etc., needed to bring this technology to
more users, as I’ll quote — maybe a little cliché being where we
are and where we’re going — but–
Timoni: Yeah, I want to hear it.
What is the problem you company solves?
Cole: Yeah. So we have to think
about not four, but 40,000 different data centers; we’re an edge
computing/edge data center infrastructure company. And with that
means you can’t Mechanical Turk what was originally done in data
centers. It works with four buildings. It doesn’t work with 40,000.
So we had to build autonomy into every aspect of our business, in
every aspect of the infrastructure. And that means building really
simple interfaces for what would otherwise be really complex
problems. And at scale, from a logistics supply chain — remote hand,
smart hands, all the things that you do in data centers — what that
means is your FedEx guy, your U.P.S. guy, a contracting company that
otherwise would need specialized training, now it’s visually assisted
capabilities for what would otherwise be a job that you would train
for and then go work in a data center. We simplify that.
Alan: So basically what you’re
saying is that you’ve given real-time tools to anybody to be an
expert on the field, in the field.
Cole: It’s fair to say that the
software is the exper, and what you need are opposable thumbs,.
Alan: Haha! Which democratizes
the whole need for training.
Timoni: You know, it’s funny; I
was just getting drinks with someone from Open AI. He is working on
the robotic hands with opposable thumbs. I wonder whether or not
that’s really necessary. It is a tremendous challenge. Okay, so from
what I got from our earlier conversation, someone goes to a data
warehouse, they’re looking at a… “RU,” you were saying?
Cole: A rack unit.
Timoni: A rack unit. Yeah,
right. Yeah. And they get some information that comes up that says if
it’s broken, if the client wants it serviced or just repaired or
replaced entirely. So anyone can have the Hololens on and, using an
image marker, know what is contextually needed for this particular
server RAC. An advantage using augmented reality for this versus just
having a bunch of displays is that the monitors don’t break; if a
Hololens doesn’t work, you get another one. That’s awesome.Is there
any other use cases that you’re using augmented reality for? Or
virtual reality? I like seeing the warehouse at scale, etc.
Cole: Absolutely. Yes. And some
of the work that you guys are doing I think is incredible. If the
Hololens breaks or if a Magic Leap breaks or whatever the hardware
happens to be, to go back to that cliché quote, Mark Andreesen said,
“software is eating the world. If something fails in hardware,
you should be able to take out your phone and have that same
experience.”
Timoni: Exactly.
Alan: I think that’s the real
key to scale.
Cole: It has to be, right? It
has to be kind of a “bring your own device.” You have to
get to that point.
Alan: Well, even the new
glasses. So Nreal launched their glasses this week in AWE and their
glasses plug in USB C into your phone. It’s using the processing
power of your phone to give you really, really good heads up AR, and
it’s positional tracking and everything, just kind of very
lightweight pair glasses for $500 bucks.
Timoni: And the image quality
was really great. I mean, the field of view, obviously it’s
constrained to the glasses. But it fits so nicely in the frame, I was
super impressed when I tried it out.
Alan: They’re very lightweight,
comfortable.
Cole: And this is what’s amazing
about Silicon Valley today. I mean, I’m just reminded of where we
are, and I just wish you guys can see this–.
Alan: You know what? I’ll take a
video of us driving up and we’ll put it in as a gif.
Timoni: It’s just like Lord of
the Rings.
Alan: We’re driving with
thousand-year-old redwood trees going up a giant mountain in the
windiest road you’ve ever seen.
Timoni: Talking about AR.
Alan: In an enormous tank of a
truck.
Cole: It’s true. But the pace of
innovation, if you think back — Alan, you and I were chatting
earlier about your first experience of VR — and I just–
Alan: Actually I got to say
this; my first experience in VR was at Curiosity Camp five years ago,
and we’re on our way to Curiosity Camp now.
Timoni: Oh, that’s amazing.
Alan: And then Chris Milk put VR
on my head and I had this kind of “aha, come to Jesus”
moment when I was like, “this is the future of human
communications.”
Cole: And isn’t that what got
you into tech?
Alan: Yeah.
Timoni: That’s… Wow. That is
so cool.
Alan: Well, I was in tech
before, but I made DJ stuff. So yeah, a little bit different.
Timoni: It’s all coming together
now.
Cole: It totally is.
Convergence, you know?
Alan: You were to talk about
kind of the speed of acceleration of technology? And I think people
neglect– because maybe they’re working in IoT, or are working in
cloud computing, or they’re working in 5G. But if you take a really
10,000-foot view, they’ll all work together. And the fact that they
all work together and they’re all in their infancy now, but all
maturing at the exact same time. You have IoT, 5G, quantum computing,
cloud edge computing, you’ve got blockchain, VR, AR, all at the same
time.
Timoni: Also, I truly believe
that — and this has been coming up a little bit more slowly than
perhaps that you just described — but the moment we started really
having sensors on computers and being able to make sense of that
data, apply semantic analysis to it — that is another turning point
in computing. That’s the equivalent of going from the mainframe
that’s the size of a room to–
Cole: I got chills, that you
actually said that. Yeah. It’s great that you did.
Timoni: Yeah. That is… wow,
you’ve really got goose bumps… this is the next great thing, having
all this world information and then having computers able to
understand what’s going on.
Cole: It’s a hundred percent
right. Lord Kelvin — just a little history — Lord Kelvin, if you
know the Kelvin scale.
Timoni: Yeah.
Cole: He said to measure is to
know; if you can’t measure it, you can’t know it.
Timoni: I love that.
Cole: It’s a really cool quote.
But think about what we could do and what we have access to. Alan,
you mentioned 5G. What we have access to in 5G is a network that is
as real as the fiber optics in the ground. With speeds that are the
same. So from a latency perspective, human eye can see 150 points
vertically and 180 points horizontally. And at every point there is a
level — you can see about 200 points of data — it’s chemical.
Timoni: And there’s different
resolutions.
Cole: And different resolutions.
But you take some mild compression associated with that to deliver a
4K experience to each eye.
Alan: And then foveated
rendering.
Cole: Its refresh rate. You’re
talking about 10 gigabits of data per second per eye.
Alan: And then that’s not
including what you’re collecting from the sensors from the outside
world to make it all synchronized.
Cole: That’s 100 percent
accurate. So I mean, think about what you can do with the
contextualised data with the real world.
Alan: That’s all I think about.
Cole: Yeah, it’s incredible, the
capabilities we’re going to have over the next five years as these
new networks come out.
Alan: It’s super human.
Cole: It’s going to change. It’s
going to change the way humanity interacts with each other.
Timoni: Yeah.
Alan: I can’t wait till we go to
a networking thing and everybody’s — it’s facial recognition — and
it puts up their their names, and who they are, in front of them.
Because I don’t remember anybody’s bloody name. That should be the
new name tag. You wear these glasses in, everybody’s face pops up.
Timoni: I just said this
earlier, but I’ll say it again for the record for the podcast: If you
meet me and we’ve met before, and I don’t remember your name or your
face, I’m so sorry. As soon as you start talking about what you
worked on, I swear to God I’ll remember that part, because that’s
always the coolest part; hearing about all the cool shit everyone is
doing. I also love that my group has something — it’s a little
clique-ish-sounding — we say like, “oh yeah, that person, they
kind of get it,” right? What I mean by “getting it”
is that we have a shared similar vision of spatial computing outside
the bounds of not just talking about augmented reality or virtual
reality. Those are components, those areR displays in this larger
ecosystem of network computers. They run on edge that are all
consistently talking to each other, and had this world information.
To me, I don’t want a computer to be a single contained piece of
hardware anymore, nor is it, really. Every device I have is
networked. But I want to live in a world where computers sort of
surround me in the most intelligent and privacy-sensitive way. But
really, just sort of customizable to the point where I can wake up in
the morning and have computers help me along my day in the way I want
them to, as opposed to having a phone, then I have to pick up the
haptic glass, or not having my Sonos talk to my shoe lights or
whatever. I really want the whole thing to be the computer.
Alan: It’s interesting. I wrote
an article recently on BCI and AI coming together as a bi-directional
brain computer interface. So, being able to insert a chip into a
brain so that you can hijack all the senses. I talked with this
example of your walking down the street, and you start smelling
cookies and coffee and it gets stronger, stronger because you’re
getting close to a Starbucks.
Timoni: Or it hides it so I
don’t have to eat the cookies.
Alan: Exactly.
Timoni: Let me give you an
example. I’m not talking so much about… like, BCI is so cool, but
do you really want everything?
Alan: I don’t know.
Timoni: I’m having trouble with
that. But I gave this example in my talk yesterday, and I talk about
all the time; when I wake up in the morning, I want to have little
snippets of display UI that are kind of scattered around my home. And
it could be a projector, could be glasses, or it could be a branch of
them or whatever, that are all just little subsets of a larger
computer session that is happening in the cloud. And I’ve customized
and put these things where I want them. Oftentimes there’s no
visuals. Maybe I’m just talking to the computer that is in the house.
Maybe I’ll have cameras all around the house. Oh, side note; some
people will say things, like, “is it really going to put a bunch
of computers and sensors in your house?” A hundred years ago,
nobody had electricity, and we either retrofitted or were willing to
take out pipes and put electricity in all of our homes.
Alan: I think everyone is going
to have a 5G repeater in their house.
Timoni: We’ll build
infrastructure as long as there is enough value to us. And as long as
we trust it enough that we’re fine with it. And I think that’s really
going to happen. I really just want computers to be distributed
little snippets of things, like a great Internet of Things combined
with the best types of displays you could possibly have in the
moment.
Cole: As you say in your side
note, you have to make the capital work. I’m reminded of the
autonomous drones, autonomous cars, and the dollars that go into
putting everything on board. And the way I see this at city scale is
that, from a Silicon perspective, why are we putting $100 computers
on 1996 sensors?
Alan: It makes no sense.
Cole: It makes no sense.
Timoni: Yeah.
Alan: It’s because we don’t have
5G yet.
Cole: Exactly correct. Exactly
right.
Alan: But here’s the thing.
There’s so many great use cases for 5G and headsets. And up until…
the problem is nobody’s going to have a headset for another five
years. Maybe more. It won’t be a thing that everybody has and it
won’t be really good enough — in my opinion — for consumer scale.
So for now, it’s enterprise. But what can we do with the phones that
still give people superpowers? And I think that’s a really practical
thought experiment where you have a device in everybody’s pocket. The
5G ones are gonna be epic. And what can we do with that that we can’t
do with what we have now?
Timoni: I really don’t want to
be running most of my compute power on local devices. Well, Edge,
sure, fine; but I don’t want to have an individual application that I
must add onto my phone, or augmented reality just simply won’t work.
But I don’t want it anyway. Like Google Studio, for example, of being
able to run enough frames in the cloud.
Alan: That’s pretty cool.
Timoni: Have them go down to the
device. That’s more effectively what I want. And the advantage is
that if we all have these compute sessions running in the cloud or
concurrently running apps, if I want to send you something, your
app’s open, and my app is open, it’s just a better way to compute.
Cole: One hundred percent. And I
don’t know, you guys might be shocked to learn this, but I’m old
guard telco. And the reality is–
Timoni: By the way, he looks
very young. This is a very strange statement.
Cole: Very. Yeah, this is like
the late 90s. But we’ve built this thing called the Internet
backbone, which we all know on the wireline side. But what a lot of
people don’t know is that on the wireless side, we built —
fundamentally — four different networks. We built the modern
networks as we know them today with AT&T, T-Mobile, Sprint,
Verizon, etc. They all built their 3G networks and their 4G networks,
etc., and we’re all plugging into this big Internet backbone because
that’s what you do. If we’re on our phones and are watching YouTube,
we’re going to Amazon or Netflix, — whatever it is — we’re back on
the wireline side. We have to get from the phone and the modem and
the phone to some radio substation that’s mounted on a macro cell
tower site. Take fiber optic cable back to a data center somewhere.
And it is so not optimized. I mean, you might be shocked to hear that
it is not.
Timoni: I always imagine the
building that Netflix had to build in New York. They had, like,
Radiolab podcasts about it or something.
Cole: Yeah, yeah. You don’t get
better at delivering these experiences by algorithms, because you’re
not going to algorithmically speed up the speed of light. This is
really where edge comes in. Right? If latency is a function of
proximity, all of a sudden, you need to move the compute as close to
the device or the user as possible.
Alan: But when it’s not…
Cole: [crosstalk] I’m all for
it.
Alan: I don’t think people are
ready for this, because once we figure out edge computing at scale
with all of these other technologies at the same time? I don’t even
know.
Timoni: What are you worried
about?
Alan: So here’s what I’m worried
about, and it keeps me up at night: AI is going to really quickly
replace large swaths of jobs, mainly accounting jobs and data
sciences, and huge office towers full of people that are literally
doing spreadsheets are gonna be wiped out because they don’t need
that anymore. Lawyers — entire swaths of lawyers — are gonna be
gone (which I don’t think any of us are really going to complain
about). It’ll be the law firms with the best AI that will win. And
that’s a whole job category gone. And it’s not that there won’t be
other jobs, there will be a transition. But the transition is going
to get quicker and quicker and quicker. And I don’t think we have the
infrastructure education-wise to retrain and reskill people fast
enough, which is why VR an AR is needed for education and training.
Your use case is literally paving the way. That’s why we want to do
this as a podcast, because you literally have built something that
will be a case study for years to come.
Cole: Look, we hope so. But I
think to your point — and you could do a whole separate podcast; in
fact, we should do a whole separate podcast on the political
implications to this. Part II! — do we tax computers? If narrow AI
is replacing sort of these first-level jobs in these companies, do we
tax them?
Timoni: Okay. So if in the
future, we have AI that does genuinely replace human workers — and I
have some reservations about whether or not that will happen, that
have nothing to do with the tech and everything to do with
socialization — I think that if we tax the AIs, does that lead to
the window tax problem? In the 16th century, “let’s tax people
by the number of windows they had,” and everyone then bricked up
their windows to avoid taxes.
Alan: I think we just have to
have a restructure of the tax situation so that corporations pay
higher tax than individuals, because what’s going to happen is
individuals are going to have fewer and fewer long-term jobs and will
be more the gig economy, which if we can fundamentally teach the
younger generations how to use the tax loopholes by incorporating
themselves and using the tax loopholes, then you’ve actually kind of
artificially changed the tax structure around; because right now the
tax structure is based on taxing the individual at the highest rate,
and corporations get all the breaks. Well, if individuals start
acting as corporations, then you get the breaks and the government
will take a while to catch on… wow, that’s beautiful. We should
stop there and take a photo.
Cole: Do you want to?
Alan: I think we should.
[Intermission]
Timoni: So one thing I point out
is that through most of human history, we did not have careers.We did
not have salaried jobs; only the landed gentry could be assured of
this consistent sort of income. And while there were other social
structures in place to make sure that most people reasonably knew
where they were going to be able to eat or, you know, it was more
that the vagaries were natural, not social. This whole return to the
gig economy means we were only changing the social structures from
the last hundred years. So while people might think of it as this
long-term system that is fundamentally changing how humans are. If
anything, the last century was the blip and this is a return to the
norm.
Alan: I think we can fix making
women who just had a baby work after they’ve gone… It’s crazy. We
should be celebrating that and making sure the parents have as much
time with their children as possible because that’s what makes the
whole thing better. Not making people work 80 hours a week.
Cole: A hundred percent. If the
computers, if you can tax them, and maybe.
Timoni: Lessen the human burden,
is what you’re thinking?
Cole: Well, look, if there’s
more time for us to do the things that we care about — not saying
that we shouldn’t care about our jobs — but some of the things
narrow AI can accomplish alleviate some of the pressures on how we
would train, how you would optimize for that function to be done by
human. Does it not make sense if we’re taxing the computer that we
create some universal basic income?
Alan: One of my friends, Floyd,
is a huge proponent of basic income and it’s something we have to. I
think nobody’s going to go for it. First of all, in America, it’s not
going to happen. But what we can do is we can change the tax laws to
tax the corporations to basically redistribute and give services to
people and make it so… because tax was there to serve the people,
not greedy corporations, that only government… something got off
track in the world. It’s not just America, but the world. I think too
many people let bankers get away with stuff.
Timoni: I think it’s gonna
settle down. The modern economic structure is not even 500 years old,
not even. I think we’re at a weird sort of inflection point and
people will start over the next few generations. I always like to
think very long term. Over the next 60 to 100 years, I think we’ll
start to calm down again. I just… dystopias rarely happen, and they
happen in blips and they don’t usually last that long. I don’t know.
And just one more thing. If it turns out that we actually in the end
should have a series of large scale-companies running the world, that
might not be terrible as long as the companies are set up correctly.
Alan: I agree with that.
Timoni: A lot of people are
going to be like, “no, corporations should not be in charge of
anything!”
Alan: Here’s my challenge to
that: What is the one measurement of the success of a corporation
that the world uses as a standard?
Timoni: Today, you mean?
Alan: Yes, right now.
Timoni: Shareholder value.
Alan: Economics. Yes. How much
money? That’s it.
Timoni: And that has always
been–
Alan: It’s artificial
shareholder value. Like, we drive the share price of companies up
based on nothing and drive them down based on nothing.
Cole: Not “nothing.”
So obviously there’s a microcosm that exists in Silicon Valley, but
for a publicly-traded company, its one of two things. Either a
dividend payout, or you’re paying off your debt and growth. That’s
what it is.
Timoni: But that has been the
case for any social structure humans care to name throughout all of
human history. You either grow or you’re stagnant and you don’t grow.
Corporations just happened to be like a very close, tight-loop
version of any society. Could have been a kingdom, could have been a
tribe. What have you. Humans: we’re consistent.
Alan: We are that.
Cole: I feel like the hardware
is more consistent. The software… I often think of us as
1st-century hardware running on 21st-century software.
Timoni: Yes, totally. Totally.
[Intermission]
Alan: So we just got back in the
car, we stopped on the side of a mountain to take some beautiful
photographs and we’re back. I don’t know where we left off, but let’s
think about this. We were talking about the future of work, how
technology is going to fundamentally change, how we train and educate
people. But also…
Cole: We were talking about
what’s the role of… maybe this is a question I’ll ask you guys.
What is the role of the first-level citizen when narrow AI is
starting to–
Timoni: Actually come into play
in the factory, and becomes a worker?
Cole: Yes. Yeah. Actually
becomes that worker.
Alan: The thing is, it’s not
going to be overnight that it becomes the worker. What it’s going to
do is slowly, one by one, take large swaths of tasks.
Timoni: So… and two things
here. First, AI is a [expletive] misnomer. People are like, “cool,
an artificial intelligence.” Nope, this is a heavily-curated and
single-purpose, like, basically extremely good algorithm that is
designed to do like one single thing. And it might be like, find cats
with whiskers versus don’t find cats with whiskers. Right?
Alan: But it could also
procedurally generate digital humans.
Timoni: That is not going to
happen for very long time.
Alan: Or buildings, which I’ve
seen.
Timoni: Oh, you mean like an AI
that makes the procedurally-generated buildings?
Alan: Exactly.
Timoni: Sure.
Alan: So the content for
something that would take a content artist — a 3D rendering artist
— maybe months to build a scene, now they’re procedurally generating
this stuff in seconds. So that’s interesting.
Timoni: So you touched on this a
little bit earlier. I mentioned this very briefly in our last round
of podcasts. But there’s a social component, right? So I have had the
good fortune to meet a lot of people who make top-tier, triple-A
content for movies and for games. These are the people who will
obsessively work, like, thousands and thousands of hours to bring you
the final scene in Avengers. And even if they have procedurally
generated content, the reality is they always feel like they can just
finesse the shit out of it, definitely. And what I see is the
creators getting more and more picky. I always go to the
post-production talks at SIGGRAPH and they’ll talk about how, like,
in Blade Runner, how long it took them to get an artificial human
looking good — the Rachel character — and how they had to argue
with the director because he wanted the scene in Vegas to have no
blues in it whatsoever. And they ended up creating this new type of
filter for the camera that had no blues; the director saw it and he
was like, “no, I still see some blue,” and had to literally
prove that there were no blues in the scene. Now, this is on top of
all of the best tech; like, these are the highest-end effects houses
in the world. These are the ones who are really pushing the limits
and they’re working together. It’s WETA, it’s Rodeo. It’s all of the
other greats. And so what do they do when they have these tools and
these great machine learning algorithms, they get more and more picky
and so on.
In Infinity War, they literally had, I
want to say like five petaflops of data, because they scanned every
single character cropped in-scene in that whole movie so could have
it in post. And OK, so sure, maybe at some point these will actually
be replaced by AIs. But I feel like humans inherently don’t trust the
machine enough, or we just want to get our hands dirty. And with
lawyers, too. Sure, you have an AI that can make a better decision.
But the reality is humans do not make decisions based on data. We use
data to justify our predetermined decision processes.
Yes, but if we did use data, we’d be
actually way more effective. Honey, I know. So that’s the thing. If
if a law firm figures this out and says, hey, wait a second, this AI
is outperforming our lawyers 10 to 1 because it will on everything,
because IBM Watson can read 5 million case files in an afternoon,
making the lawyer read five maybe. But what are you trying to do?
You’re trying to set a precedent, be looking for precedents. Maybe
you’re looking for. You could scan the entire country’s records,
looking for precedents in seconds. But who created a.
Cole: So I always come back to
the the whole, “do computers dream of electric sheep” and
the morality. Look at the divisiveness going on in social media
politics right now. It’s been determined that the coders that write
this stuff up, they have cognitive bias and they write that into
their code. And the code gets trained and it becomes biased.
Timoni: And that’s how you end
up with a hella racist [AI].
Cole: [Laughs] that’s exactly
correct!
Timoni: But here’s a cool thing.
I actually love this about the AIs; when we are concerned about bias
and data that we need to de-bias the data — and honestly, we’ve only
begun work on what that even means — what does it mean to be biased?
What is a cultural bias versus a universal human value that needs to
be removed or cleaned up or gotten rid of? Then also it forces us to
think about ourselves. And I really love that if you meet a machine
learning algorithm that happens to be biased, that teaches you
something about yourself. Right? In the cold light of day, if
something was you and did the same thing five billion times until it
trained up to, like, the essence of you? I think that’s a great
learning tool. No one’s gonna think about it like that. And now it
has. They just sort of talk about it and reactionary way. But there’s
some real value to that. I actually love the idea of having this sort
of listener that I can talk to that helps me work through my biases,
because it can see where any individual’s action I take or statement
I make, like, where that can go, taking to its logical extreme.
Cole: Yeah, I don’t know if
humans are ready yet for it for the self-reflection that would take
to actually get over it. I think we live in a world of naive realism.
Timoni: You know, I love that.
Yeah. I think you’re right. Well, it’s true. For example, I’m a
designer by trade. Did UX for years (we called it information
architecture before). I studied literature in college. And we always
viewed literature through what we called it at the time, “different
critical lenses,” which I now know are just different mental
models to a different context, or at least that’s how I would
describe it. And there are a lot of people online. There are entire
communities around rationalism and mental models and really trying to
get to the right decision. To your point. Like we don’t care about
being… let’s see, what’s a good way to put it?
Alan: We don’t need to feed
people’s egos. We just need the right answer.
Timoni: Exactly. It’s what is
known about being right. It is about seeking the truth and finding
the truth. And this goes all the way back to the pragmatism of
William James, early 20th century. Well, he probably even before
that. I don’t know. I do feel like we’re… maybe it’s just because I
live in my tiny little rationalist bubble, but I do see more and more
people talking about this stuff and interested in this stuff. And I
can’t help but think inherently most people would rather be right
than… I don’t know. You’re laughing.
Cole: No, I think you’re
optimistic. But I think you’re right. I just think it’s going to take
some time.
Alan: I think this road has
gotten sketchier.
Cole: Yeah, it’s gotten
narrower, for sure.
Alan: The road went from two
lanes in a winding road to one lane.
Timoni: It’s very s-curvy right
now.
Alan: It’s pretty beautiful.
Timoni: And it’s right dappled,
like the sunlight as we as we go further into the woods.
Cole: Right. We’re going further
down the rabbit hole.
Alan: Yeah. This is great.
Timoni: So getting back to
machine learning and artificial intelligence, I do think, as I
mentioned earlier, I really want to see people starting from where
they want to end and start with what their vision is. And then we can
work backwards from there to figure out what could potentially go
wrong. What I see instead is people sort of being alarmist about what
could possibly go wrong with no real end in sight. And that just
always ends in this kind of dystopian picture of, well, imagine a
world where people know your every thought, emotion, and hope, and
therefore can constantly feed this to you.
Alan: I’ve got news for
everybody listening. Guess what other people are thinking about you:
Nothing. They’re thinking about themselves, really.
Timoni: I mean, it’s true.
Marketing is designed to manipulate you. But there’s given a
population, too. Right?
Alan: “Make better health
decisions. Exercise. Think positive things.”
Sure. But if I’m at Disneyland, which
is a gigantic mega corporation and I’m waiting in line for Indiana
Jones, the line is designed to make me feel like it’s taking less
time than it is. That is a great example of good environment design
that does, in fact, manipulate you. And yet, it’s to everyone’s best
[interest]
. You can’t make the lines shorter. Right? So why not make
it more pleasant along the way?
Alan: It’s a great analogy. OK,
so let’s go deep down, since we’re going down the rabbit hole.
Timoni, what is your vision for the future?
Timoni: So I would like to see a
world in which everyone is able to use computers to the best of their
abilities, imagination, and intelligence. Right now, people are using
computers all the time. We talk to computers more than we talk to any
individual human throughout the day. And yet we have this sort of
siloed set of experiences where people can do a task per application.
Part of this is just due the nature. Then I have a niece, for
example, who is on TikTok probably six hours a day. But if she wanted
to describe and illustrate and animate a dream that she had last
night, she would have no ability to do this. I think there are
several different ways that we can attack the problem. First is to
make creation tools that are easier to use, which I think we’re
continuing to evolve, and AI can actually help with that with
procedurally-generated things. But I also think that we just need
computers to be able to listen and react to humans specifically. We
do not have operating systems right now that can listen for what I
call “the no.” If a computer does something I don’t like,
if an application does something I don’t like or don’t want it to do,
there is no “no.” We see this slightly with notifications
where it’s like, oh, do you want less notifications or to turn them
off? We’ve had computers for 60 [expletive] years and that’s as far
as we’ve gotten?
Alan: Turn off all your
notifications. It’s actually liberating.
Cole: Yes.
Alan: Turn them all off. You’re
going to check the apps anyway.
Timoni: I do the same thing.
Cole: How about a better life
hack: leave your phone at home a couple of times a week.
Timoni: Oh, interesting.
Alan: Wow.
Timoni: You do that?
Cole: Yeah.
Alan: Do you really?
Cole: Yeah. Yeah. I mean, yeah.
Timoni: Are you carrying an
iPad? Are you cheating here?
Cole: No it’s so good for your
mental health. I mean, I’ve not had cell phone service on my phone
for 45 minutes now and I’ve not had to look at it. So no, it’s not
there. Or if you can’t leave it at home while you go take a hike in
the woods with no spectrum, go to dinner with friends. Hang out with
real people. Put your phone underneath the salt shaker and whoever
picks up the phone first–
Alan: Pays the bill.
Cole: Pays the bill! Put it
down.
Alan: Yeah.
Timoni: That can lead to some
lengthy small talk after dinner.
Alan: “We’re not leaving,
but the bills getting higher.”
Timoni: That’s cool. But any
case, getting back to a vision for the future. I want to have my
computing session in the cloud. I want apps to work interoperably. I
want to have a series of continuous… like, in my house. I have
pasted a little digital interface here, a little digital interface
there, I put up a little inventory. And it’s got all the things that
I want. And I can combine them in any given way to do whatever it is
I want to do on a computer. Actually, a RISD… I think MFA student
recently posted a concept OS that he made called Mercury. I highly
encourage you all to take a look at it. It just came out, I think on
the 28th. If you just do a Google search for Mercury OS, it’ll come
up. And he had rethought the concept of an operating system as a
series of stated tasks — like “I want to check my email”
— and then everything that you would need in order to effectively
check your email, which does not just include your inbox, comes into
a set of containers and this is called a flow state. So you’ve got
your inbox, but then maybe you’ve also got your calendar open, and
your map open, and your to-do list–.
Alan: Ahh, the tabs that you
need for that.
Right. And you can drag and drop in
between all of these different what we now think of as applications.
But if you remove the data layer itself from the container, from the
visualization, then you have a really robust way to interact with the
computer and digital objects, and in a way that makes sense for you
at the time, in the mode that you’re currently in. The cool thing
about this for augmented and mixed reality is that it makes no
difference if you’re doing this on a 2D screen display or if you’re
doing this in a headset that is also showing your 2D screen, or a 3D
screen, or a 3D object if that makes more sense, depending on what
you’re doing now. This is really essential for augmented reality that
we start to remove the data layer from the container layer, because
if I am in augmented reality, if I’m looking through my Hololens and
I have two apps open with two cubes that look identical, one in each
application, I cannot — and I can’t expect any user — to context
switch between what one app does and the other app does. Like, if one
of them is a modelling app and the other one is an interactivity app
and I’m dealing with the same cube, it needs to be the same cube in
the same place. So what I’m talking about is, like, far afield; 50 or
100 years. We’re going to have to rethink computers, but we’d start
talking about it now. We’ve been talking about it since the early
2000s. Let’s just continue to push this idea forward.
Cole: Quantum, baby. It was only
a matter of time before quantum computers came into this podcast.
Timoni: Ok, so let’s talk
quantum.
Alan: Let’s talk quantum.
Timoni: Should we pause and
quantum later?
Cole: I think we should quantum
later.
Alan: So, part 3 of this podcast
is going to be quantum later, around the campfire.
[Intermission]
Cole: …government was any
block as part of a blockchain. Now all of a sudden, you are in
control. You have an immutable record of truth.
Alan: And you can cancel it.
Cole: Exactly. You can expose…
Timoni: So for context, we’re
talking about why is it bad to have your data collected? Why do we
keep hearing people say, “oh, but what if the insurance agents
know that I have cancer before I do and then [expletive] me over?”
The answer to this question as they shouldn’t be able to [expletive]
you over. Right?
Alan: That’s the simple answer.
Timoni: In the world that we
want to live in, it should be great that everyone knows you have
early stage cancer so you can fix it. So now Cole’s talking about
this concept to the smart citizen who has ultimate control over their
data.
Cole: Yeah. And I just think
it’s going to take… so A) It goes back to what you talking about
before, which is how do we get corporations to realize that the data
they collect is for the benefit of all mankind? And that’s hard to
get them over because right now the dividend or the capital power or
the debt pay off, etc. or the growth is what these guys get rewarded
on today. So I think it takes a–
Alan: What if — ready? What if
we had a new system and it was an education-based system that,
instead of charging people to learn, you got paid to learn? Every
time you learn something–
Timoni: What is this,
Scandinavia?
Yeah, well, I come from Canada — we
have socialized everything, c’mon. But we paid you to learn. And so,
little micro currency; five minutes of reading gets you five points
or whatever on you on your blockchain ledger. Right? But what if that
same system also took care of your health care, your insurance, your
banking needs, everything that you need. Kind of like WeChat, but
instead of one corporation owning it, as you progressed in the
education component and you graduate, you become an equity
shareholder in the company? And the company that has paid you to
learn the whole time now is selling you all the services you need,
but you own that company that’s selling you the services? So you’ve
basically created like a shareholding system, but nobody can own more
than anybody else. Everybody’s equal in it, and it automatically
waterfall distributes the profits.
Timoni: What the profits are..?
Alan: The profits would come
from a number of different ways. So the participants in the program
are being educated and trained in mindset and maybe it’s a percentage
of their income in perpetuity or something like that. But they always
own this company. I don’t know what that looks like in the long-term,
but the company itself can make products that are sold outside of the
network. We can make products like a health care product; if you make
the best health care insurance in the world for your members, other
people outside are going to want it as well. And you can give it,
make it available to other people, make it a huge profit center. They
didn’t grow up in the system. They can only access certain services
that are profitable to the system and the people. You have to go
through the system to be part of it. You can’t after 18; you’re not
allowed in. Like, you have to go through the system, graduate from
it. Now you’re in and you’re in it for life.
Timoni: Sounds a little caste-y.
I don’t know.
Alan: I’ve been working on
different ways to solve this at scale around the globe. So then now
you have global citizens all interconnected with the only purpose of
helping each other.
Timoni: So one thing that has
always puzzled me is why doesn’t socialism work unless it’s kind of
sneaked in? Like, socialism is a great idea, actually, I think as a
concept — like, on paper. Absolutely, this is sort of a tribal
communal way humans kind of inherently want to work and think anyway.
And yet at scale, when people claim they’re going to start socialism
or a communist country, it always ends up being a cult of
personality. Right? It always ends up actually being a fascist
society instead. And yet they call themselves that.
Alan: Because it’s usually by
some egotistical leader.
Timoni: But why do you think
that is?
Alan: Well, why do you think you
have the president you have here? The public can be easily swayed by
showmanship and flashy shit.
Cole: Is it fair to say that
power corrupts and absolute power corrupts? Absolutely.
Alan: I see what you did there.
Timoni: But why can’t we just do
socialism from the get-go as a stated goal? Sans cult of personality.
Alan: This is what I’m
proposing; a new social exchange where everybody benefits from going
into the experience economy. So we’re not going to buy cars. We’re
really not going to need to buy houses. We can live anywhere.
Imagine, you don’t need to own anything, but you need to have access
and experience everything. And so what if part of the program was
experiences? And as you educate yourself more and became more
valuable to the organization, you got invited to more and more
experiences?
Cole: Yeah, I think this has to
start. So today we talk about the… I think it’s up to 75 billion
connected devices by 2025? Ridiculous.
Alan: 44 zettabytes by 2020.
Cole: Correct. And 120
zettabytes by 2025.
Timoni: I didn’t even know what
a zettabyte was.
Cole: A zettabyte is a thousand
terabytes.
Alan: Thank God we have a data
scientist.
Cole: Beyond the Internet of
Things, I think as you pointed out, the experience. So today we have
what’s called the knowledge economy. I think after IoT, after the
Internet of Things, we start thinking about the Internet of Skills.
And with that, with the Internet of Skills, I think you’re going to
finally get to a place where, end-to-end, you could build the proper
incentives for contextualizing what’s good for a corporation in
context of what’s good for a human.
Alan: And what if the only goal
of the corporation was the well-being of the students and of the
members and owners of the corporation?
Timoni: Have you read The
Diamond Age?
Cole: Yes.
Timoni: The Diamond Age has
corporations, as sort of citizen state economies where — with class
systems.
Alan: I’ve got to read that.
Cole: Yeah, it’s good. That’s
funny; I’m reminded of a company… do you guys remember, back in the
early, early, early 2000s, around Napster and what was happening? It
had prompted me to think about a company that it would — it would be
illegal. — put your hat on for like 20 years ago and this is
controversial. But could you build a company where if you were a
monthly subscriber where that came in the form of some sort of stock
in the company, you are a shareholder and you can provide that
platform to your shareholders. Back then, could you do some kind of
peer-to-peer network where, as a shareholder, you’re entitled to the
content that sits on that network?
Timoni: Like Usenet?
Cole: A little bit. A little
bit. So. But it takes… I mean, good luck. Have you ever read Flash
Boys?
Timoni: Oh, I’ve heard of it.
Cole: Fantastic Michael Lewis
book. And he actually has a list of podcasts. And he did a recent
podcast, fairly recent, called The Magic Shoe Box. Really
interesting. And it’s all about high-frequency trading and the
latency associated with high-frequency trading. And a general by the
name of Ronan Ryan, who who went and created entirely new stock
exchange just to create fairness in the stock exchange. So coiled up
miles and miles of fiber, got rid of the HFT guy. So it took no skill
at all. The idea of stock exchanges, where you are informed enough
about the mission of a company that you feel like you want to invest
in what that company is doing and where it’s going, high-frequency
trading came about. And a lot of these companies, just because they
were one or two milliseconds faster, they just saw big buys happening
so they could get in front of that. They could buy it, then sell it
to the actual purchaser. Billions of dollars.
Alan: I think the major problems
in the world can be solved by putting a leash on bankers, because
they tend to make the rules in their own favor. And that’s how you
ended up with–
Cole: Money makes the world go
around.
Alan: You ended up with a
trillion dollar bailout or other.
Cole: That’s right.
Alan: And here’s the thing. Many
people don’t realize this right now in America. They don’t realize
that the problem in 2008 with the subprime mortgages, it’s being done
all over again with retail properties. Some of these big malls that
are now empty because the big players have pulled out, the malls are
dead. They’re all still being treated as if they were full of
triple-A real estate rates.
Cole: So you know, the guy that
ran that hedge fund–
Alan: Yeah.
Cole: –is that Curiosity Camp,
today. Yeah.
Alan: Wow.
Timoni: All right. Well, we’ll
talk to him.
Alan: Let’s have a little
conversation about this.
Cole: He was the guy that went
to went to Goldman Sachs and said, “hey, this is what we want to
do and we can short all of this stuff.”
Alan: Well, he saw an
opportunity.
Cole: If you guys are out for
it, whether it’s at Curiosity Camp next year or else, I’m down to
make this an annual special edition podcast.
Alan: I love it.
Timoni: This is awesome.
Cole: You were talking about,
you can just create the rules.
Timoni: Yeah. Yeah. Exactly.
It’s to see a curious property of money. And I think also, again,
just like the modern economics is just not very old. It’s like as old
as the enlightenment and started trading in the rise of what is now
the modern day corporation. You do what you want until a law stops
you. Usually you can do what you want at least once before someone
makes a law to stop you from doing it again.
Cole: Do you guys know this guy,
John LeFevre?
Timoni: No.
Cole: John LeFevre has the
Twitter handle “@GSElevator.”
Timoni: OK. OH! Yes. Yes. Yes.
Yes. Yes. “Goldman Sachs Elevator Overheard.”
Cole: So he actually never
worked at Goldman Sachs, but he wrote a book. He did run the
syndicate desk for City, if I remember correctly. But he wrote a book
called Straight to Hell: Deviance, Debauchery and Billion-Dollar
Deals. All of that investment banking in the 90s. And it is… I
highly recommend it. It’s very short, but a really good read. And you
do make up the rules as you go along.
Timoni: It’s partly like
interior social motivation. Like, obviously you want to win. People
who do this are highly competitive by nature, etc. But also you do
have a mandate to make money. However you feel you can best do that
within the law, within your own ethical practices, or whatever you
think. I was listening to Knowledge Project recently and I forget who
was being interviewed, but they were talking about Enron and how the
CEO thought of himself as a fundamentally moral person who was doing
the right thing. People are going to laugh, but even his, I guess,
pastor vouched for him as just being this fundamentally good person.
And I think there comes a point, especially when you are authority,
it means that you have the responsibility for multiple different
large-scale entities, the corporation itself, to shareholders and
then the people in that corporation. I can see where you think you’re
really working in the best interests of all against all of what
anyone would call conventional morality. And the whole banality of
evil, yadda yadda. I get it. But like, I understand where this is not
something humans are good at thinking through on a macro scale.
Cole: Yeah, I couldn’t agree
more. You end up having to do some pretty gigantic mental gymnastics
to get to what Enron did and say yep, there were ethics and good
intent behind those decisions.
Timoni: Right. You can see how
they got there.
Cole: Totally. C’est la vive.
Timoni: No, “So say we
all.”
Alan: Interchangeable. So we
just pulled into this…Oh my God!
Timoni: I feel like rowing a
boat in choppy water.
Alan: We are in a boat of a
truck. Scout camp is beautiful.
Timoni: Oh, so speaking of, you
were talking about your first time doing virtual reality. My first…
53 min
February 04, 2020 Talking AI and Future of Work in XR -- In a Truck -- with Timoni West and Cole Crawford
This week’s episode goes all the way back to last year’s Curiosity Camp, when Alan shared a ride with Unity Lab’s Timoni West and Vapor IO CEO Cole Crawford, recording a podcast along the way. The three discuss the challenges that will arise as AI begins to replace human workers.
Alan: In a very special episode
of the XR for Business Podcast, we're driving in a car with Timoni
West, head of XR... Research?
Timoni: Director of XR in Unity
Labs.
Alan: Director of XR at Unity
Labs, and Cole Crawford, CEO of Vapor IO. So we're driving on our way
up to Curiosity Camp through these beautiful winding roads, and we
decided that we would record a podcast, because Cole, in his
incredible company building the infrastructure of cloud computing,
they built an AR app to help service that. And I thought, what a cool
way to use this technology and this time on this beautiful drive.
Wow. Look at the size of those trees.
Timoni: They are enormous.
Alan: Oh, my God. Wow. Well,
anyway. Timoni, how are you doing?
Timoni: Excellently. And I'm
also enjoying the view. Yeah. Yeah, actually, Cole, I'm really
interested to hear more about why you chose to go with that, and what
the process was like. My team is working on tools for mixed reality.
So for Unity itself, that's used to make, I think, 90 percent of all
Hololens applications right now. Century is using Unity for that. But
the tools that we're making today are allowing, I think, for you to
more easily make robust, distributed applications that can work
across various devices and for various users.
Cole: And that's very needed.
First off, Alan, I just want to say, you sound like you should be a
podcast DJ.
Timoni: So it's cool that you
are.
Cole: Well done. But yeah, I
mean, the issue for us when we started down this journey was very
much a question of, how robust can we make an experience, about how
widely could we make that experience? And the vertical integrated
solutions that you had to choose from in the early days of AR/VR, I
think, are primed for disruption. I'm super glad to hear that Unity
is working on the open APIs, etc., needed to bring this technology to
more users, as I'll quote -- maybe a little cliché being where we
are and where we're going -- but--
Timoni: Yeah, I want to hear it.
What is the problem you company solves?
Cole: Yeah. So we have to think
about not four, but 40,000 different data centers; we're an edge
computing/edge data center infrastructure company. And with that
means you can't Mechanical Turk what was originally done in data
centers. It works with four buildings. It doesn't work with 40,000.
So we had to build autonomy into every aspect of our business, in
every aspect of the infrastructure. And that means building really
simple interfaces for what would otherwise be really complex
problems. And at scale, from a logistics supply chain -- remote hand,
smart hands, all the things that you do in data centers -- what that
means is your FedEx guy, your U.P.S. guy, a contracting company that
otherwise would need specialized training, now it's visually assisted
capabilities for what would otherwise be a job that you would train
for and then go work in a data center. We simplify that.
Alan: So basically what you're
saying is that you've given real-time tools to anybody to be an
expert on the field, in the field.
Cole: It's fair to say that the
software is the exper, and what you need are opposable thumbs,.
Alan: Haha! Which democratizes
the whole need for training.
Timoni: You know, it's funny; I
was just getting drinks with someone from Open AI. H
53 min

About XR for Business

From the publisher's feed

Meet the leaders who are changing the face of virtual and augmented reality

Share XR for Business

Sign up to save your podcasts

XR for Business

XR for Business episodes

About XR for Business