Many scenarios that might be improved by an augmented reality heads-up display shouldn’t require an overly arduous selection process; most gizmos will do if you’re checking a weather app while jogging. The same can’t be said for picking out a device to help oil rig workers work safely and efficiently in the middle of the Permian Basin. Shell’s VR incubator lead Michael Kaldenbach talks with Alan about the things his team had to consider when selecting the right device for the job.
Today’s guest — Michael
Kaldenbach – is an augmented, mixed, and virtual reality incubation
lead at Shell, the global oil company. He is a driven, goal-oriented,
resourceful, and creative person, who really understands the
usefulness of this technology, and bringing how to bring it to the
market. He’s chosen the family motto of Arctic explorer Sir Ernest
Shackleton, as it accurately reflects how he approaches any challenge
or goal: “victory through perseverance,” or “Fortitudine
Vincimus.” He strives
to apply entrepreneurial mindsets and thinking up out-of-the-box
solutions and approaches when working in this technology. If you want
to learn more about his work, you can visit Shell.com.
want to welcome to the show, Michael Kaldenbach. Welcome to the show,
Hi, Alan. Thank you very
It’s my absolute
pleasure. I’m really excited. I want to dig right in here, because I
know you guys at Shell have been doing a ton of work in everything
from kind of marketing and trade shows, right through to oil wells
previsualization. So let’s talk about some of the ways that you and
your team are using virtual/augmented reality right now.
So I think One of the
better case studies we have is around augmented reality remote
assistance, and I’m sure you’ve seen examples in the wider industry
for that one. But for us at Shell, that means that we utilize a
head-mounted display — in this case specifically, the Realwear —
and it is used for our operators; for quick resolution, and to get
remote expertise to be brought in.
think it always helps if I provide a little story to set the scene;
think of an offshore oil platform out there in the ocean. Typically,
the most senior person is the control room operator, and there are
more junior operators that are assisting the running and maintaining
of these kind of assets. If in the control room, they see a deviation
on one of the many dashboards they have, they send out a more junior
operator to investigate — normally with a radio phone or walkie
talkie — and then they guide them through, they get back to “what
is the situation; what’s the sound the machine is making?” But
where we really revolutionize that process is with a head-mounted
display. It is as if the experienced operator has immediate eyes on
the situation. So think about [it] — you see (or I see) what the
junior operator is seeing, and thereby, I can use my years of
expertise to resolve the issue, and get back to safe operations.
case where my expertise set is also not sufficient, we can quickly be
joined by a remote expert who can be onshore — can be anywhere in
the world — to join that same virtual room, so that a three-way
conversation happens. Not only that: instead of having those
conversations like, “I recognize the problem; you need to switch
off the third button from the left, it’s kind of greenish on the left
side, bottom side of the machine,” instead, we use something called
“telestration,” and that’s the benefit of having a head-mounted
display, whereby I — as the remote expert — can draw on my screen
and the same visual is replicated to the junior operator, so in his
line of sight, he gets an annotation — a circle or an arrow,
whatever is helpful; it could also be a video — to resolve the
situation. Thereby, we quickly resolve issues that might end on
production deferment, or something even more serious. It is really
helping to revolutionize our operations.
Just to kind of recap:
for people the junior operator’s got a pair of glasses on. He goes on
site, or she goes on site. The senior operator is sitting at a desk,
being able to monitor multiple junior operators, I would expect.
Correct.
And that person is now able to remote see exactly what the junior
operator is seeing, and not only that, but annotize on top of their
Correct. And as you can
imagine, the environments that we as an energy business operate in,
they can be far-flung. They they can potentially be dangerous. So,
you really want to have the knowledge with everyone out there in the
field. This world is changing so fast, and it’s really difficult for
everyone to keep up with all these changes. By having the ability to
pool your resources — bring them via virtual reality into the
situation — that is really helping us.
So this is real-time. So
This isn’t like virtual reality, where you know you’re training in
something — and then obviously, you can use that as well — But this
is real-time. See-What-I-See interactions. Now, what about some of
the lag issues? Or are you guys experimenting with 5G technologies to
As
you can understand, connectivity is a struggle, whether it’s 3G, 4G,
and the signal range when you’re out there in the Permian Basin, or
out there in the North Sea — it’s difficult to get that signal.
Wi-Fi is also very difficult. We tend to have a lot of steel
environments, and line of sight obstructions. And in general, if you
think about telecommunications – 3G, 4G – that’s pretty much
centered around populated environments. So definitely, we struggle,
and we look to all sorts of possible solutions for connectivity —
whether that is low-part/wide-area network, whether that’s 5G,
whether it’s low-orbit satellites — it’s always dependent on the
actual business case, and what makes sense in that situation.
I always find it interesting: when we’re talking about connectivity,
and you actually cut out right when you’re talking about 5G.
[both
laugh] I
mean that’s why we need 5G.
It always happens right
around the connectivity conversation. “How do we get connectivity
stable?” And you’re like, “oh-ah-erk-uhh.” Okay! [If] we can’t
make a podcast recording work, what are we supposed to do in an oil
site in the middle of the Congo?
You guys are really
solving massive problems, using this technology. You mentioned
Realwear; maybe you can walk listeners through what the Realwear
glasses are, and why you chose them over other pairs of glasses?
Well, again – it’s
about the environments that we are in. (And let me know when I start
cutting out again).
are predominantly in something we call ATEX zones, which means that
it’s specific safety regulations due to the substances we use in our
production environments; they require our devices to be intrinsically
safe. For that reason, we look towards the market. The hardware
market, and identify the pieces of equipment that are licensed to
operate in that environment.
we started looking at these head-mounted displays about two years
ago, we did try pretty much everything that’s out there in the
market, ranging from the old-but-true Google Glass, to an ODG or
Realwear. There used to be a fair number of opportunities for us, but
again, we were sort of restricted by that ATEX certification
requirement. We went through quite a rigorous phase of proof of
concept. We tested connectivity – is this even going to work at our
sites? We looked at ruggedization – if we drop it, does it
immediately get damaged? And for us, the Realwear — which markets
itself as a head-mounted tablet, which… uh, fair enough – it’s a
seven inch screen. I don’t know if you want to call it a tablet, or
just a phone-sized screen, but it came out on top. It’s very
ruggedized, it comes with an ATEX certification, and it has all the
essentials we need; a good camera, a good microphone, and a good
screen, for us to look at the information that we need.
You mentioned good
camera good microphone. I think these are essentials, and one of the
things that I saw recently at CES was a company called Kopin, and
they make the micro displays – I think they even make the one for
Realwear – they make the actual micro displays. Something that they
introduced me to was their Whisper technology, where you could – in
very loud environments – you could talk to your device, and it
would understand it, and it got rid of all of the external noise
altogether. It was really quite amazing; somebody who is standing
right next to me was trying to tell it the instructions, and it would
only respond to my voice, which I think – as we move into these
headsets that do more than just be able to bring them information,
but they’re actually bringing up information using AI algorithms and
stuff to give us real-time information – I think having a system
that allows you to whisper or talk in really loud environments is
you chose to go with Realwear, how do you then start developing for
that? Are there off-the-shelf products? Are you looking to startups
to to help develop these things? Are you building things in-house?
Well… if you don’t
mind, let me get back to what you said, because you did trigger me a
little bit. The reason why I’m passionate about this technology, and
I’m passionate about the team within Shell, is because I firmly
believe that we’re on the cusp of a paradigm shift. We went from
using your mouse and keyboard to get information, to using the speed
of your thumb, whereby we use our thumb to operate a mobile phone and
tablet. And now we’re at the cusp of changing that technology to
voice-driven.
you are absolutely right in saying it becomes exceedingly important
for these kinds of voice recognition algorithms to not only work in
noisy environments, but to then also accurately translate what you
need, whether you have an accent or not. This journey started a
couple years ago with the likes of Alexa coming to the market. We
have other players like Cortana and Google Allo. But really, I’m so
excited. I also don’t think that we should separate voice and natural
language processing from the idea of the extended realities, whether
it’s virtual or augmented reality. For me, they represents the
foundational blocks upon which we build everything.
sorry. I wanted to leave that with you, because I feel so passionate
I think that’s a really
good point, and I think we should just dig into this a little. On my
last interview, I interviewed Rori DuBoff from Accenture, and she
mentioned XR – or extended realities – you’ve got virtual and
augmented reality, mixed reality. So, virtual reality; you put on a
headset, and you can change the whole environment you’re in for
training and stuff like that. Augmented reality; overlays data. And
then mixed reality; overlays data in context to the real world –
when you’re looking at a machine, it can give you very specific data
around how to turn the knobs and stuff like that. As an extension of
that, you’ve got voice recognition, or human language processing
(which I guess falls under the category of artificial intelligence),
but then you also have computer vision, and machine learning, and big
data analytics, and no one of these technologies on their own is a
useful solution.
need to understand that this is a continuum of a number of
technologies coalescing together, and I think this – as somebody
who studies the future of humanity – we’re reaching this
exponential growth point, where 5G connects with artificial
intelligence/computer vision/machine learning/human language
processing/virtual and augmented reality, and all running on a block
chain. If you take all that technology separately, they’re amazing;
put them together, and now we’re really revolutionizing businesses.
I think that’s, for me
also, what’s exciting about working for a larger enterprise. Because
we do have all those individual components. We have dedicated teams
working towards that, which allows the digital realites team that
we’re leading – I’m just gonna put it out there; it’s going to
stretch the mind a little bit – but we believe in this thing called
“Digital First,” whereby sort of the digital twins and the
various digital realities become a fundamental component of
operations.
idea there is that any action or operation is initiated through these
kind of digital methods, with the physical reality only being an
outcome of that digital reality. Whether it’s like collaboration in
virtual reality in order to prepare for a complicated task with
colleagues across the globe, or it’s about simulating the outcome of
like a task with a digital twin, where you run through all the
permutations. Or – one of my favorites – step-by-step guidance in
augmented reality to really perfectly execute difficult procedures,
so that we are much safer and much more efficient. That sort of
digital-first mindset builds upon all those capabilities – and some
people call it the “metaverse,” I like to term “the
digitalverse” – but honestly, it’s an exciting time, and I hope
“Metaverse,” for the
people that are listening, the term was coined in a book called Snow
Crash about, oh, maybe 15 years ago? The term refers to the world
you’re in when you’re in virtual and augmented reality – you’re in
the “metaverse” – and it’s kind of got this worldwide thought
to it.
am a little partial to that word. [laughs]
No, no; fair. But, as I
was sort of ranting on, I am reminded that what we’re doing, it’s
more than just technology. It really is about people, and creating
more agile ways of working, and for safety. And I don’t want people
to forget that; all the amazing things that we can do, it is for the
benefit of people. Just wanted to put that out there.
Let’s unpack this one by
one, here; you talked about collaboration and VR. Whether that’s
training or previsualization for something, what does that mean to
people? What is the benefit to the people in your organization, being
able to collaborate in VR?
Well, I think it comes,
first, from a mindset that we recognize that we don’t know
everything. They always said – I don’t know if I have this saying
correctly – it takes a village to raise a child. In this case, it
takes a large organization to tackle some of these big problems worth
solving. In that sense, we really need to work together. But we’re in
all places in the world. In order to really utilize the diversity
that we have within the company, you want to bring them together, but
at the same time, not be a burden to the earth, in terms of having
everyone fly in from all over the world. Where virtual reality is
definitely changing the game here is: it’s as simple as putting one
of these headsets on, and then you meet your co-workers wherever they
What platform are you using for that type of thing in virtual
reality? Did you build your own, or are you experimenting still?
For that, we are very much in the experimentation phase. The idea’s
there – and I mean when I say that the idea’s there, the core idea
has been around for ages – and when we first started this journey
two years ago, we saw maybe three players out there in the market. In
the last year, they have popped up like mushrooms.
Probably
three coming every month now.
[Laughs] Exactly!
So that makes it a
The ones we’ve tried are Roomi, Alt Space – obviously, is not
really one for professional – but Alt Space, Roomi, VR Chat, High
Fidelity. There’s one called Glu out of Finland. There’s a bunch that
have come up. I tried the Glu one; I was really impressed with that
one, actually. It was really good.
Well, it sort of reminds
me – I might be betraying my age, now – but it reminds me of
Second Life, and when they started, how amazed people were, and how
it brought people together – not in the sense of a game, but in the
sense of a community. You saw the community efforts build beautiful
things, and functional things as well. We saw companies go to Second
Life and open up their branch there, for customer support.
when I say that, I see this sort of same moving happen out there in
the market right now. But we’ve learned from Second Life, and these
kind of relatively uncontrolled environments are difficult to manage
for an enterprise such as ours – never mind the plethora of legacy
systems that we have in operation that you would like to tie into
these kind of environments. The current market, with the many, many
players out there, is not conducive for us to select one. Right now,
we’re very much testing the waters; finding what works for us, and
what not does not work for us. In honesty, I think we’re going to
have to go with one partner, and then help them achieve what we need
Yeah, I think that’s
what other people are doing, as well. They find the solution that’s
closest to what you need, and everybody rolls up their sleeves to
just make it what you needed to be.
is a great example – and it’s a great segue – because something
that we’re about to announce… I’m not going to talk about it on the
show right now, but something we’re going to announce is that
collaboration between enterprises, like Shell, and startups that are
building these technologies. Because you guys could probably build it
in-house, but you’d have to find the people to do it; find people
that are passionate. It’s better just to partner with a startup, give
them some funding, and say, “here’s some funding so you don’t have
to worry about paying your bills, and we’re going to be your client,
and we’re going to tell you what we need, and help you build it for
us. Then you can have it as a product after that.”
Correct. And in that
sense, companies like Shell, we’re not – I have to be careful when
they say it is a little bit – we’re not an IT company. We have a
lot of IT components, and we spend a lot of money on IT systems; but
in our nature, we are not a company that’re going to build these kind
of platforms, and then sustain them, make sure that you evolve into
the market. In that sense, we’ve adopted a platform as a service;
software as a service preference. So, market standard, unless we have
a severe gap in what the market is doing. In that way, we prefer
working with startups. We prefer working with the market, so that the
products that we use get also used by other parties, and that then
evolves the ecosystem in ways that we could not have imagined if we
were to come in heavy-handed and build it ourselves.
the people out there: don’t be afraid of the larger corporations. I
can only speak for Shell, of course, but we’re definitely open to
these kind of collaborations, and sometimes, it’s only a five-man
team that comes up with something brilliant. We definitely keep our
eyes and ears open to the whole market.
Some of the best things
that we’ve seen recently have been… it’s – again – that three-,
four-, five-person team, because here’s the thing: with digital, as
long as you have some talented people, and you have a vision, you can
build something – pretty much anything! If you have a vision, and
you have a strong team that can build it; it won’t take that long. A
lot of these platforms have been built over the course of six months
to a year, maybe a year and a half. And these are really robust
think one of the challenges that you mentioned is integrating with
legacy systems. I think this is going to be the biggest challenge for
any startup, building something; how do you build it to work with all
the different, weird legacy systems that companies are using? That’s
going to be a challenge, I know.
Oh definitely,
definitely. But not insurmountable.
No, definitely not! And
I think working with partners like yourself, that understand that
working together is beneficial for everyone, is the key. For the
businesses listening: it’s really vital that you… there’s some
companies that I’ve heard, they’ll invite startups in, they sit down
with them, they get as many ideas as possible, and then they build
their own team and try to cut the startup out of it. I’ve seen it
happen over and over again, and it’s not a good way of doing business
in the long run. You’re going to miss out on opportunities by doing
that. It may save some money in the short run, but by embracing these
startups and really working with them, I think it’s really beneficial
for everybody in the entire ecosystem as well. And it gives you the
first foot in the door to acquire them, should you find that this is
something you really want to have.
second thing you mentioned was simulating outcomes using digital
twins. Can you maybe unpack that a bit for people? What do you mean
by simulating outcomes using digital twins?
Well, let me start by
perhaps explaining what we mean with a “digital twin.” For us,
the definition of a digital twin is “a digital representation of
something physical,” and the reason why I say that “for us,” a
digital twin can be as small as a screw that is critical to a
process, but it can also be the size of a full-blown asset, or
offshore plant. It takes a lot of these digital twins to work
together to have logic behind them.
That then allows us to simulate certain scenarios. If you can imagine a situation where you want to perform a certain type of maintenance – whereby we have to take a compressor offline – a piece of equipment in the whole process. Then, the rest of the process usually continues, but everyone has to take a step up and run at 120-130 percent efficiency. Now, when you have these digital twins, you can perform the operation in – typically, we use virtual reality for these kinds of scenarios – whereby you simulate, “okay, if I take this compressor offline, then the other compressor is going to have to work at 130 percent. However, we also have quite detailed maintenance logs.” So, using very advanced analytics – and honestly, that’s not my field, so I won’t comment further on that – we are able to make probability statements around potential failures, what the probability of those failures are, and then, how do we visualize the impact of that failure? Does that result into a spill? Does that result into something more serious? All these variables allow us to be more prepared, before we do the actual operation. It’s all with the preparedness mindset where we use – again, usually virtual reality – but you can also use augmented reality to simulate, before you do something, “what am I going to expect in the future?”
Can you give an example?
Like, a specific example.
Well,
I mean, I just gave the
example of the compressors.
When you’re talking
about the compressors, though, what you mean? So, you watching a
compressor, and then… what would that look like, when when you guys
A good example is we had
to update a piece of equipment in an asset recently, and it needed to
be lifted out of the asset for the new one to be placed back in. What
we did is, we had the digital twin in virtual reality, and by the
power of being in a virtual reality where everything is rendered by
computer, we were able to lift that piece of component up way into
the sky, to detect whether it would hit any cross beams, or any other
pipes that were still in the vicinity. And we lifted it up, no
problem. It cleared everything. So that means that, if we were to use
a crane, we could just lift it straight up and we could go ahead.
the new piece of equipment actually had a protrusion. The base was
still the same, so the technical drawings, they looked to be okay,
but one of the protrusions was about a meter out to the side, which –
when we lowered it back in – we had a collision detection on the
digital twin over our environment, with the new digital twin of the
component. That allowed us to preemptively say, “okay, well, in
this case, we’re gonna have to reroute certain pipelines, in order to
get the new piece of equipment in.”
kind of simulation is very powerful, because if you were to
experience this in the real
reality, you would have manpower there; it would constitute a
potentially unsafe situation. And, when lowering the piece of
machine, you would actually detect, “we can’t do this. You have to
lift it back up.” Then you have to take action. So, these kind of
processes get delayed, and we are working on the clock, with very
specific permits there. So it’s a very costly affair. By having these
digital twins available, you save a lot of cost and time.
Okay. Let’s just take
that one specific example. What kind of cost savings do you think
that created? Just a ballpark, I mean; like, is it in the tens of
thousands or hundreds of thousands?
Yeah, I don’t like to go
into specifics on that one.
But it far outweighs the
costs of creating the digital twins, would be my… that’s what
People
think, “it’s expensive to create these scenarios,” or whatever,
but the consequence of not
creating the scenarios is exponentially more expensive.
Correct. I’ll subscribe
Perfect. And the last
one you talked about – and we’ve kind of dug into this a bit more –
is remote collaboration. Are there examples of how this has
benefited… have you seen a time where remote collaboration has
averted a downtime that would have costs millions of dollars to have
downtime? Because I would assume for every hour of downtime of an
oil-producing facility is going to be in the millions of dollars of
lost revenues – or lost production, anyway. Do you a specific
example of when remote collaboration saved the day?
I do not have, at the
top of my mind, an example where we save the day. But we had a
collaborative session last week, which is why it’s sort of at the
front-and-center of my mind, where we were gearing up for a workshop
to tackle a very difficult challenge.
got into the virtual space, and we did a little bit of an intro, and
one of the gentlemen who were joining us from… I believe he was
somewhere in the Americas region, he explained his background and
experience, and he mentioned offhand that it felt very similar to
something he experienced before. So – by the power of being in
these kind of virtual realities – we conjured up the documents,
which were stored on a server somewhere, and we found out it’s
actually quite similar and (what I was not looking forward to) what
was going to be a three-day workshop turned into a 30-minute
exercise, whereby we had a clear idea of how we were going to resolve
and tackle the serious challenge.
me, that’s also the power of this collaboration. It is utilizing the
knowledge that might not be readily available – might not even be
documented – but because this person was willing to join us in this
virtual space, and we were able to connect – and not not via chat,
but he was talking, we got to know each other – thereby, we linked
on something that’s – well, again – not saving the day, but
certainly saved a lot of time and money.
Well, yeah. I mean, if
you take – I don’t know how many people were there – but let’s
say five people, and you took something that would have taken three
days to 30 minutes; that’s an enormous savings for any company, and
just that one simple use case probably paid for all the technology
investment that you made in those VR headsets. It’s crazy. The
exponential savings and profitability from using these technologies
cannot be ignored anymore.
What is the most
important thing businesses can do to leverage the power of XR
technologies right now? What is the thing that you would say – for
a company that’s listening now that maybe hasn’t even experimented,
doesn’t know really anything [about XR] – what’s the first step?
What’s the most important thing that they can start to do?
This
is a little bit going back to university for me, but honestly, ask
the question “why” first. I’m very passionate about all the
digital realities, but you have to take it into account; is it worth
it for the business? Why are you doing it? And what is the
differentiator?
to reflect back on the example I gave around collaboration; could we
have done the same with a Skype call, rather than a virtual room that
we were in? Make sure that you tell the story of why
something is differentiating from a current capability. It might not
always be down to, “you’re gonna save X
amount,” but it is about the intangible aspects; you’re going to
save time, or this is allows you to operate safer. But in a lot of
cases, you really need to have the “why” clear, then the “how”
sort of follows up on that. And the “how” typically describes
which of the digital realities you would want to use, because the
situation will lean towards one or the other. Again: preparedness is
more virtual reality, when you’re out there in the field; augmented
reality is preferred because… well, if you’re out there, you’re not
going to put up a headset, and thereby lose your complete line of
sight by doing virtual reality when you’re out there.
focus on the “why,” and then the others – the “hows,” and
especially the “how much”
My last question for
you, Michael – and this is more of a personal one – what problem
in the world do you want to see solved using XR technologies?
For me, I’m going to
throw some jargon at this: it’s the idea of instant upskilling. Now,
I’m going to explain that one a little bit, “instant upskilling.”
What that means is – again, it’s a little bit of a jargon – but
when we started this conversation, I told you that right now, the
world is changing so fast, that you can learn everyday. You can learn
every hour,
and still something new will pop up. So there is a competency gap
that is growing.
these kind of extended realities – whatever you want to call it –
fill that void, where we have much more immediate access to expert
knowledge, allowing us to go much broader and – when necessary –
that you have it available to you. I don’t want to go too far into
the future. I don’t want to go Matrix style, where we inject it into
your brain, so you have it immediately. But think about augmented
reality. You’re out there, you have to change some wiring. You may
have done that specific wiring course three years ago – again, I’m
just painting a picture. Then you can conjure up the step-by-step
instructions of how you actually need to change that wire. Then, you
can continue doing what you need to be doing, so that… I want to
say “at your fingertips,” but that’s outdated – by talking to
your voice-driven device, you can have that information available to
you. That’s, for me, the big idea.
Amazing. I love the fact
that you [said, “have all the information at your fingertips…
wait a second – that is outdated!” That’s a crazy statement, that
having all the information “at your fingertips” is now an
That was the catch
that’s going to get people hooked on this podcast, for sure.
Michael, I want to thank you so much for your insights and your
input; it’s been very valuable. I’m sure listeners – if they want
to reach out to you – how can they find you?
The best ways to find me
on LinkedIn. Again, I am very eager to connect to the wider market.
really respect, Alan, what you’re trying to do. As a whole, it is
about raising awareness about the potential, and then following
through with it. So everyone, please reach out; follow us on
LinkedIn. We tend to share what we do there, via press releases and
blog posts, et cetera. So, happy to connect.