Dr. Lia DiBello is the Chief Science Officer of ACSILabs, Inc, which makes the FutureView Platform — a virtual reality training platform used by the US military and by certain large businesses to accelerate expertise. I last talked to Lia about her groundbreaking work explicating the mental model of business expertise, and in fact created this podcast to interview her. She’s back today to talk about her use of AI to accelerate expertise. 

Neil Sahota is the CEO of ACSILabs. Neil was part of the original IBM Watson team that won the Jeopardy challenge. After that challenge, he was, amongst things, the person responsible for IBM’s Watson ecosystem strategy, and an IBM Master Inventor (which is a designation for IBM employees who have made outstanding contributions to the IP creation process at IBM). Since leaving Big Blue, Neil has been Artificial Intelligence Advisor to the United Nations, published a book about AI in 2019, and started the UN’s AI For Good Initiative, which is currently hosted under the International Telecommunications Union.

In this episode, we take a human-first perspective on using and deploying AI in real world business and military contexts. We discuss Lia and Neil’s history with AI, talk about good and bad implementations of AI they’ve seen in real-world environments, and discuss what it means to get more folks to an expert-level at human-AI symbiosis. 

Shownotes

ASCI Labs — https://acsilabs.org/

ASCI Labs in the Navy STP marketplace — https://acsilabs.org/acsi-labs-has-met-the-requirements-to-be-listed-in-the-navy-stp-virtual-transition-marketplace/

AIQ (Artificial Intelligence Quotient): Helping People Get Smart about the Smart Machines They Are Using — https://medium.com/about-work/helping-people-get-smart-about-smart-machines-they-are-using-f9e0095846fe

AIQ: Artificial Intelligence Quotient — https://www.psychologytoday.com/us/blog/seeing-what-others-dont/202007/aiq-artificial-intelligence-quotient

Hoffman, R.R., Mueller, S.T., Klein, G., & Litman, J. (2018). "Metrics for Explainable AI: Challenges and Prospects." Technical Report, Explainable AI Program, DARPA, Washington, DC. — https://arxiv.org/abs/1812.04608

Timestamps

• (00:00) - Lia DeBello and Neil Sahota on Human-AI Symbiosis

Stan Slap is the founder and president of SLAP, an international consulting firm that helps organisations get maximum commitment from their manager, employee, and customer cultures. SLAP has worked with the who's-who of the Fortune 500, companies that you've heard of like Costco, Oracle, HP, Caterpillar, Microsoft, Amazon, General Electric, Intel, Viacom, Tesla, Glassdoor, Warner Music Group — and some companies that you haven't; all told, SLAP has worked with companies doing business in 44 countries. Stan is also the author of two books: Under The Hood and Bury My Heart in Conference Room B.

This podcast dives into Slap's ideas about understanding, managing and shaping effective employee cultures. His model for culture is the first coherent model that I've found that is useful and tested at scale; it is therefore more powerful than most other formulations of employee culture that you'll find.

SLAP Company — https://slapcompany.com/
Under The Hood — https://www.goodreads.com/book/show/22571574-under-the-hood
Commoncog summary of Under The Hood — https://commoncog.com/under-the-hood/
Bury My Heart at Conference Room B — https://www.goodreads.com/book/show/7810760-bury-my-heart-at-conference-room-b

• (00:00) - Introduction

Colin Bryar joined Amazon really early in its life and spent twelve years as part of Amazon's senior leadership team.

For two of those years he was 'Technical Assistant' to Jeff Bezos, as known as 'Jeff's shadow', during which he spent each day attending meetings, traveling with, and discussing business and life with Jeff. After Amazon, he and his family relocated to Singapore for two years where Colin served as Chief Operating Officer of e-commerce company RedMart, which was subsequently sold to Alibaba. Along with his ex-Amazonian colleague Bill Carr, Colin is co-author of Working Backwards, a book on an insider's look at how Amazon works. Bill and Colin are co-founders of Working Backwards LLC, where they coach executives at both large and early-stage companies on how to implement the management practices developed at Amazon.

This podcast is a really deep dive into the practice of the Amazon Weekly Business Review, which remains to this day one of Amazon's secret operating weapons, and a big part of what makes for a great operator.

Working Backwards, book: https://www.goodreads.com/en/book/show/53138083
Working Backwards https://www.workingbackwards.com/
Commoncog summary of Working Backwards: https://commoncog.com/working-backwards/

• (00:00) - Introduction

Eric Nehrlich is an executive coach, and was formerly the chief of staff on the Google Search Ads team. Before becoming chief of staff, Eric was part of the team that got Google's revenue forecasting down from an error rate of 10-20% to an error rate of around 0.5%. We open up with some wild stories of Google's early attempts at revenue forecasting, and then dig into how that forecasting success happened. Along the way, Eric explains how he, his boss, and his team developed a fingertip feel for variation in data.

We then switch gears to talk about Eric's coaching practice. We discuss the differences between mentorship and coaching, and talk about the tricky art of helping leaders grow their impact. Eric has a wealth of knowledge on coming up with small experiments in order to make personal growth easier. We talk about how he comes up with those experiments, and what some of those experiments look like in practice.

Finally, we close with a chat about Eric's upcoming book.

- Eric's Executive Coaching practice: https://www.toomanytrees.com/
- Eric's upcoming book is named You Have a Choice, and you may sign up for updates here: https://www.toomanytrees.com/book
- Eric's LinkedIn (where he posts insights every week): https://www.linkedin.com/in/nehrlich
- Eric's blog: https://www.nehrlich.com/blog/
- Eric's newsletter: https://www.nehrlich.com/blog/newsletter/ (which I highly recommend)
- Full list of coaching and leadership development resources that Eric recommends: https://www.toomanytrees.com/resources
- Surely You’re Joking Mr Feynman!: https://www.goodreads.com/book/show/35167685-surely-you-re-joking-mr-feynman
- Becoming Data Driven in Business Series: https://commoncog.com/becoming-data-driven-in-business/
- Be Good To Your Mentors: https://commoncog.com/be-good-to-your-mentors/
- Eric Nehrlich — Commitment, Competence, Structure: https://www.nehrlich.com/blog/2017/10/20/why-dont-we-change/
- James Clear — Atomic Habits (on Structure): https://www.goodreads.com/en/book/show/40121378
- Robert Kegan and Lisa Lahey — Immunity to Change: https://www.gse.harvard.edu/hgse100/story/changing-better
- Eric Nehrlich's post on Immunity to Change https://www.nehrlich.com/blog/2018/06/18/immunity-to-change-methodology/
- Claire Hughes Johnson — Scaling People: https://www.goodreads.com/book/show/63063173-scaling-people
- Eric's thoughts on Scaling People: https://www.linkedin.com/posts/nehrlich_google-stripe-leadership-activity-7044441138623127552-jhau
- Dan Martell — Buy Back Your Time: https://www.goodreads.com/en/book/show/60880804
- Terrence Real — Us Getting Past You and Me: https://www.goodreads.com/book/show/58502657-us
- Eric's post on Us Getting Past You and Me: https://www.nehrlich.com/blog/2023/02/21/us-getting-past-you-and-me-to-build-a-more-loving-relationship-by-terrence-real/

• (00:00) - Introduction

Lesley Sim coached the Singaporean Ultimate Women's World Championship team in 2020. We open with an introduction to the sport of Ultimate (sometimes known as frisbee), her experience coaching the women's team in late 2019, and then move on to her remarkable approach to pedagogical development and skill acceleration in the game of Ultimate.

Along the way, we talk about desirable and undesirable problems in training, playing to play vs playing to win, and how she used a training method originally designed for dolphins and dogs and adapted it to humans — with great success!

Lesley's Twitter — https://twitter.com/lesley_pizza
Lesley's Personal Site — https://lesley.pizza/
Newsletter Glue — https://newsletterglue.com/
Karen Pryor's Book Reading the Animal Mind: Clicker Training and What It Teaches Us About All Animals (on TAG Teach) — https://www.goodreads.com/en/book/show/2412884
How to Get Rich by Felix Dennis — https://www.goodreads.com/el/book/show/1837402.How_to_Get_Rich
Sticky.fm, Lesley's Podcast on Building Sticky Newsletters — https://sticky.fm/

• (00:00) - Introduction

David MacIver is most known for pushing the adoption and ergonomics of property testing in software with his testing library Hypothesis. Hypothesis is well regarded and widely used in the Python programming language community, and it introduced a handful of innovations that are now quite widespread in the practice of property testing. You’ll hear more about Hypothesis during the podcast, as we talk about what he’s learnt pushing the boundaries of a domain. Then, we shift gears to talk about his coaching practice. David specialises in helping programmers with self improvement, more effective learning, and developing soft skills, which many computer programmers are likely to struggle with, in ways that may limit their careers or their personal development.

David’s Substack — https://drmaciver.substack.com/
David’s Twitter — https://twitter.com/DRMacIver
Hypothesis — https://hypothesis.works/, docs: https://hypothesis.readthedocs.io/en/latest/
Ethics of False Negatives in Interviewing — https://www.drmaciver.com/2019/06/the-ethics-of-false-negatives-in-interviewing/
Life as an Anytime Algorithm — https://notebook.drmaciver.com/posts/2020-03-23-15:52.html
If You’re Stuck, Try Something Different (on chopsticks) — https://drmaciver.substack.com/p/lateral-moves
How To Do Hard Things — https://www.drmaciver.com/2019/05/how-to-do-hard-things/
Stargate Physics 101 (fanfiction) — https://archiveofourown.org/works/3673335
David on why people struggle with mathematics — https://twitter.com/drmaciver/status/1422208261349052420
How to Explain Anything to Anyone — https://www.drmaciver.com/2018/10/how-to-explain-anything-to-anyone/

0:00 Introduction
1:09 What Hypothesis Is
3:47 The Story of Hypothesis
6:43 Hypothesis’s Contribution to Property Testing
12:51 Exploring the Design Space for Hypothesis
17:24 When David Knew He Was On To Something with Hypothesis
20:35 From Hypothesis to Coaching
25:21 Emotional Reactions as Legacy Code
29:08 Why David’s Approach to Self Improvement Works for Programmers
31:15 Ethical Problems with Optimising False Positive in Hiring
37:44 Ways that Programmers Harm Themselves in Their Careers
43:28 What Non-Technical People Get Wrong when Dealing with Programmers
48:00 Applying Lessons Learnt from Hypothesis to Coaching
50:03 Rigour in Self Improvement Writing
56:30 Explaining Computers to Non Technical People
01:02:55 The Nature of Mathematical Expertise
01:11:32  David’s Practice with Teams and Organisations
01:14:23 Getting Better at Sprint Planning

Dr Lia DiBello is the CEO, President, and Director of Research of WTRI (Workplace Technology Research Inc), and Senior Scientist at Applied Cognitive Sciences Labs Inc. She is a cognitive scientist as well as a businessperson. In the late 2000s Dr DiBello discovered in an NSF-funded study that all great businesspeople share a common mental model of business, and that mental model can be used for all sorts of interesting things, including the assessment of business expertise, which she did — she was the principal inventor of something called the FutureView Profiler. 

In more recent years, Dr DiBello is more well known for her work on accelerated expertise — she published a book with a few other researchers in 2016 with that very title. She and her team have created something they call the Strategic Rehearsal, and this actually stemmed from her PhD work, where it was called the OpSim. What the Strategic Rehearsal allows WTRI to do is to accelerate the acquisition of business expertise in the businesses that they consult for, and her training interventions have been used in industries as diverse as biotech, pharma, manufacturing, financial services, and others.

- The Triad Mental Model of Business, paper: https://wtri.com/wp-content/uploads/2015/06/Informed-By-Knowledge-Chapter-12.pdf 
- A summary blog post of the triad mental model: https://commoncog.com/blog/business-mental-model/
- FutureView Platform — http://futureviewplatform.com or https://acsilabs.org/
- WTRI — https://wtri.com/
- List of publications by WTRI, which, if read chronologically, includes a full history of the Strategic Rehearsal — https://wtri.com/research/publications-by-wtri/
- Lia's LinkedIn — https://www.linkedin.com/in/liadibello/
- Lia's Twitter — https://twitter.com/LiaDiBello4
- Gary Klein on WTRI's training with Rio Tinto, on miner safety — https://www.psychologytoday.com/sg/blog/seeing-what-others-dont/201803/training-if-your-life-depended-it
- The Oxford Handbook of Expertise — https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780198795872.001.0001/oxfordhb-9780198795872
- Accelerated Expertise — https://www.amazon.com/Accelerated-Expertise-Research-Applications/dp/184872652X

00:00 Introduction
01:32 Lia’s Story
04:49 The Triad Mental Model of Business
10:38 Transfer of the Triad Between Business Domains
16:18 The FutureView Profiler Explained
20:42 How She Gets Her Clients To Accept Profiler Recommendations
25:21 The Midwest Foundry Story
40:14 Cognitive Agility
47:32 How Great Businesspeople Learn in the Real World
51:08 How This Has Affected Her Practice as a Businessperson
54:42 What Lia is Currently Working On
1:01:10 The Cognitive Science Behind the Strategic Rehearsal
1:09:47 Piaget and Vygotsky’s Theories of Expertise Development
1:17:50 The General Form of the Strategic Rehearsal
1:21:56 Non-Business Applications of the Strategic Rehearsal

The coffee company Lia mentioned: https://amorperfecto.com/
The Hollywood Bowl performance: https://www.hollywoodbowl.com/events/performances/1258/2021-08-27/carlos-vives-with-the-la-phil

Some words from Amor Perfecto to describe what they do:

This is a coffee movement launch. For almost two centuries coffee has been at the service of the intermediaries and not of the coffee growing countries. Amor Perfecto, in collaboration with amazing people like Carlos Vives, his wife, Claudia Elena Vasquez and their Tras La Perla Foundation, is going to push to rebalance the unfairly tilted value chain of coffee. Every one in the coffee world knows it exists. Equally important, the product of Roasted at Origin coffee by Amor Perfecto, harvested, roasted and set to you from the mountains of Colombia is a uniquely amazing coffee. It might be the best tasting coffee available today, because producing coffee where it is grown makes a much fresher, better tasting coffee. As Luis Fernando Velez says “have you ever seen the French exporting containers full of grapes to make champagne in Brooklyn?” Coffee should be no different.

A short trailer for Commonplace Expertise, the new podcast from Commonplace https://commoncog.com/blog/