Episode 265: Ryan Fuller started questioning online reviews after two terrible stays at highly-rated vacation rentals. Both had 4.9-star ratings. After Ryan posted negative reviews, the people behind each listing independently offered him £200 to change his review to five stars.
"It was disappointing," he said, "that clearly many people took them up on this offer." On the flip side, the experience explained something that had been bothering him for a while: Places with thousands of positive reviews could still turn out to be terrible.
So Ryan, founder of Larrocos Labs and a serial entrepreneur, decided to investigate.
He built an AI system that has now analyzed hundreds of thousands of reviews covering roughly 22,000 restaurants. One example of what he found: In a focused analysis of 4,000 restaurants, about 70% showed at least one unusual review pattern. Some had sudden bursts of five-star reviews. He found one case in which roughly 100 reviews had been submitted within a two-minute period. Others featured suspiciously similar language, unusual concentrations of reviews mentioning particular employees, or reviewers whose histories raised questions about their credibility.
None of that proves a review is fake. As Ryan explains, the goal is not to establish absolute truth, but to determine how much confidence we should place in what we are seeing.
Ryan discusses what this means for consumers, platforms, and anyone relying on AI-generated information. He also explores an alternative: systems that show their work, identify potential sources of distortion, and tell us how confident they are rather than presenting uncertain conclusions as facts.
He sums it up well: "I think we're living in a world where understanding what is real or what is true is going to become one of the most difficult problems for humans to deal with."
Guest: Ryan Fuller, Founder, Larracos Labs
Host: Rob Markey, Partner, Bain & Company
Give us feedback: Customer Confidential Podcast Feedback
Send us a note: Contact Rob
Timestamped Topics
00:39 – How two 4.9-star vacation rentals led Ryan to question online reviews
03:11 – Why hours spent researching restaurants became a problem worth solving
04:49 – Using AI to automate the process of assessing restaurant reviews
05:26 – Building a database of 22,000 restaurants in six months with 300 lines of code
06:15 – Finding unusual review patterns across 70% of a 4,000-restaurant dataset
07:20 – Identifying sudden clusters of ~100 reviews for one restaurant
08:10 – Treating suspicious review activity as a question of confidence
08:40 – Using repeated staff names and customer incentives as review signals
09:12 – Detecting reviews that describe offers of money for positive ratings
11:05 – Assessing reviewer credibility through history, photos, and detail
12:11 – Building an AI product that checks authenticity and extracts review data
13:01 – Evaluating restaurant experiences across 36 distinct dimensions
Notable Quotes
18:00 "I think with AI, it's like rocket fuel, where things can be done at incredible scale and with high degrees of believability in ways that were never true before. I think we're living in a world where understanding what is real or what is true will become one of the most difficult problems for humans to deal with on even just everyday stuff like picking a restaurant."
26:00 "Platforms that start to be more transparent about sources and degree of confidence and potential risks for any information they're providing will become more popular."
28:00 "It's almost just like an arms race. There's no sudden advance in tech or approach that's gonna solve the problem. I'm just pontificating here; I don't have good answers, but I do think this is maybe the problem of our time."
33:00 "The devious thing about AI is that it sounds so credible; we're going through a transitional phase where we're not as well attuned to what the tells are when confidence isn't there."
37:00 "I always try to show the work. One thing we do is we have a letter grade that we assign each restaurant for overall quality. So, it can be an A, a B-plus, whatever. And the very first thing underneath the score, the overall letter, is why it got a B-plus."
Additional Resources
- Read Ryan Fuller's analysis of unusual patterns and manipulation in London restaurant reviews: https://www.linkedin.com/posts/ryantfuller_london-ai-share-7399118135099973633-Iquu/
- See Ryan Fuller's restaurant data visualizations and discussion of how platform algorithms shape restaurant visibility: https://www.linkedin.com/posts/ryantfuller_datavisualization-londonlife-urbananalytics-ugcPost-7404913381431844864-cM7l/
- Here's the article referenced about how a man created his own top-rated Dulwich restaurant: I Made My Shed the Top Restaurant on TripAdvisor: https://www.vice.com/en/article/i-made-my-shed-the-top-rated-restaurant-on-tripadvisor/