Is artificial intelligence custom-made for legal tasks better than general AI tools like Google Gemini and ChatGPT? That is the topic of this episode featuring Legalbenchmarks.ai Founder Anna Guo. Anna is a former BigLaw lawyer who left the practice to become an entrepreneur and now focuses her energies on quantifying the utility of AI in the legal industry. Anna's initial anecdotal research for colleagues quickly revealed a strong community interest in a systematic approach to evaluating legal AI tools. This led to the creation of Legalbenchmarks.AI, dedicated to finding out where the promise of humans plus AI is truly better than humans alone or AI alone.

The core of the research involves measuring the "delta," or the extent to which AI can elevate human performance. To date, Legalbenchmarks.ai conducted two major studies: one on information extraction from legal sources and a second on contract review and redlining.

• Accuracy vs. Qualitative Usefulness: The highest-performing general-purpose AI tools (like Gemini) were often found to be more accurate and consistent. However, the legal-specific AI tools often received higher marks in qualitative usefulness and helpfulness, as they align more closely with existing legal workflows.

• Methodology: The testing goes beyond simple accuracy. It includes a three-part assessment: Reliability (objective accuracy and legal adequacy), Usability (qualitative metrics like helpfulness and coherence for tasks such as brainstorming), and Platform Workflow Support (integration, citation checks, and other features).

• Human-AI Performance: In the contract analysis study, AI tools matched or exceeded the human baseline for reliability in producing first drafts. Crucially, the data demonstrated that the common belief that "human plus AI will always outperform AI alone" was false; the top-performing AI tool alone still had a higher accuracy rate than the human-plus-AI combo.

• Risk Analysis: A significant finding was that legal AI tools were better at flagging material risks, such as compliance or unenforceability issues in high-risk scenarios, that human lawyers missed entirely. This suggests AI can act as a crucial safety net.

• Strengths Comparison: AI excels at brainstorming, challenging human bias, and performing mass-scale routine tasks (e.g., mass contract review for simple terms). Humans retain a significant edge in ingesting nuanced context and making commercially reasonable decisions that AI's instruction-following can sometimes lack.

• [0:00] – Introduction and background of Anna Guo and Legal Benchmarks AI.

• [4:30] – The impetus for starting systematic AI benchmarking.

• [6:00] – Explaining the concept of measuring the "delta" in performance.

• [9:00] – Detailed breakdown of the three-part AI assessment methodology.

• [15:00] – Discussion of the contrasting results: general LLM accuracy vs. legal AI qualitative value.

• [19:00] – Results on AI performance matching human reliability in contract drafting.

• [21:00] – Debunking the myth about Human + AI always outperforming AI alone.

• [23:00] – The finding that legal AI excels at surface material risks that lawyers miss.

• [27:00] – A SWOT analysis of when to use humans and when to use AI.

• [30:00] – Future roadmap for Legal Benchmarks AI research.

Justin Wales, Head of Legal for the Americas at Crypto.com, and author of The Crypto Legal Handbook visits the show to provide his unique perspective on pivoting from a career in Constitutional Law, including work on high-profile appellate cases like the Obergefell gay marriage decision, to becoming a trailblazer in crypto law and blockchain technology. 

He shares his serendipitous journey stemming from a law school article that launched his legal career and his subsequent deep dive into crypto, sparked by WikiLeaks accepting Bitcoin donations.

The discussion covers the evolution of his practice to one of the first successful crypto legal groups at a large law firm. Justin emphasizes the necessity for any lawyer in the space to use the technology and become a generalist to navigate the multi-jurisdictional and rapidly evolving industry. 

Finally, the conversation touches on the critical distinction between centralized finance (CeFi) and decentralized finance (DeFi), and his outlook that crypto's infrastructure will ultimately serve as the underpinning for future advancements like Artificial Intelligence (AI) and the Internet of Things (IoT).

Episode Highlights:

 * Justin's start in Constitutional Law, beginning with a law school paper on unconscionable cruise ship employment contracts that led to Supreme Court brief writing.

 * The shift to crypto law: WikiLeaks and Bitcoin's role in it introduced Justin to blockchain technology.

 * Building a pioneering crypto legal practice at a large firm in the early days (2013-2015).

 * Why the best in-house lawyers, especially in a cutting-edge field like crypto, need to be generalists, not hyper-specialized.

 * The inspiration and philosophy behind writing The Crypto Legal Handbook—creating an affordable, regularly updated, and candid resource for students and practitioners.

 * Advice for aspiring crypto lawyers today: the field is more mature and requires blending traditional financial regulatory expertise with an industry-wide approach.

 * The fundamental distinction between Centralized Finance (CeFi) and the legally complex, more innovative world of Decentralized Finance (DeFi).

 * Rule #1 for Crypto Lawyers: Why you must use the technology and the risks of lawyers who are frozen in time with their technical understanding.

 * High-level overview of US policy efforts to regulate crypto, including the GENIUS Act (Stablecoins) and the Clarity Act (Securities vs. Commodities regulation).

 * Justin's crystal ball: The long-term view that crypto infrastructure will eventually become an unseen layer beneath the rise of AI agents and IoT.

 * Book: The Crypto Legal Handbook by Justin Wales

 * Book: Read Write Own by Chris Dixon

 * JustinWales.com (for more information on the book and author)

Avaneesh Marwaha, the CEO of Litera, visits the show to discuss his journey from IP lawyer to becoming a legal tech CEO and investor. He discusses the motivations behind his career pivot, including his desire to be a decision-maker and his passion for the business side of law.

The conversation delves into the evolution of Litera, from its origins as a document-focused migration software company to its current role as a comprehensive legal tech ecosystem. Avaneesh highlights the company's strategic shift from acquiring to building new technologies. He also emphasizes the importance of Litera's strong partnership with Microsoft and the company's focus on enhancing law firms' productivity and client service.

Key Takeaways:

• Pivoting from Law to Business: Avaneesh's move from practicing IP law to an in-house role and eventually becoming a CEO was driven by his desire for a more active role in business decision-making. He saw lawyers as reactive to business decisions, and he wanted to be at the forefront of the action.

• The Value of a Law Degree in Business: Avaneesh shares his perspective on the value of a law degree for business professionals, highlighting the critical thinking and risk tolerance skills it provides. He notes that the corporate transaction course he took in law school was particularly beneficial, teaching him about corporate governance and fiduciary duties.

• Litera's Evolution: Avaneesh discusses how Litera has grown through strategic acquisitions and a recent focus on internal development. He explains how the company integrates various tools, like Kira and Foundation, to create a seamless ecosystem for lawyers within their daily workflows, primarily in Outlook and Word.

• The Impact of Generative AI: The discussion touches on the transformative impact of AI on the legal tech industry. Avaneesh explains how AI has enabled Litera to accelerate its product development and introduce features that proactively assist lawyers with tasks like document comparison and client relationship management.

• Build vs. Buy Strategy: Avaneesh outlines Litera's shift from a buy-heavy strategy to a more balanced build-and-invest approach. This change is largely due to the increased efficiency and speed of development enabled by AI tools.

• Future of Legal Tech: Avaneesh shares his vision for the future of Litera, which includes a focus on "maniacal client service" and helping law firms proactively grow their business. He believes that as automation handles routine tasks, lawyers will have more capacity for business development and client-centric work.

• Kira Systems - A company acquired by Litera specializing in AI-powered contract analysis.

• Foundation Software - A business acquired by Litera focused on knowledge management.

• Doxly - A document formatting and repair tool.

A conversation with Ben Chiriboga, Chief Growth Officer at legal CRM company Nexl and host of the This Legal Life podcast. Ben shares his journey from a litigator to a legal tech innovator. He recounts how an early e-discovery tool using natural language processing completed in six hours what had taken him six months, leading to an epiphany about the power of legal technology.

The conversation delves into the legal industry's historical resistance to technology, prioritizing billable hours over efficiency, and how Ben leveraged his legal experience to transition into a new role in tech. Ben discusses the challenges of building a legal tech company and the critical distinction between a general CRM and one designed specifically for law firms. The episode concludes with a thought-provoking discussion on the impact of AI on the legal profession, focusing on what aspects of law are truly "fundamental" and what new roles and jobs AI will create.

• The Power of Tech: Legal tech tools can drastically improve efficiency, as demonstrated by an e-discovery tool that performed six months of work in just six hours.

• Transitioning Careers: A background in law can be a valuable asset in the legal tech sector, but success requires adapting to new skills like sales, marketing, and growth strategy.

• The "Why" Behind a Legal CRM: General CRMs are built for sales teams with built-in incentive structures, whereas a legal-specific CRM, like Nexle, is designed to passively collect and enrich data without requiring lawyers to manually input information.

• The Fundamentals of Law: The human element—relationship building, counseling, and strategic advisory—is the core function of a lawyer that AI cannot replicate.

• The Future of Legal Jobs: AI's true impact will not be in better-drafted motions, but in the creation of new "legal adjacent" roles and business models that merge legal expertise with technology and data.

We welcome back legal marketing expert Gyi Tsakalakis, President of AttorneySync and host of the Lunch Hour Legal Marketing podcast. Five years after Gyi's first appearance on Technically Legal, he discusses the dramatic shifts in digital marketing for law firms, driven largely by the rise of artificial intelligence.

Gyi highlights how AI is changing the landscape for lawyers, from automating back-office functions and data analysis to influencing content creation and search engine optimization (SEO). He explains why law firms must diversify their marketing channels beyond just Google to adapt to the new era of conversational search and AI overviews. Despite these technological advances, Gyi emphasizes that the human connection, including a simple handwritten note, remains the most powerful tool in an attorney's arsenal. He argues that as technology automates the mundane, it forces lawyers to focus on the relationships and empathy that truly differentiate their practice.

Key Takeaways:

• Diversify Marketing Channels: Law firms must move beyond a "one-trick pony" approach with platforms like Google and diversify their client acquisition strategies to include social media, email, and local events.

• The Problem with Client Intake: Gyi discusses the persistent issue of lawyers not responding to inquiries and how AI can help automate certain parts of the intake process, though the human touch is still crucial for building trust.

• Do What the Bots Cannot: Attorneys should leverage AI for tasks that can be automated (e.g., scheduling, data analysis) to free up time to focus on complex legal work and building meaningful client relationships.

• AI for Data Analysis: A powerful, overlooked use case for AI in a law firm is analyzing internal data, such as call transcripts and financial records, to identify blind spots and inform marketing strategies.

• The Future of SEO and Search: Gyi predicts that search will become more conversational and influenced by AI Overviews and that ad-based models will be integrated into all major AI platforms.

This episode features a conversation with Andrea Muttoni, President of Story Protocol. The discussion explores how blockchain technology is being used as a foundational infrastructure for intellectual property (IP), aiming to simplify and modernize the process of registration, licensing, and monetization for creators and IP owners. Muttoni details his journey from a bedroom music producer to a product manager at Amazon and eventually to a leader in the blockchain industry, driven by a passion for the intersection of technology and creativity.

The conversation delves into the core problems Story Protocol aims to solve, particularly the complexities of copyright, fair use, and attribution in the age of AI-generated content. Muttoni introduces key concepts like the Programmable IP License (PIL), a customizable and on-chain license that makes IP rights more transparent and accessible. He also provides an overview of Poseidon, a new initiative that leverages the Story Protocol to create a marketplace for IP-safe, real-world data needed to train AI models, ensuring that data creators and owners are fairly compensated.

• Andrea's Background: From a bedroom hip-hop producer to a product manager at Amazon working on Kindle and Alexa, Muttoni shares his path to the world of crypto and blockchain.

• What is Story Protocol?: A deep dive into Story Protocol's mission to create an open IP blockchain. It's a way to register, license, and monetize IP in a more liquid and accessible way.

• The Programmable IP License (PIL): Explanation of how this universal, customizable, and on-chain license simplifies IP transactions and makes it easier for others to legally use a creator's work.

• Introducing Poseidon: Details on the new platform built on the Story Protocol to address the "data gold rush" for AI. Poseidon allows for the licensing of real-world, IP-cleared data to AI companies, ensuring fair compensation for contributors.

• Promoting Adoption: Discussion on how Story Protocol uses a web3 incentive model to encourage widespread adoption and create a network effect for its IP infrastructure.

• Why Story Protocol?: Muttoni explains what makes their platform uniquely suited for IP rights compared to other blockchains, emphasizing that the network was built from the ground up specifically for this use case.

• Story Protocol: story.foundation

• Story Protocol IP Portal: portal.story.foundation

• Poseidon: psdn.ai

• Story Protocol Developer Docs: docs.story.foundation

• WIPO Report: A report on intangible assets, mentioned as a key indicator of the value of intellectual property.

Crypto lawyer Nelson Rosario returns to the show to discuss the evolving landscape of decentralized AI and his role as General Counsel for the the Advanced AI Society (formerly known as the Decentralized AI Society), an association focused on engineering best practices, advocating for policy, fostering community, and enabling capital formation for decentralized AI startups

Nelson discusses the changing definition of decentralization, highlighting its origins in early crypto's focus on resilience and censorship resistance, and its current intersection with artificial intelligence. He explains the core reasons driving decentralized AI efforts: the potential for single service providers to control our access to AI models, computing power, and the data that underlies it all.

The conversation explores how decentralized AI aims to offer an alternative to large tech companies like Facebook, Microsoft, and OpenAI,  But Nelson also points out the challenges of competing with resource rich organizations and suggests that decentralized AI might complement existing systems by "chopping it up" into distributed components. 

The discussion also touches upon current players in the decentralized AI space, notably Bitensor and HyperCycle, and highlights the role of decentralized storage solutions like Filecoin and Arweave.

Nelson concludes by reflecting on the potentially profound societal impact of AI, likening it to other "tectonic technological revolutions" in human history. Drawing from the "Butlerian Jihad" concept in Frank Herbert's Dune, he cautions against blindly trusting AI and underscores the need for humans to remain intentional in their use of technology to prevent a decline in critical thinking and creativity.

Key Takeaways:

• Decentralization in AI aims to prevent single points of control over models, compute, and data, promoting censorship resistance.

• Decentralized AI faces challenges in competing with large, centralized tech companies due to their immense resources, but can serve as a complementary set of systems.

• DePIN (Decentralized Physical Infrastructure) is crucial for the robust existence of decentralized AI applications.

• Industry associations like the Advanced AI Society play a vital role in setting standards, advocating for policy, building community, and attracting capital for decentralized AI projects.

• AI represents a transformative technological shift, and intentional human engagement is critical to navigate its societal implications.

In this episode, Dave Siegfried discusses the company he heads, Official AI -- a company focused on helping its customers create and verify authentic AI content. Siegfried, a "recovering accountant" with a background in tech and media licensing, discusses his journey from co-founding Audiosocket, a music licensing platform, to addressing challenges in generative AI.

The conversation highlights the evolution of intellectual property protection in the digital age. Siegfried explains how Official AI empowers individuals and brands to create AI-generated images and videos with consent, credit, and compensation for the use of their name, image, likeness, and voice. He details Official AI's "vault technology," which creates a "blueprint" of a person's likeness from a small set of images, enabling authorized creation of synthetic media.

The discussion also delves into provenance technology and the Coalition for Content Provenance and Authenticity (C2PA) open-source standard, which Official AI leverages to track content origin and changes, ensuring authenticity. Siegfried touches on the challenges of verifying content on social media platforms and the future role of blockchain technology in creating immutable records of consent. The episode concludes with insights into Official AI's business model, customer base (including athletes and historic estates), and future roadmap.

Key Takeaways:

• Generative AI presents significant challenges and opportunities for intellectual property rights, particularly concerning name, image, likeness, and voice.

• Official AI aims to proactively address these challenges by providing a platform for consent-based AI content creation.

• Provenance technology, including C2PA, is crucial for verifying the authenticity and authorized use of AI-generated content.

• The legal landscape surrounding AI and publicity rights is rapidly evolving, requiring adaptable technological solutions.

• Official AI Website: Visit theofficial.ai to learn more about their platform and services.

• Contact Dave Siegfried: Dave can be reached at [email protected].

• Content Authenticity Initiative (C2PA): Explore the open-source standard for content provenance and authenticity at c2pa.org.

Editing and Production: Grant Blackstock

Theme Music: Home Base (Instrumental Version) by TA2MI

This episode is a conversation with Spellbook co-founder Scott Stevenson about intersection of technology and creativity. Spellbook is a AI contract co-pilot for transactional lawyers that plugs into Microsoft Word. Despite founding a legal technology company, Scott is not a lawyer but is computer engineer by training.

As a kid Scott was into video games and in fourth grade he talked his parents into getting him a computer because he wanted to figure out how to create them. By middle school he was building websites and eventually landed an internship at Electronic Arts.

Scott is also interested in electronic music and he launched his first start up, Mune, with a music professor, to create a whole new musical instrument that combined the power of digital music with an acoustic instrument. It was during his time at Mune that Scott started to think about building a legal tech company.

After he got his first legal bill he figured there might be a more efficient and less expensive way to do legal work. So he and lawyer buddy founded Rally, a document automation and templating engine for law firms which later begat Spellbook, but he funny thing about Spellbook, it was originally conceived as a marketing idea to generate leads for Rally.

Mune (Scott's Digital Instrument Startup)

Editing and Production: Grant Blackstock

Theme Music: Home Base (Instrumental Version) by TA2MI

A conversation with Kelly Twigger, an E-Discovery veteran and founder of ESI Attorneys and Minerva26. Kelly shares her journey into E-Discovery, stemming from the first-ever ABA event on the topic in the early 2000s, and the rapid evolution of case law and technology that led her to develop E-Discovery Assistant which has now been rebranded Minerva 26.

The discussion delves into how Minerva 26, a strategic command center for E-Discovery, curates case law, rules, checklists, and offers on-demand education. Kelly explains the reasoning behind rebranding from "E-Discovery Assistant" to "Minerva 26," emphasizing the shift from data tools to knowledge and strategic insight. The conversation also explores the current challenges legal professionals face in adopting technology, the impact of AI on the legal industry, and the need for updated federal rules of civil procedure to address technological advancements.

• Proactive E-Discovery: The importance of engaging with E-Discovery issues early in a case to save time and money, and to maximize the utility of electronically stored information (ESI).
• Technology's Transformative Power: ESI, unlike paper, offers immense opportunities to leverage technology for more efficient and effective legal analysis and fact-finding.
• Overcoming Fear and Expense: The primary barriers to technology adoption in the legal field are fear and perceived expense, which can be overcome through education and strategic planning.
• AI's Role in Legal: AI is not replacing lawyers but enhancing their capabilities by improving efficiency, accuracy, and providing valuable insights.
• The Need for Rule Updates: The rapidly changing technological landscape necessitates updates to the Federal Rules of Civil Procedure to provide clarity and consistency, especially concerning issues like control over personal devices and the use of advanced E-Discovery tools like Technology Assisted Review (TAR).

Editing and Production: Grant Blackstock

Theme Music: Home Base (Instrumental Version) by TA2MI

• Want to keep up to date about new episodes? Technically Legal Update List.

• Want to learn more about Percipient (percipient.co)?

• Follow Chad on Linkedin: Chad Main | LinkedIn

• Follow the podcast on LinkedIn: Technically Legal | LinkedIn

• Follow the podcast on Instagram: Technically Legal | Instagram

• Follow the podcast on X: Technically Legal | X