The AI Sidebar

No AI model has cracked 60% on real-world contract review, even as the legal tech industry keeps announcing new launches. In this month’s recap, host Phillip Vaden breaks down an independent study from Ivo and MicroOne that tested seven models on 350 attorney-scored contract tasks, with Grok leading at 58.3%, just ahead of Claude Opus 5.5 and Kimi K3. He also looks at how Anthropic is expanding Claude for Legal with a gallery of ready-to-use artifacts, including tools built by Goodwin, Foley and Lardner, and PwC, plus more than 60 legal data connectors added since May. Other stories include Conceal, a local desktop app from Sprint Four that masks client information before it reaches ChatGPT or Claude at a price aimed at solo and small firms, and Davis Wright Tremaine’s peer learning pilot to get about 70 attorneys using AI well. Vaden also covers Google and Meta in-house counsel reporting that AI-drafted patent applications are often rated as good as or better than human drafts in blind tests, and closes with the American Legal Technology Awards, where Open Laws took the AI category.

Resources:

Transcript:

Phillip Vaden

Hey everyone. Welcome to a special tech update edition of the AI Sidebar, where we talk all things AI and legal tech with an eye towards dehyping a very hyped space. I'm Phillip Vaden, Partner Manager at InfoTrack. Let's dive in. A lot happened over the past month. A major move from Anthropic into legal-specific tooling, a promising new security tool, an independent benchmark that should humble anyone who thinks AI has fully solved contract review, and a few stories about how firms are actually getting their people to use this stuff. Let's get into it. So Anthropic goes deeper into legal.

We'll start with what may be the single most important story of the month. Anthropic, the company behind Claude, significantly expanded its legal offering. Back in May, Anthropic launched Claude for Legal, which came with more than 20 model context protocol connectors and a dozen practice area plugins. Late last month, they built on with a gallery of ready-to-use artifacts specifically for legal work. Think interactive tools and dashboards, not just chat responses. Anthropic built 6 of these themselves, but here's the part I find more interesting. 4 more of these tools were built by outside partners and shipped inside Claude. Goodwin built an AI governance benchmark.

Hardy built an intake tool. Foley and Lardner built an obligations checker, and PwC built a subpoena response tool, with more reportedly coming from Legora and Oric. On top of that, Anthropic is funding a batch of access to justice tools, things like an immigration papers reader and a record sealing screener built with legal aid organizations, which I think is a great thing. And since May, Anthropic says it's added more than 60 legal data connectors covering things like Bloomberg Law, Filevine, and Legora. The takeaway here is that Anthropic isn't just offering a general purpose model that happens to be useful for lawyers anymore. It's actively building out a legal ecosystem with named practice groups, implementation partners like Accenture, Deloitte, and KPMG, and direct relationships with legal legal tech vendors. That's a meaningfully different posture from where they were just six months ago. All right, next up we're going to talk about a privacy shield for solo and small firms.

So here's a story that I think matters a lot for anyone running a smaller practice. A company called Sprint Four launched a desktop app called Conceal that runs entirely on your own device and masks sensitive client information before. Before it ever reaches tools like ChatGPT or Claude. The pitch is simple: a lightweight AI model runs locally, it detects client names, addresses, and case details in whatever you're typing or whatever document you're working with, and either redacts that information permanently or it swaps it for placeholder labels that you can restore later from a local mapping file. Nothing about the actual document content leaves your machine. It works inside ChatGPT and Claude desktop apps directly, and it's priced at under $30 a device per month, which is an incredibly attractive price point. Sprit4 is explicitly targeting solo and small firms here, pointing to ABA data that, that roughly 7 in 10 lawyers work in firms of 10 attorneys or fewer, meaning most lawyers carry the exact same confidentiality obligations as 1,000-lawyer firms but without any of the enterprise security infrastructure to back it Up. That is a great and promising thing, and I can see that catching on very, very quickly.

So now a bit of a reality check, because it's easy to get caught up in all the hype around all these new tools. A joint study from Ivo and the data lab MicroOne tested 7 major AI models against 350 real-world contract review tasks scored against criteria defined by actual practicing attorneys. The result? Every single model scored below 60%. So Grok came out on top at 58.3%. That was just ahead of Claude's Opus 5.5 at 58.1% and KIMI K3 at 57.1%. While KIMI came in third there, it was faster by 13% and significantly cheaper on a token cost basis. But the point isn't just flagging these to pick on any one model. It's to say that for all the funding rounds and flashy product launches we just talked about, independent attorney-graded benchmarks still show these tools topping out well under under the pass mark you'd want before trusting them unsupervised on real contract work.

It's worth keeping in your back pocket the next time a vendor pitch promises full automation. All right, so now let's talk about teaching lawyers how to actually use AI. So on the adoption side, Davis Wright Tremaine announced a pilot program this month aimed at the human side of getting lawyers to actually use AI well, which, let's be honest, Is usually the harder part of the problem than the technology itself. So working with a company called Keep Company, DWT is running about 70 attorneys through a peer learning program. Small cohorts of eight to ten people, six virtual sessions each, led by outside coaches trained specifically in change management. The sessions focus on things like approaching AI with an inventive mindset, and maybe most interestingly, working through what should stay human in legal practice. Even as the tools keep getting better and better. DWT says it'll track usage data throughout and use the findings to shape firm-wide AI training going forward.

It's a good example of a firm treating this as a culture and training challenge and not just a procurement decision. I look forward to monitoring that and see how that goes. I think it's a great thing to sort of figure out how to get the adoption up and not just simply sign a contract and start paying for the products. Okay, so let's talk about AI versus humans on patent drafting. All right, now this one is very interesting. So at a recent conference, current and former in-house patent counsel from Google and Meta discussed how far AI has come specifically at drafting patent applications. In blind tests, experienced patent attorneys are now often rating AI-drafted applications as better than ones written by their human colleagues. Patent drafting has long been considered one of the more technically demanding corners of legal writing.

So here's Hearing in-house counsel at 2 of the largest tech companies in the world say the gap is closing, or in some cases reversing, says something real about where specialized legal drafting is heading beyond the routine contract review and research task where AI has already made some pretty significant inroads. So we'll wrap up on a nice note. The American Legal Technology Awards held their 7th annual gala in Boston this past month, honoring innovators across categories like access to justice, courts, and enterprise technology. The dedicated artificial intelligence category went to Open Laws for its structured database of US law, with additional honors going to legal, legal aid organizations using AI tools to expand access to justice for people who can't otherwise afford a lawyer. After a month full of billion-dollar valuations and acquisition sprees, it's nice to have a reminder that a meaningful slice of the AI work happening in the industry is aimed squarely at closing the justice gap rather than just winning market share. Well, that will do it for this special edition of the AI Sidebar. Full sourcing for everything we've covered in the show is in the notes below. I'm Phillip Vaden, partner manager at InfoTrack US.

Thank you for joining me for this episode of the AI Sidebar. Take care, everyone. Have a good one.

Law firms don’t have to build their businesses on someone else’s AI infrastructure. Dan Schnurbusch, a probate and estate planning attorney at Schnurbusch Law in the St. Louis area and founder of the Legal Tech Collective, a Facebook and Discord community of lawyers, legal ops professionals, and builders, shares what he’s seeing in real time. He explains what an agent harness is, why he runs the open-source Hermes desktop agent, and why he believes solos and small firms have a major advantage over larger firms still working through governance and security at scale. He also makes the case that open-weights models like DeepSeek and GLM are keeping prices in check for Western providers, why he expects capable small models to run on phones within a year, and why he doesn’t think subsidized AI subscriptions are headed for a price explosion. Schnurbusch walks through his own daily workflow of supervising agent sessions on probate matters, why he sees AI as a force multiplier for most lawyers, and why he lands as a mostly optimist who stays cognizant of what could be lost along the way.

The line between legal information and legal advice is disappearing, and AI is the reason. Damien Riehl, a lawyer and technologist at Clio who chairs the Minnesota State Bar Association’s AI Committee, explains why unauthorized practice of law concerns look different in the age of ChatGPT and shares three ways legal aid providers can stay on the right side of the line: sticking to legal information, using empathy, and offering statistics. He explains why transactional lawyers have embraced AI faster than litigators, why the more than 1,000 cases involving hallucinated citations are really a bad lawyering problem, and why a tool grounded in current case law can hallucinate less than a junior associate. Riehl also looks at how firms like Kirkland are trying to keep institutional knowledge from walking out the door with lateral hires, why Jevons paradox suggests cheaper legal work will create more of it, and how his own neighbor dispute over a fence showed what an experienced litigator still brings that AI cannot. He predicts flat fees and subscriptions will take a much larger share of legal billing, points to AAA’s bot arbitration as a glimpse of where access to justice could go for the 92% of legal needs that go unmet, and closes by rejecting the optimist versus pessimist framing in favor of deciding what kind of future lawyers want to build.

Recorded at Concorda's booth in Startup Alley at ILTACON, this conversation with founders Sam Oh and Ke Ma digs into the gap they saw in legal AI tools built mostly for transactional work while litigation teams were left without platforms that understood local court rules, judges' standing orders, and the formatting requirements tied to filing and service. Oh and Ma, both licensed attorneys, walk through how they built out that database starting with PACER and federal courts before expanding state by state to national coverage, and why their approach to firm adoption leans on embedding directly with attorneys and mapping the technology to a firm's actual workflows rather than dropping in an off-the-shelf tool. The conversation closes with their shared warning about the industry's biggest blind spot: no matter how advanced the models get, hallucination risk never fully disappears, which is why Oh argues that quick, visual source verification matters more than trusting a tool at face value.

Recorded on the show floor at ILTACON, this conversation with Thomas Bueler-Faudree, founder of August, tackles the question he says he hears most from law firms: does adopting AI mean undercutting your own billable hours. Bueler-Faudree argues that legal work is inelastic, so firms that lower costs with AI tend to win more work rather than less, pointing to case studies of firms doubling revenue and scaling from 100 to 400 lawyers after adopting AI systems. He also breaks down why Australia and the UK have outpaced the US in AI adoption due to their different ownership rules, why long pilot programs with a mix of skeptics and early adopters matter more than rushing a rollout, and why he pushes firms to measure ROI in top-line revenue rather than time saved. The conversation closes with Bueler-Faudree's read on where the market is headed next, including the shift from firms using AI individually to firms restructuring around it at a firm-wide level.

Most conversations about AI and the law focus on attorneys, but the courts themselves are moving fast in ways that get far less attention. Matt Davis, Vice President of Product Development at Tyler Technologies with over 20 years of experience in public sector courts and justice software, breaks down the three major categories where courts are actually deploying AI today: internal chatbots training up new staff, automation of repetitive clerk tasks like e-file review, and tools that help judges absorb case information faster without replacing their decision-making. Davis walks through real examples, including Orange County's internal chatbot, Alaska's AI tool for self-represented litigants navigating probate, and the Arizona Supreme Court's use of AI avatars to explain rulings in plain language for the public, and the conversation also covers the data custodian responsibilities courts carry, why judges and courts remain wary of AI touching outcomes even as they embrace it for analysis, and how Tyler's acquisition of FTR is shaping a new AI-powered experience for judges working with digital transcripts, before Davis and host Phillip Vaden close things out with a wide-ranging discussion on humanoid robots, AI safety, and where they each land on the optimist-versus-pessimist spectrum.

A packed couple of weeks in legal tech coming out of ILTACON 2026 in Nashville, TN. Host Phillip Vaden covers Google's launch of Gemini Enterprise for Legal, Thomson Reuters' new proprietary legal LLM, LexisNexis's shift toward an orchestration layer, and iManage's new platform and partnerships. He closes with a California bill that could restrict attorneys from delegating legal work to generative AI, and what it could mean for the legal AI market. 

Recorded live at ILTACON 2026, this conversation with Dino Eliopulos, SVP of Innovation at Fulcrum GT, digs into why AI shouldn't be trusted to grade its own work. Eliopulos explains the case for building external systems that audit AI outputs rather than letting the AI evaluate itself, and how that traceability matters when a firm has to defend a disputed bill or explain how an AI-corrected time entry came to be. He also talks through the discipline behind Fulcrum's internal startup incubator, how unstructured legal work product like briefs and narratives is finally becoming accessible to AI, and what he tells firms asking how to measure ROI on their AI investments without having a business case in place from the start.

Paralegals have been prompt engineers long before the term existed, and that background is shaping how they're approaching generative AI today. Kristine Custodio Suero, California Paralegal of the Year and paralegal educator at the University of San Diego School of Law, and Kelli Radnothy, a 20-plus year litigation paralegal with the Nevada Attorney General's office and chief ideas officer of KLR Paralegal, join the show to talk through how they're using AI tools in practice and in the classroom. They walk through their advice for paralegals just starting out, why starting low stakes and low risk matters before touching client data, and how they trained their own LLMs, nicknames included, to build confidence with the technology. The conversation covers the guardrails needed for confidentiality and ethics, the shift from task-based work to strategic work that AI is pushing paralegals toward, and why billing models may need to evolve alongside it. Suero and Radnothy also share how government and firm settings shape what tools they're allowed to use, their takes on Claude versus ChatGPT, and why they both land firmly on the techno-optimist side of the debate.

Generative AI usage among attorneys jumped from 31% to 69% in a single year, but most law firms still haven't caught up with formal policies or training. Attorney and legal tech columnist Nicole Black, who has covered the intersection of law and technology for two decades, breaks down what's driving that gap and what it means for firms trying to keep pace. She explains why solo and small firms tend to adopt new technology faster than their mid-size and large counterparts, and why immigration attorneys in particular are using generative AI at a much higher rate than other practice areas. Black also discusses the slow erosion of the billable hour, why in-house counsel and insurance defense attorneys are among the most likely to push firms toward flat rate pricing, and what the current wave of legal tech consolidation means for the tools attorneys rely on day to day. The conversation turns to the rise of vibe coding in the legal industry, including attorneys building their own case management software from scratch, and Black shares her own experience using AI tools to navigate a New York liquor license application for a winery she and her husband are launching. She closes with her honest take on where she lands between AI optimism and AI pessimism, and why she thinks regulation may already be behind the curve.