System Prompt

Episode 34: The State of AI | Where’s the ROI?

The models keep getting better. So why is it still so hard for businesses to see a return?

In this off-the-cuff episode of System Prompt, Peter and Val catch up on frontier model releases, AI security, and the economics of running all this compute. The conversation gets practical when they ask what happens after a company buys AI licenses and someone has to figure out how the work actually gets done.

Val talks about what it takes to get employees to explain workflows they’ve spent years figuring out on their own. Peter shares how he used a local model on his ROG Flow Z13 to build an invoice tool, with code handling repeatable steps and the model interpreting the invoices.

They also get into hardware costs, consolidation among AI providers, and what open-weight models mean for people who want more control over where and how they run AI. Peter closes with a preview of his ticket-to-PR software factory and the decisions that went into building it.

IN THIS EPISODE

• Frontier model releases and AI-assisted security
• Enterprise AI ROI, licensing, and rollout friction
• Finding out how work actually gets done
• GPU costs and the economics of specialized cloud providers
• Running local models on a tablet
• General-purpose assistants and task-specific agents
• Open-weight models, competition, and access to compute
• A preview of Peter’s ticket-to-PR software factory

TOOLS & RESOURCES

• ASUS ROG Flow Z13 — Peter’s local AI setup
• Qwen 27B and Gemma 4 — local models discussed in the episode
• Hermes and Kokoro — local agent and text-to-speech tools

KEY TAKEAWAYS

START WITH THE WORKFLOW
Ask the people doing the work where the process slows down and what a good result looks like. Those answers give you something to build and measure.

USE THE MODEL WHERE IT CAN HELP
Code can handle repeatable steps and enforce checklists. A model can help interpret messy inputs, like invoices that come in different formats.

LOCAL MODELS GIVE BUILDERS ANOTHER OPTION
Running a model on hardware you own gives you more control over how you use it. Peter’s invoice tool is one example of what that looks like in practice.

CHAPTERS

00:00 Introducing the State of AI
00:37 Frontier Model Releases and AI Security
06:47 Why Enterprise AI ROI Is Hard to Measure
11:13 How Work Actually Gets Done
14:11 Checklists and Repeatable Workflows
19:57 AI Valuations and the Chip Market
33:20 GPU Cloud Economics and Local AI on a Tablet
35:56 Building Agents Around Specific Tasks
39:38 Where AI Stands Heading Into 2027
45:33 Coding as a Business Use Case
52:15 Local and Frontier Models in a Software Factory
55:47 Why Open-Weight Models Matter
59:44 What Smaller Models Can Build
1:01:33 Previewing the Ticket-to-PR Factory

Episode 33: AI Software Factories | The Future of Software Engineering?

What happens when AI coding agents move beyond individual tasks and become part of a repeatable engineering process?

In Episode 33 of System Prompt, Peter and Val explore AI software factories: systems designed to take engineering work from tickets through implementation, verification, review, and delivery.

Drawing on AI DevCon NYC, Peter shares lessons about verification-aware programming, engineering standards, and deterministic guardrails. They discuss how preconditions, postconditions, and invariants can make specific software properties mathematically verifiable, reducing reliance on AI-generated tests and model judgment alone.

But building a reliable factory introduces new challenges. How do you prevent agents from drifting away from specifications? Who makes architectural decisions? And can multiple agents coordinate without introducing more complexity than they solve?

Val raises the economic question: do software factories actually reduce development costs, or do inference, orchestration, and supervision erase the savings?

They close by examining greenfield development, the future of engineering teams, and how AI factories could reshape software development and SaaS businesses.

IN THIS EPISODE

• Coding agents vs. AI software factories
• Lessons from AI DevCon NYC
• Verification-aware programming and mathematical proofs
• Engineering constitutions and specifications
• Preventing agent drift and unreliable testing
• Multi-agent orchestration and independent review
• Ticket-to-PR workflows and human oversight
• Open-source models, inference costs, and ROI
• Greenfield development and future engineering teams

TOOLS & RESOURCES

AI DevCon NYC:
https://tessl.io/devcon

LemmaScript — Verification-aware programming:
https://docs.lemmascript.org/

Tessl — Agentic software engineering:
https://tessl.io/

KEY TAKEAWAYS

THE FACTORY IS MORE THAN THE AGENT
Reliable factories need repeatable workflows, engineering standards, authorization boundaries, and trustworthy verification.

SPECIFICATIONS DRIVE THE OUTCOME
Agents can implement misunderstood requirements. Clear specifications and human judgment remain essential.

VERIFICATION GOES BEYOND AI REVIEW
Formal contracts can mathematically verify specified properties. A second AI reviewer is useful but cannot replace mechanical verification.

THE HARNESS MATTERS
Tools, execution environments, context, and deterministic controls shape reliability as much as model capabilities.

ECONOMICS DETERMINE ADOPTION
Cheaper inference doesn't guarantee cheaper development. Cost per accepted task is what matters.

ENGINEERING ROLES WILL EVOLVE
Architecture, design, verification, and domain expertise may become increasingly important as implementation becomes automated.

CHAPTERS

00:00 Introducing AI Software Factories
01:52 What Is an AI Software Factory?
04:43 Engineering Standards and Workflows
07:28 Lessons from AI DevCon NYC
09:57 Mathematical Verification and Contracts
13:47 Verification-Aware Programming
15:43 More AI DevCon Insights
20:35 Engineering Challenges and Specifications
23:42 Quality, Security, and Coordination
27:00 Engineering Constitutions
28:16 Optimizing Prompts and Skills
30:56 Factory Workflows in Practice
32:46 Starting with Ticket-to-PR
35:03 The Software Factory Assembly Line
38:22 Models and Coding Harnesses
41:37 Do Factories Make Economic Sense?
44:59 Cost per PR and Business ROI
47:36 Model Selection and Cost per Task
48:00 Greenfield Development
50:53 The Future of Engineering and SaaS
55:59 AI Infrastructure and Competition
57:14 Final Thoughts

Episode 32: How to Pick the Right AI Model | DeepSeek vs Qwen Evals

You've seen how an AI agent is built, configured, and given a job. But how do you know if the model behind it is right for the work?

In Episode 32 of System Prompt, Peter and Val close out their agent series with a live demonstration of AI evaluations, or "evals." Instead of trusting public benchmarks or model reputation, they test the tasks that matter to a real workflow.

Using smevals, they compare DeepSeek V4.1 Flash running locally across two NVIDIA DGX Sparks against Qwen 3.8 27B. Their tests cover structured data extraction, instruction following, reasoning, and coding.

The smaller Qwen performs surprisingly well. But the real lesson comes from unexpected failures: DeepSeek adds a timezone to an otherwise correct answer, while Qwen initially returns no output on a task it passes when rerun. Peter and Val examine the outputs, question the grading rules, and troubleshoot live rather than presenting a polished happy path.

They also explore how harnesses handle model behavior, why larger models aren't always necessary, and how to connect performance, hardware costs, and business ROI. The goal isn't to choose the biggest model. It's to choose one that can reliably do your job at a sensible cost.

IN THIS EPISODE

• Why benchmarks don't always represent your workload
• How smevals uses tasks, runners, graders, and reports
• DeepSeek V4.1 Flash vs Qwen 3.8 27B
• Structured data, instruction following, reasoning, and coding tests
• Diagnosing grading mismatches and missing responses
• Why repeatability and failure severity matter
• Harness engineering and deterministic validation
• Local hardware costs, model selection, and business ROI

TOOLS & RESOURCES

smevals — Open-source evaluation framework:
https://github.com/prime-radiant-inc/smevals

KEY TAKEAWAYS

BENCHMARKS AREN'T YOUR WORKLOAD
Test against your real inputs, constraints, and expected outputs—not a leaderboard alone.

NOT ALL FAILURES ARE EQUAL
A formatting mismatch, empty response, or strict grader doesn't automatically prove a model can't do the task. Investigate the cause.

SMALLER CAN BE GOOD ENOUGH
For routine tasks, a 27B model may meet requirements that don't justify the cost of running a much larger model.

EVALUATE THE EVALUATIONS
Graders, runtimes, and model outputs can all vary. Rerun tests and examine failures before drawing conclusions.

THE HARNESS MATTERS
Prompts, tools, validation, and deterministic logic shape the reliability of the full system—not just the model.

ROI STARTS WITH THE WORKFLOW
Understand the task, its cost, and the value of automating it before deciding which model to deploy.

CHAPTERS

00:00 Closing the Agent Series: Picking a Model
01:19 Why Evals Matter
04:52 Introducing smevals
07:33 DeepSeek vs Qwen
09:25 Running the Evaluation Suite
12:04 Structured Data Results
14:23 Instruction-Following Failures
15:49 Investigating a Grading Mismatch
18:29 Failure Severity and Impact
19:48 Deterministic Output Handling
21:14 Harness Engineering
23:03 Models vs Agent Harnesses
25:19 Investigating Qwen's Missing Response
26:28 Repeated Evaluations
27:21 Rerunning Tests
28:57 Human Bias vs Data
29:41 Coding Results
32:01 When Smaller Is Good Enough
34:56 Productivity and ROI
37:55 Understanding Business Problems
40:09 Wrap-Up

You've seen what goes into an AI agent. Now what does it take to actually configure one and give it a job?

In Episode 31 of System Prompt, Peter and Val move from the architecture of Hermes Agent to a hands-on walkthrough of its Windows desktop interface. With Hermes already installed and connected to DeepSeek V4.1 Flash running locally across two NVIDIA DGX Sparks, they explore the settings that shape an agent's behavior, access, memory, and execution.

They work through model selection and fallbacks, chat personalities, workspace configuration, approvals, file checkpoints, context compression, providers, tool integrations, and plugins. Not every feature needs changing, and not every experiment works as expected. That's part of the demonstration: no prearranged happy path.

Then they give Hermes a real task: create a scheduled 8 a.m. AI news briefing, add sentiment analysis, inspect sources and output, and request a more polished PDF format. The conversation expands into what makes an automation useful—from daily operational reporting and ad analytics to finding RFPs and preparing proposals. The question isn't just what an agent can do. It's which steps in your existing workflow are worth handing off.

• Configuring Hermes through the Windows GUI
• Primary models, reasoning settings, fallbacks, and auxiliary models
• Personalities, attachments, interface settings, and workspaces
• Command approvals, secret redaction, file checkpoints, and browser access
• Persistent memory and context compression
• Agent step limits, subagents, and local versus cloud providers
• Search tools, custom integrations, MCP, skills, and plugins
• Testing web access with a locally hosted model
• Creating an automated AI news brief with sentiment analysis
• Applying agents to real business workflows

KEY TAKEAWAYS

CONFIGURATION IS PART OF AGENT ENGINEERING
Choosing a model is only the beginning. Workspace access, approval rules, memory, tools, and execution limits determine how an agent operates.

LOCAL DOESN'T MEAN UNRESTRICTED
A locally hosted model can use web tools and run workflows, but access to browsers, files, credentials, and commands still needs deliberate controls.

YOU DON'T NEED TO CHANGE EVERY SETTING
Some defaults are fine. Understand what each setting governs, then adjust it based on the task and the risk.

A WORKING DEMO ISN'T A FINISHED EVALUATION
Hermes creates a schedule and generates an initial briefing, but repeatability, source quality, and output consistency still need testing.

START WITH YOUR WORKFLOW, NOT THE TOOL
Look at what you retrieve, compare, update, email, and report each day. The valuable agent is the one that removes real work from that process.

CHAPTERS

00:00 From AI Infrastructure to a Working Agent
02:29 Models, Reasoning, and Fallbacks
03:30 Chat Personalities and Attachments
05:39 Appearance and Desktop Preferences
07:14 Workspaces, Approvals, and File Safety
08:44 Memory, Compression, and Voice
10:51 Advanced Settings and Agent Limits
12:00 Providers, Local Models, and Gateways
14:00 Tool Integrations, Plugins, and Skills
16:55 Can Hermes Build Its Own Tools?
18:24 Testing Web Access
19:20 What Should You Actually Do With an Agent?
21:12 Building a Scheduled AI News Brief
28:18 Adding Sentiment Analysis
30:58 Agent Personalities and Human Interaction
33:45 Reviewing the Briefing and Sources
37:20 Requesting a PDF Output
39:01 RFPs and Business Automation Opportunities
41:59 Finding Workflows Worth Automating
47:05 Wrap-Up and What's Next

READ THE FULL EPISODE PAGE: https://devmesh.tech/podcast/anatomy-of-an-ai-agent

An AI agent isn't just a model with tools. What actually happens between your prompt and a completed task, and what keeps the system from falling apart when something goes wrong?

In Episode 30 of System Prompt, Peter and Val take apart the Hermes Agent harness to examine the software engineering behind an AI agent. Building on their episodes about AI hardware and inference runtimes, they follow the agent loop from context assembly and model calls through tool execution, persistence, security, recovery, and observability. Hermes is the example, not the endorsement: the point is to show where probabilistic model decisions end and deterministic software takes over.

They explore why switching model providers involves more than changing an API endpoint, why a tool call is a request rather than an action, and how seemingly ordinary components like SQLite, file-based memory, and approval rules support a sophisticated system. Along the way, they discuss evaluation, concurrency, prompt caching, local execution risks, and the trade-offs that come with every architecture.

The agent loop may be simple. Making it durable, safe, and useful against real systems is the engineering challenge.

• The model versus the harness: who decides what
• The core agent loop and turn management
• Context construction, caching, and token costs
• Provider normalization across APIs and local models
• Tool registries, validation, approvals, and execution
• Persist-before-execute and crash recovery
• Iteration limits, errors, and fallback behavior
• Memory, skills, and what deserves to be remembered
• Security boundaries and dangerous-command detection
• SQLite, concurrency, observability, and evaluations
• Hermes' strengths, limitations, and engineering trade-offs

KEY TAKEAWAYS

THE MODEL REASONS. THE HARNESS GOVERNS.
The model proposes tools, arguments, and responses. The harness determines what can run, where it runs, what gets persisted, and which actions require approval.

A TOOL CALL IS NOT EXECUTION
The model can request an action, but deterministic code validates the request and enforces permissions before a tool actually does anything.

PROVIDER COMPATIBILITY IS MORE THAN AN ENDPOINT
Reasoning formats, message structures, streaming, and error handling differ. Normalizing those differences is substantial engineering work.

RECOVERY IS BUILT FROM ORDINARY SOFTWARE
Session leases, database writes, iteration controls, and error handling make a simple model/tool loop survivable when processes crash or requests fail.

SECURITY IS AN ENGINEERED BOUNDARY
Instructions alone aren't permission controls. Approval states, restricted execution environments, and configuration determine what an agent can actually do.

EVERY ARCHITECTURE HAS TRADE-OFFS
Hermes uses practical building blocks, but memory growth, provider complexity, local permissions, and SQLite concurrency still require deliberate choices.

CHAPTERS

00:00 Intro: Anatomy of an AI Agent
04:15 Why Is the Hermes Repository So Large?
06:33 Agent Loop, Turn State, and Context Assembly
10:47 Specialized Harnesses and Context Costs
12:35 Provider Abstraction and Tool Registries
17:46 Who Actually Executes a Tool Call?
19:41 Approvals, Failures, and Evaluations
23:48 Recovery, Memory, and Skills
27:35 Deterministic Security and Execution Boundaries
31:21 Concurrency, SQLite, and Observability
36:25 Testing, Prompt Injection, and Memory Decisions
40:43 What the Model Decides vs. What the Harness Decides
45:24 Hermes Limitations and Engineering Trade-Offs
50:32 Final Takeaways: What Makes an Agent a System
52:02 Wrap-Up and What's Next

READ THE FULL EPISODE PAGE: https://devmesh.tech/podcast/saas-vs-ai-part-2-revenge-of-the-swe

SaaS isn't dead. But what happens when the people who know a business best can build the software they need themselves?

In Episode 29 of System Prompt, Peter and Val return to SaaS vs. AI for an unscripted sequel: Revenge of the SWE. They debate whether agents become interfaces to existing SaaS products or replace some of those products entirely. From an accountant building tools with Claude Code and SQLite to small businesses recreating only the features they use, the question isn't simply whether AI can write code. It's who understands the process, validates the outcome, and maintains the result.

The conversation explores software engineering, domain expertise, deterministic workflows, harnesses, MCP, AI operations, local inference, and the challenges of organizational data and persistent memory. They also discuss frontier subscriptions versus local models, AI devices, and why AI-generated answers still need scrutiny.

This is a riff, not a prediction that every SaaS company disappears. The cost of code is changing. The cost of judgment is another story.

• SaaS as the engine behind agents, or something agents replace
• Domain experts building their own software
• Claude Code, SQLite, and small internal applications
• CRUD applications and custom alternatives
• Engineering beyond writing application code
• Deterministic workflows, validation, and TDD
• Harnesses, MCP tools, and specialized agents
• AI Ops: runtimes, hosting, access, and reliability
• Local inference economics and AI hardware
• Data quality, context, memory, and relevance
• Questioning AI-generated plans and answers

KEY TAKEAWAYS

THE AGENT MAY REPLACE THE INTERFACE OR THE APPLICATION
Val sees agents operating SaaS through integrations. Peter asks when a business can skip a product and build only what it needs. Both may happen, depending on the workflow.

DOMAIN EXPERTISE CHANGES THE EQUATION
An accountant or operations lead knows the requirements and exceptions. AI can lower the implementation barrier, but verifying the result still matters.

THE COST OF CODE HAS COLLAPSED. THE PRICE OF JUDGMENT HAS NOT.
Generating software is easier. Deciding what should exist, whether it works, who can access it, and how to maintain it still requires engineering judgment.

AI OPS IS STILL SOFTWARE ENGINEERING
Someone must own runtimes, harnesses, integrations, memory, evaluations, and infrastructure, whether inference is local or hosted.

MEMORY IS NOT JUST STORAGE
Saving information is one thing. Knowing what is relevant, outdated, or worth retrieving for a particular task is harder.

CHAPTERS

00:00 Intro and an Off-the-Cuff Episode
01:05 Anthropic, AI Sentience, and the Pope
03:20 SaaS vs. AI: The Debate Returns
08:17 When the Accountant Becomes the Builder
11:41 Does AI Change the Need for UI/UX?
14:00 Deterministic Paths and AI Costs
17:36 The Value of Domain Expertise
19:02 Local Models and the Future of Copilot
22:18 Harnesses and Specialized Agents
26:16 AI Devices and Distributed Inference
30:37 AI Ops, Data Quality, and Validation
34:03 AI Gadgets Versus Your Phone
37:37 Small Businesses and Bespoke Software
41:36 CRUD, Rails, and SaaS Alternatives
45:40 The Cost of Code vs. the Price of Judgment
48:41 Have We Solved AI Memory?
53:24 SaaS, Copilot, and the Cost of AI
58:28 Local Models and What's Next
1:01:01 Closing Thoughts

READ THE FULL EPISODE PAGE :https://devmesh.tech/podcast/youve-bought-ai-hardware-now-what

Buying AI hardware is only the beginning.

In Episode 28 of System Prompt, Peter and Val follow up on Before You Buy AI Hardware, Watch This by moving from planning into the actual hosting layer. Using GLM-5.3-Flash across two NVIDIA DGX Sparks with vLLM, they break down what the serving command is doing, which settings actually matter, and why hosting a model is very different from simply loading one. The conversation covers tensor parallelism, distributed execution, RoCE and NCCL networking, Docker, model mounts, KV cache, context length, concurrency, Mixture-of-Experts execution, chat templates, reasoning and tool-call parsers, DFlash speculative decoding, prefill, decode, and live serving metrics.Then they start the real cluster, bring the worker online before the head node, watch the model load across both Sparks, and run GLM-5.3-Flash through a coding workload. The live metrics show speculative decoding acceptance changing with the task, with generation throughput moving from the teens into the 30s and 40s as the workload becomes more predictable.

• What happens after you buy AI hardware

• Hosting GLM-5.3-Flash across two DGX Sparks

• vLLM and distributed inference

• Tensor parallelism across multiple devices

• RoCE, NCCL, and cluster communication

• Docker, model mounts, and runtime configuration

• Context windows and KV cache

• FP8 KV cache vs. FP16

• Concurrency and max active sequences

• Mixture-of-Experts execution

• Chat templates, reasoning parsers, and tool-call parsers

• DFlash speculative decoding

• Prefill, decode, and generation throughput

• Why real workloads have to be tested

QUICK CORRECTION

At one point I say the ConnectX-7 link is 200 gigabytes per second. I meant 200 gigabits per second, or 200 Gb/s. Yes, I know the difference. It was a live recording, I misspoke, and I am correcting it here.

KEY TAKEAWAYS

RUNNING A MODEL IS NOT THE SAME AS HOSTING ONE

Loading the weights is step one. Hosting adds networking, scheduling, memory management, concurrency, cache behavior, runtime compatibility, and observability.

THE MODEL IS ONLY ONE PART OF THE STACK

Runtime, quantization, KV cache, speculative decoding, context limits, scheduling, and hardware topology all affect system behavior.

YOU DO NOT NEED TO ENGINEER EVERY LAYER YOURSELF

Start with a known-good path, understand the major knobs, test it on your hardware, and go deeper only where the workload proves you need to.

SPECULATIVE DECODING ONLY HELPS WHEN THE DRAFT IS RIGHT

When DFlash acceptance is low, speculative work can become overhead. When the workload becomes more predictable, acceptance and generation throughput can rise significantly.

THE WORKLOAD DETERMINES THE CONFIGURATION

There is no universal best setup. Context, concurrency, precision, cache size, and speculative decoding all need to be tested against the actual task.

CHAPTERS

00:00 Intro and the Follow-Up

03:23 Two DGX Sparks, vLLM, and GLM-5.3-Flash

07:00 Breaking Down the Hosting Command

11:00 Model Mounts, DFlash, and Runtime Patches

14:12 NCCL, RoCE, and Distributed Networking

17:30 API Behavior, Tool Calls, Reasoning, and Chat Templates

21:40 The vLLM Knobs You Actually Tune

25:45 FP8 KV Cache and Memory Tradeoffs

27:30 DFlash and Speculative Decoding

31:38 Starting the Two-Node Cluster

41:51 Running GLM-5.3-Flash

45:56 Reading the Live Metrics

50:17 Why You Have to Test

54:22 Coding Workloads and Higher Draft Acceptance

59:37 What This Means for Businesses

1:04:01 Closing Thoughts

READ THE FULL EPISODE PAGE

https://devmesh.tech/podcast/before-you-buy-ai-hardware-watch-thisMost businesses do not need more AI hardware. They need a better understanding of the workload they are actually trying to run.

In Episode 27 of System Prompt, Peter and Val break down inference runtime, the layer that sits between the model and the hardware serving it, and why runtime decisions matter before a business starts buying GPUs, DGX systems, or other local AI infrastructure.

The conversation covers Ollama, llama.cpp, vLLM, SGLang, quantization, context windows, KV cache, prefill, decoding, batching, speculative decoding, and parallelism. But the bigger question is business capacity. How many users will actually be active at once? Which departments need the most inference? When does demand spike? What should run locally, and what should stay in the cloud?

Using examples from a 150-person company to enterprise GPU deployments, Peter and Val argue that infrastructure should be sized from real usage data, not assumptions. A business may not need 150 concurrent sessions just because it has 150 employees, and it may not need more hardware when better scheduling can solve the problem.

• What inference runtime actually is
• Ollama, llama.cpp, vLLM, and SGLang
• Why concurrency should be measured instead of guessed
• How quantization affects memory, performance, and quality
• Context windows, KV cache, prefill, and time to first token
• Why continuous batching matters
• Speculative decoding and runtime optimization
• Local devices vs. shared inference
• Why expensive hardware can sit mostly idle
• How businesses should pilot before scaling

KEY TAKEAWAYS

AN UNTESTED CONFIGURATION IS WINDOW DRESSING

If a model, runtime, quantization, and hardware configuration has not been tested against the real workflow, it is still an assumption. Production decisions should be based on measured behavior.

HEADCOUNT IS NOT CONCURRENCY

A 150-person company does not automatically need infrastructure for 150 simultaneous sessions. Real usage depends on departments, workflows, working hours, and demand patterns.

RUNTIME CHANGES HOW HARDWARE GETS USED

Batching, caching, scheduling, memory management, and parallelism all affect how efficiently requests turn into useful GPU work.

START SMALL AND COLLECT REAL DATA

Pilot with one department or workload, measure throughput and bottlenecks, then scale from evidence instead of buying for theoretical maximums.

IDLE CAPACITY IS STILL WASTED MONEY

A powerful GPU cluster does not create ROI by existing. If expensive accelerators sit idle most of the day, the infrastructure may be oversized or poorly scheduled.

THE BUSINESS PROBLEM COMES FIRST

Before choosing a model, runtime, or piece of hardware, define what the system needs to accomplish, how often it will be used, and what level of performance the workflow actually requires.

CHAPTERS

00:00 Intro and 20,000 Views
01:51 Why Businesses Are Exploring Local AI
03:25 What Inference Runtime Actually Is
06:51 Real Concurrency vs. Headcount
10:29 llama.cpp, GGUF, and Quantization
14:13 An Untested Configuration Is Window Dressing
15:22 Context and KV Cache
17:06 vLLM and Continuous Batching
18:28 Prefill, Decode, and Time to First Token
21:12 Prefix Caching and Runtime Optimization
25:17 Speculative Decoding
29:05 Engineering for Capacity
30:47 Local Devices vs. Shared Infrastructure
35:09 When Local Inference Makes Sense
39:53 Idle Hardware and Wasted Spend
48:10 Scheduling Inference by Department
49:40 The Full AI Infrastructure Stack

READ THE FULL EPISODE PAGE

https://devmesh.tech/podcast/your-ai-isnt-dumb-its-missing-context

Most people do not have an AI problem. They have a context problem.

In Episode 26 of System Prompt, Peter and Val break down why AI gives generic answers when it is handed generic information, and how a structured context document can turn the same model into something much more useful for real business work.

Using a cloud-services account example, they compare three approaches: a short paragraph with a vague question, the same paragraph with a better prompt, and a full client context profile built from account, usage, support, contract, and relationship data. The difference is not subtle. The model goes from plausible advice to specific, evidence-backed next steps tied to the actual customer.

The conversation also gets into where that context should come from, how teams can combine CRM data, tickets, emails, usage metrics, notes, and subject-matter expertise, why humans still need to verify the final document, and how this can replace long cross-functional information-gathering meetings with a much tighter workflow.

• Why AI gives generic answers even when the model is good

• What a business context document actually is

• Why better prompting helps — but only up to a point

• How CRM, ticketing, email, usage, and contract data fit together

• How structured context changes the quality of account recommendations

• Why every risk and recommendation should be tied to evidence

• How context documents can become a living source of truth

• Why account executives and technical teams still need to verify the data

• How prompts encode what your business actually cares about

• Why context engineering matters more than performative prompting

• How AI can reduce cross-functional meeting overhead

• Why many AI opportunities are process-shaped business problems

KEY TAKEAWAYS

BETTER MODELS DO NOT FIX MISSING CONTEXT

A strong model can still only reason from what you give it. If the information is thin, ambiguous, or disconnected, the output will usually be generic.

PROMPTING HELPS, BUT CONTEXT CHANGES THE GAME

A better prompt improves structure and discipline, but a full context document gives the model enough evidence to produce specific risks, opportunities, questions, and next steps.

THE CONTEXT DOCUMENT SHOULD BE A LIVING SOURCE OF TRUTH

Customer history, usage, support issues, contracts, upcoming changes, relationship details, and decisions should evolve with the account instead of being rebuilt from scratch every time.

HUMANS STILL OWN THE FACTS AND THE DECISIONS

AI can synthesize information, but the people closest to the account still need to confirm the numbers, correct drift, and decide what actually matters.

THE REAL ROI IS PROCESS COMPRESSION

The value is not that AI “takes over” the work. It can compress hours of gathering, reconciling, and summarizing information into a much shorter review process.

CHAPTERS

00:00 Why Context Matters

02:00 What a Context Document Is

07:59 The Generic Account Example

12:48 Better Prompt, Better Output

16:40 Building the Full Client Context Profile

21:01 What Changes With Complete Context

26:19 Replacing Long Cross-Team Meetings

29:37 Context + Prompt = Business Logic

32:12 Process Problems, Not Just Technical Problems

33:52 Guardrails, Verification, and Workflow Design

READ THE FULL EPISODE PAGE

https://devmesh.tech/podcast/the-reality-of-physical-ai-with-hiten-sonpal

Physical AI gets a lot more serious when a bad answer can move thousands of pounds of machinery. In Episode 25 of System Prompt, Peter and Val sit down with Hiten Sonpal, CEO of RISE Robotics and a robotics veteran whose career spans iRobot, defense robotics, autonomous lawn care, Electric Sheep, and heavy industrial machinery.

The conversation gets into what changes when probabilistic AI meets deterministic machines, why safety envelopes matter, and why Hiten believes the future of robotics is more likely to be specialized machines that amplify people than general-purpose humanoid replacements.

Hiten also breaks down RISE Robotics’ Beltdraulic technology, the SuperJammer robotic arm that lifted more than 7,000 pounds, where AI is actually useful in engineering today, and why impressive technology still does not become a business until customers want it and will pay for it.

• What changes when AI starts moving physical machines
• Human-in-the-loop robotics and deterministic safety systems
• Why AI should not have unrestricted control of heavy equipment
• RISE Robotics’ Beltdraulic alternative to hydraulics
• The SuperJammer arm and its 7,000+ pound world-record lift
• What Beltdraulics does better — and worse — than hydraulics
• How RISE uses AI for software, sourcing, and calculations
• Why robotics has a harder training-data problem than language models
• Desirability, viability, and feasibility as tests for real innovation
• Why Hiten is skeptical of humanoid-robot hype
• Specialized robots versus general-purpose human replacements
• What software engineers misunderstand about hardware
• What robotics engineers underestimate about modern AI
• Which jobs robotics is most likely to automate first

KEY TAKEAWAYS

PHYSICAL AI NEEDS DETERMINISTIC BOUNDARIES

A model can make high-level decisions, but safety-critical machinery still needs hard constraints. The system has to know when the answer is simply “no.”

SPECIALIZATION BEATS HUMAN REPLACEMENT

A robot does not need arms, legs, hands, and twenty degrees of freedom if the task can be solved with wheels and a purpose-built mechanism. Complexity has to earn its place.

HARDWARE CHANGES THE FAILURE MODEL

Software bugs can often be patched quickly. Hardware failures involve parts, lead times, prototypes, shipping, installation, and sometimes redesigning the machine itself.

NOVELTY IS NOT A BUSINESS

A technically impressive product still has to solve a problem customers care about, at a price they will pay, with economics that work.

AI IS ALREADY HELPING ENGINEERING — WITH CHECKS

RISE is using AI to accelerate software work, research components, and assist with calculations, but Hiten is clear that engineering outputs still need human verification.

CHAPTERS

00:00 Physical AI Gets Real
02:21 What Is Different About Robotics Now?
03:18 Failure, Safety, and Human Control
06:22 Should AI Directly Control Heavy Machinery?
09:03 Robots, Jobs, and Human Amplification
12:11 Why Robotics Needs a Hardware Revolution
13:05 The Origin of Beltdraulics
14:46 What Beltdraulics Does Worse Than Hydraulics
18:05 Building a 7,000-Pound-Lift Robotic Arm
20:42 How RISE Uses AI
24:11 The Robotics Training-Data Problem
27:01 Novel Technology vs. Real Business
28:52 The Problem With Humanoid Robots
36:06 The iRobot Lawn-Mower Story
39:58 Which Jobs Robotics Takes First
41:57 What AI People Misunderstand About Robotics
44:26 What Robotics Engineers Underestimate About AI
46:04 The Robotics Problem That Still Is Not Solved