LLM.co

By LLM.coBusiness Technology Management

LLM.co episodes

November 01Why Your AI Server's Firmware Is the New Security Frontier
Most on-prem AI security discussions end at the network perimeter — firewalls, SSO, prompt logging, segmented VLANs. But a pair of high-profile research disclosures in late 2024 revealed a class of attack that bypasses all of it. This episode of LLM.co examines why AI server firmware has become the new security frontier, what the Eclypsium and HD Moore findings actually demonstrated, and how infrastructure leaders should respond when acquiring hardware to run large language models on-premises.
The conversation covers the full attack surface hiding inside a modern AI server chassis — and what disciplined hardening looks like at each layer:
The chassis is a stack of computers: The baseboard management controller (BMC) runs its own OS, boots before the hypervisor, and has full host control — yet sits largely outside conventional endpoint security tooling.

DPUs and GPU firmware are live attack surfaces: A data processing unit with PCIe fabric access has a direct path to GPU memory; compromise at the VBIOS or UEFI level survives a full OS reinstall.

Model weights are the prize: Weights loaded into high-bandwidth memory at inference time generally cannot be encrypted while live — exactly where the Eclypsium researchers demonstrated silent tampering, with traditional integrity checks returning clean results throughout.

NIST SP 800-239 sets the baseline: The draft publication calls explicitly for hardware root of trust, confidential computing, and security-by-design across the data center lifecycle — including what happens to hardware after it leaves your facility.

Hardening is not exotic: Isolating the management plane on a physically separate network, enforcing firmware signing across BMC, GPU VBIOS, and DPU, enabling IOMMU to restrict PCIe device visibility, and storing weight-encryption keys in a dedicated HSM are the core moves — and any serious vendor should be able to speak to them before you sign a purchase order.

On-prem ownership cuts both ways: Running LLMs on your own production AI hardware means you can inspect and control firmware in ways a shared cloud tenancy never permits — but it also means the hardening responsibility lands squarely on your team.
The episode also addresses decommissioning: returning an accelerator to a vendor with weights still resident is a data exposure event, not a logistics footnote. Procurement teams are encouraged to get data-destruction terms in writing before any hardware acquisition — a consideration worth raising early when using a vendor-neutral hardware sourcing process to evaluate AI server options.
For more on the intersection of production AI systems and model improvement, the episode From Production Traces to Fine-Tuned Model: The Langfuse Training Loop is a natural companion listen.
hardware.co
6 min
October 25GPU Server Configs for Open-Weight LLMs: Match the Build to the Model
Choosing the right GPU server for an open-weight LLM isn't as simple as headcount or budget — it comes down to model size, interconnect bandwidth, KV cache math, and concurrency requirements that only become visible once hardware is on a purchase order. This episode walks through the detailed guidance in the GPU server configuration guide for open-weight LLMs published on hardware.co, translating the key trade-offs into practical decision-making across four distinct build tiers.
Here's what the episode covers:
Start with the model, not the chassis. Four parameter bands — 7–14B, 24–34B, 70B dense, and large MoE — each carry specific VRAM, RAM, and interconnect requirements that should drive the server spec, not the other way around.

VRAM and KV cache are the two numbers that matter most. Weights run ~2 bytes per parameter at BF16 and drop to ~0.5 bytes at INT4; KV cache scales with context length and concurrency and can easily exceed model weight storage at long-context serving loads.

Single-GPU workstations are a legitimate production tier. An 80 GB card can serve a 70B model at 4-bit, a 32B at 8-bit, or any 7–14B at full precision — making it the right landing spot for most pilots and small-team deployments. Teams planning this kind of build can estimate their exact VRAM needs with the AI Hardware Estimator.

Two-GPU builds live or die by their interconnect. Whether to run dual independent replicas, tensor-parallel a larger model, or dedicate one card to embeddings and reranking depends almost entirely on whether the cards share NVLink or are limited to PCIe peer-to-peer — a gap that can represent an order-of-magnitude difference in cross-GPU bandwidth.

Four GPUs is the honest tier for 70B-class dense models and medium MoE. Mixtral's MoE architecture means you pay for ~46.7B parameters in VRAM but compute at roughly 13B-dense speeds — a useful planning target that explains why four-GPU configs suit MoE workloads so well.

Eight-GPU nodes unlock frontier MoE and serious fine-tuning — but DeepSeek-V3's 671B total parameters require multi-node InfiniBand or RoCE setups, a parallel filesystem, and 8–12 kW of power capacity even before accounting for KV cache. For teams scaling to this level, a purpose-built GPU cluster for large models and high concurrency is worth evaluating.
The episode makes clear that GPU count is the headline spec, but interconnect, system RAM, and NVMe storage are often where deployments succeed or fail. For more on the cost and security trade-offs of running models on your own hardware versus a managed service, check out the related episode Private LLM vs ChatGPT Enterprise: Cost, Security, and Performance.
hardware.co
6 min
October 21Private LLM vs ChatGPT Enterprise: Cost, Security, and Performance
Choosing between a private LLM deployment and ChatGPT Enterprise is one of the most consequential AI infrastructure decisions a company can make right now — and most teams approach it with the answer already baked in. This episode works through the decision analytically, drawing on the full cost, security, and performance comparison published by LLM.co, and gives enterprise buyers a sharper framework for evaluating both paths on their actual merits.
Here's what the episode covers:
What each option actually is: ChatGPT Enterprise is a fully managed SaaS product — you never touch the model weights or inference stack. A private LLM means an open-weight model (Llama 3.3 70B, Qwen 2.5 72B, Mixtral, etc.) running on infrastructure your organization controls — on-premises, inside a VPC, or fully air-gapped.

Where the cost crossover really is: ChatGPT Enterprise runs roughly $60 per seat per month with a minimum commitment, putting a 200-seat deployment at around $410K over three years. Private infrastructure carries higher upfront hardware costs but the per-user economics invert sharply at scale — by 2,000 seats, private can land well under half the SaaS cost over the same period. The decision is a utilization problem, not a sticker-price problem.

What ChatGPT Enterprise's security controls actually cover — and where they stop: SOC 2 Type II, ISO 27001, SAML SSO, SCIM, admin-controlled retention, and a BAA for healthcare are all real and substantive. But prompts still leave your network, traverse OpenAI's infrastructure, and are stored in their tenancy. Enterprise Key Management protects stored ciphertext, not the inference path itself.

Why regulated industries often can't accept that data flow: Law firms operating under privilege, hospitals bound by HIPAA, banks under GLBA, and defense contractors under CMMC 2.0 face data-residency and audit requirements that a private deployment satisfies by design — prompts stay in the VPC or on the rack, logs land in your own SIEM, and keys live in your own HSM. Teams with strict compliance obligations may also want to explore what a private LLM setup looks like for security-focused organizations.

The real performance gap (and where it disappears): Frontier closed models like GPT-4o and GPT-5-class systems do lead on broad reasoning benchmarks. But on the narrow tasks most enterprise users actually run — document extraction, retrieval-grounded Q&A, code generation, classification — the delta is often small to zero. Fine-tuned open-weight models on a domain-specific corpus frequently outperform general-purpose frontier models on those tasks.

Throughput and latency control: An eight-H100 node running vLLM can serve Llama 3.1 70B at roughly 2,800 tokens per second in aggregate — enough for hundreds of concurrent users at interactive speeds. With ChatGPT Enterprise, latency depends on OpenAI's load and routing, neither of which you can observe or influence. Organizations exploring fully self-hosted options can also look at LLM appliance deployments as a fast path to private infrastructure.
For more from the show, check out Build Your Private AI Server in the Right Order — Starting with the Workload, which pairs well with this episode's infrastructure discussion.
LLM.co
hardware.co
6 min
October 18Build Your Private AI Server in the Right Order — Starting with the Workload
Ordering a GPU chassis before anyone has mapped the actual workload is one of the most expensive mistakes in enterprise AI infrastructure — and it happens constantly. This episode of LLM.co breaks down the correct sequence for building a private AI server, drawing directly from this step-by-step private AI server build guide published by hardware.co. Each decision in the chain flows from the one before it: workload first, then model, then VRAM, then GPU class, then chassis, then power and cooling, then software.
The episode walks that full sequence, covering:
Workload definition before any spec sheet: concurrent users, context window length, and whether the system needs inference only or training — a distinction that can shift memory requirements by an order of magnitude and dramatically change the hardware cost.

Picking GPUs from the model, not the brochure: a 70-billion-parameter model like Llama 3.1 needs roughly 140–178 GB at full FP16 precision, but drops to 35–45 GB at INT4 quantization — a difference that determines whether you need two GPUs or eight. The AI Hardware Estimator for GPUs and VRAM is a practical tool for running this math before committing to a configuration.

Quantization trade-offs in practice: INT8 halves memory with minimal quality loss; INT4 cuts memory by 75% and can nearly double inference throughput — but should always be validated against your own evaluation data before a rack is ordered.

Interconnect bandwidth and its real-world impact: once a model spans more than one GPU, NVLink versus PCIe is the difference between usable latency and a model that stalls mid-response.

Networking and storage planning before the rack arrives: the three storage tiers (fast NVMe for weights and cache, warm for embeddings, bulk for source data), and why undersizing NVMe is the most common cause of cold-start delays on day one.

Software, governance, and power as deliberate choices: continuous batching (with throughput improvements of up to 23×), identity integration, prompt logging, EU AI Act compliance timelines, and the reality that a modern AI rack can draw 40–142 kW — far beyond what most enterprise facilities were designed to handle.
For teams operating in regulated or data-sensitive environments, the AI Governance Software for audit and redaction covers the logging and compliance layer discussed in the episode. Listeners planning a healthcare deployment may also want to revisit Building a HIPAA-Compliant Private LLM: The Spec You Actually Need, which covers the compliance specification process in depth.
hardware.co
6 min
October 14Building a HIPAA-Compliant Private LLM: The Spec You Actually Need
With healthcare data breaches averaging $7.42 million in 2025 and regulators tightening the screws on electronic PHI, deploying an LLM in a clinical environment is no longer a product decision — it's a compliance architecture problem. This episode of LLM.co cuts past the "should you do it?" debate and gets into the actual build, drawing directly from the full HIPAA private LLM specification that technical leads need to get it right the first time.
The episode walks through the layered decisions that determine whether a private LLM deployment survives a regulatory review or becomes an incident report, covering:
Business Associate Agreements at every layer: HIPAA liability follows the data, meaning your vector store, embedding model, log pipeline, key manager, and identity provider all need BAAs — or must be infrastructure you own outright.

Why open-weight, locally hosted models win compliance reviews: When the model provider never touches inference traffic, the compliance perimeter shrinks to auditable infrastructure — the core advantage behind the LLM appliance approach.

Model selection and the hallucination trade-off: Llama 3 70B via vLLM is the practical on-premises default; medical fine-tunes like MedGemma and BioMistral show promise but can hallucinate more than strong general-purpose base models on clinical tasks.

The 2025 HIPAA Security Rule rewrite: HHS's proposed rule would make encryption at rest/in transit and MFA mandatory rather than "addressable," with a 240-day compliance window after finalization — and FIPS 140-3 validated modules, not just AES-256, are the line that matters in an audit.

Audit logging as a required control: Prompt text, retrieved document IDs, model version, user identity, completion hashes, and timestamps must flow to tamper-evident, append-only storage — and be reviewed on a schedule, not only after an incident.

Applying the minimum necessary standard to RAG: Retrieval systems must filter by user role, not just query text; de-identification of training and analytics corpora must happen upstream, before fine-tuning, because models can memorize and reproduce sensitive content.
The episode also addresses the administrative and physical safeguard layer — where most Office for Civil Rights findings actually originate — and why workforce training now has to include prompt hygiene alongside traditional access controls. Organizations exploring HIPAA-compliant AI for healthcare will find this a practical framework for scoping what a compliant build actually requires. For teams thinking about infrastructure scale, the related episode How Many GPUs Do You Actually Need? A Practical Sizing Guide covers the GPU concurrency and context-length math in depth.
LLM.co
hardware.co
6 min
October 11How Many GPUs Do You Actually Need? A Practical Sizing Guide
GPU sizing for on-premises AI rarely starts with the right question. Vendors lead with reference architectures built for worst-case scenarios, and organizations end up over-provisioned before a single workload requirement has been defined. This episode of LLM.co works through a practical, math-first approach to figuring out how much GPU memory and how many cards a real inference workload actually needs — drawing on the practical GPU sizing guide from hardware.co as its foundation.
The episode walks through each layer of the sizing calculation in sequence, from model weights to concurrency planning, covering:
Weight memory as the baseline: why the two-gigabytes-per-billion-parameter rule in FP16 is your starting point, and how quantization (FP8, INT4) can cut that figure dramatically before you ever consider adding hardware.

The VRAM floor vs. ceiling distinction: weights only occupy 60–70% of usable VRAM in a well-planned deployment — the rest goes to KV cache, framework overhead, and serving headroom.

Why context length is the most common sizing mistake: at long contexts and high concurrency, the KV cache can dwarf model weights entirely, and three software-side choices (grouped-query attention, capped context, paged attention) address it before any hardware purchase.

Separating throughput from latency: aggregate tokens-per-second and per-user time-to-first-token are different constraints that drive different decisions — and the episode shows how to multiply through both to arrive at a defensible GPU count.

Memory bandwidth over TFLOPS: why LLM inference is memory-bandwidth-bound, and what that means for reading GPU spec sheets.

Sizing to the second-worst normal hour, not the theoretical peak: reference architectures assume your worst day is every day; real workloads — whether bursty document review or steady clinical summarization — are not shaped that way.
The five numbers to write down before requesting any quote — target model and quantization, p95 token lengths, peak concurrent users, acceptable token rate, and time-to-first-token — are the same inputs that inform decisions about comparing AI server configurations across different deployment sizes. Teams exploring on-premises deployment may also want to look at fine-tuning and model training services, where GPU memory requirements shift significantly from inference workloads. For more on keeping production LLM deployments stable during infrastructure changes, the episode How to Replace a Public LLM API Without Breaking Production covers the operational side of that transition.
hardware.co
6 min
October 07How to Replace a Public LLM API Without Breaking Production
Switching from a public LLM API to a private deployment sounds straightforward — until Monday morning standup reveals broken outputs and a hasty rollback. This episode of LLM.co breaks down why most first attempts fail and how a structured 30/60/90 migration plan changes that outcome. Drawing from this deep-dive guide on replacing a public LLM API safely, the discussion covers everything from the initial inventory to the final cutover decision, with practical sequencing that engineering and compliance teams can actually follow.
Here's what the episode covers:
Start with an inventory, not a sprint. Ninety days of API logs reveal the full surface area — chat completions, embeddings, transcription, vector stores, fine-tuned checkpoints — each requiring its own migration path and success criteria.

Rank workloads by sensitivity and volume, not convenience. High-sensitivity, high-volume tasks — patient record summarization, loan file extraction, controlled source code review — move first. That forty-dollar-a-month ad hoc job waits for wave two. Teams in regulated industries such as healthcare have particular incentive to prioritize these routes early.

The gateway comes before any model. A routing layer (LiteLLM Proxy being the common choice) that speaks the OpenAI wire format on the client side lets every application keep calling the same endpoint while backends change underneath — the strangler fig pattern applied to inference.

Shadow mode is where migrations are won or lost. Duplicating production traffic to the candidate model — without users ever seeing the response — builds the evidence base for cutover and exposes prompt transfer failures before they become incidents. Prompts tuned for a frontier model routinely drop below 70% accuracy on open-weight alternatives without rework; budget two engineering weeks per material workload.

Three parity thresholds before you look at results. Task accuracy on a held-out eval set, 95th-percentile end-to-end latency, and refusal/format drift must all be defined in advance — not negotiated after the numbers come in.

Canary before blue-green, and keep a thin public-API connection. A ten-to-fifty percent canary ramp under real production load reveals failure modes that shadow mode can't surface. And not everything should move: frontier reasoning tasks, bursty workloads, and experimental features often justify retaining a metered public-provider connection alongside a custom private LLM deployment.
The episode also covers why the gateway doubles as an audit boundary — logging model ID, token count, latency, and routing decisions to your SIEM in a format that satisfies SOC 2, HIPAA, and EU AI Act obligations the moment you flip the private backend on.
For more on infrastructure costs associated with private deployments, check out the related episode What a Private GPU Cluster for 200 Users Actually Costs. More from LLM.co on the full migration topic is available at the source article linked above.
LLM.co
hardware.co
6 min
October 04What a Private GPU Cluster for 200 Users Actually Costs
Most budget conversations about private AI infrastructure start and end with GPU sticker prices — and that's exactly where the planning goes wrong. This episode of LLM.co delivers a rigorous, component-by-component total cost of ownership analysis for a private large language model cluster serving 200 knowledge workers at a regulated organization, drawing on this detailed GPU cluster cost breakdown current to late 2026. Whether you're at a mid-sized bank, a hospital network, or a defense contractor, the math is more tractable — and more illuminating — than most teams expect.
The episode walks through every major budget line over a three-year horizon, explaining not just what things cost but why each variable moves the total the way it does:
Concurrency shapes hardware, not headcount. At 10–15% peak concurrency, 200 users typically require just two eight-GPU nodes — not a data center — and the right GPU choice (H100 vs. L40S) depends entirely on model size and inference workload.

Capital expenditure for a two-node H100 build runs roughly $820K–$1.4M over three years, covering GPUs, high-speed networking, storage for weights and vector indexes, and rack infrastructure; an L40S equivalent comes in at about half.

Power and cooling are a hidden budget line. Two nodes drawing ~22 kW of IT load, factoring in a realistic facility PUE, translate to $280K–$580K in colocation and power costs alone over three years.

Software is free to download but expensive to run. Open-weight runtimes cost nothing upfront, but commercial support across 16 GPUs adds $210K–$300K over the life of the system — a line item many initial estimates omit entirely.

Staffing is the largest non-hardware cost. A realistic 1.5–2 FTE model (platform engineer plus partial ML, security, and data engineering support) runs $400K–$700K over three years, dwarfing networking and storage budgets combined. For organizations exploring broader deployment, custom LLM deployment services can shift some of that operational burden off internal teams.

All-in, the three-year TCO for a production-grade H100 cluster with high availability, software support, and staffing lands at $1.9M–$3.2M; an L40S build for smaller models runs $1.2M–$1.9M. For teams in regulated sectors like healthcare or finance — where private LLM infrastructure for financial services carries additional compliance requirements — understanding these ranges before procurement is essential.
The episode also covers a practical headroom rule: clusters running at 90% utilization leave no room for the fine-tuning jobs that inevitably surface next quarter, making 60% steady-state utilization the smarter design target. If you're preparing for procurement conversations, the related episode What to Put in a Private LLM RFP Before You Sign is the logical next listen.
LLM.co
hardware.co
5 min
September 30What to Put in a Private LLM RFP Before You Sign
Signing a private LLM vendor agreement without the right RFP language is how companies end up locked into data topologies, logging schemas, and termination terms they never agreed to — they simply never required anything different. This episode of LLM.co walks procurement leaders, CISOs, and heads of AI through the six contractual line items that actually determine whether a private LLM deal is enforceable — drawing directly from the private LLM RFP requirements checklist published on LLM.co.
Here's what the episode covers:
Data and weight ownership: Five questions every RFP must force in writing — who owns training data, fine-tuned weights, and derived telemetry — with any hedge treated as disqualifying. Vendors who retain rights to derived weights will leverage that claim at renewal.

Air-gap and update mechanics: "Air-gapped deployment" is a marketing phrase until a vendor submits a network diagram covering every ingress and egress across install, inference, and patching. The episode details what a genuine offline deployment commitment looks like in writing, relevant to teams evaluating offline AI agents and air-gapped LLMs.

Audit log schemas: Logging "all activity" means nothing without a field-level specification. The episode walks through required fields at the request, agent, and administrative levels — all in structured JSON, streamed to the buyer's SIEM, with retention controlled by the buyer, not the vendor.

Red-team acceptance tests: Final payment should be gated on adversarial testing performed on the buyer's infrastructure against the buyer's data — anchored to the OWASP Top 10 for LLM Applications 2025 and the NIST Generative AI Profile (AI 600-1), with defined pass thresholds and re-test rights after every model or RAG index update. Teams managing ongoing risk may also want to look at LLM security audit and AI model review services.

Incident notification windows: Vendor default timelines routinely consume half a regulated buyer's compliance clock before the buyer even learns of an incident. The episode lays out the specific 24-hour discovery-to-notification requirement and why named breach coordinators and customer-led disclosure rights belong in the RFP.

Exit, escrow, and the kill clause: Assume the relationship ends. Source and weight escrow with a neutral third party, data portability across all fine-tuned artifacts and RAG indexes, and a defined kill clause triggered by acquisition, insolvency, or unpatched critical vulnerabilities are the three provisions every exit section needs.
For more on structuring the contract type itself — fixed-scope, managed appliance, or co-build — the related episode Fixed-Scope, Managed Appliance, or Co-Build: Picking the Right Private LLM Contract covers the trade-offs in depth.
LLM.co
hardware.co
6 min
September 27Fixed-Scope, Managed Appliance, or Co-Build: Picking the Right Private LLM Contract
With AI pilot failure rates hovering near eighty percent and tens of billions in enterprise spending producing little measurable return, the contract model behind a private LLM deployment deserves far more scrutiny than most buyers give it. This episode of LLM.co examines the three structures dominating the private LLM market today — drawing on the detailed breakdown of private LLM contract models on the LLM.co blog — and explains how each one allocates risk, ownership, and exit rights in ways that can make or break an enterprise AI program.
The episode covers:
Why contract structure is the real risk decision: Who absorbs a bad assumption — the buyer or the vendor — is determined entirely by which contract shape is signed, not by which model is chosen.

Fixed-scope builds: Best for well-bounded use cases with stable data and written acceptance criteria; regulated-industry builds often add $200K–$600K over baseline hosting, with vendors pricing in a 15–30% contingency buffer for unknowns. Teams evaluating a private AI contract review deployment are a prime fit — but only when the scope can be described unambiguously before signing.

The scope-drift failure mode: Fixed-scope contracts break down when evaluation criteria shift after stakeholders see early outputs — a hallucination that looked acceptable in the SOW can become a showstopper in demo week, and every change becomes a margin-carrying change order.

Managed appliance subscriptions: The vendor operates a private model inside the buyer's network perimeter; risk for model performance and drift shifts to the vendor while the buyer retains data governance. Modern hardware enables fine-tuning models up to 70B parameters on a single high-memory node, making a true LLM appliance viable even in air-gapped compliance environments.

Exit-clause essentials for appliance contracts: Before signing, secure a portable-format egress clause for weights and vector indexes, a data-destruction attestation with a defined timeline, and a right to run the system in read-only mode during wind-down — or "managed" quietly becomes lock-in.

Co-build engagements: Time-and-materials or retainer arrangements where vendor engineers embed under the buyer's technical leadership; the right fit when requirements are genuinely emergent, but IP allocation must explicitly name ownership of fine-tuned weights, training corpus derivatives, and evaluation harnesses — silence on any of those becomes costly at renewal.
For more on EU compliance obligations that intersect with all three contract structures, listen to EU AI Act Conformity for Self-Hosted LLMs: What Your Evidence File Must Prove. Source material and further reading are available at the LLM.co blog link above.
LLM.co
hardware.co
5 min

About LLM.co

From the publisher's feed

Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.

Share LLM.co

Sign up to save your podcasts

LLM.co

LLM.co episodes

About LLM.co