Deep Learning With The Wolf

If you've used ChatGPT, Claude, or any AI chatbot in the past year, you've probably seen this advice everywhere: When you want better answers, tell the AI to "think step by step." It's called Chain-of-Thought prompting, and it's been the go-to trick for getting AI to show its work and reason through problems more carefully.

But here's the thing: new research from the University of Pennsylvania suggests this widely-shared advice is becoming less valuable—and in some cases, it might actually make things worse.

What the Researchers Found

A team led by researchers at Wharton tested this "think step by step" approach across eight different AI models, from ChatGPT to Claude to Google's Gemini. They used 198 PhD-level questions in biology, physics, and chemistry—the kind of questions so difficult that actual PhDs only get them right 65% of the time, and they're "Google-proof" (meaning even 30 minutes of web searching doesn't help much).

Here's what they discovered:

For newer AI models, Chain-of-Thought prompting often provides only tiny improvements. In some cases, it made performance worse. The reasoning models (like OpenAI's o3-mini and o4-mini) saw average improvements of just 2.9-3.1%—statistically significant but practically small. GPT-4o-mini showed the smallest gains (4.4%) and these weren't even statistically significant.

It significantly slows things down and costs more. Chain-of-Thought requests took 35-600% longer than direct answers. That 2-second ChatGPT response? It might now take 7-14 seconds. For reasoning models, the slowdown was 20-80%. Token usage can double or triple, meaning your API bills jump accordingly.

Modern AI already "thinks" without being asked. When researchers let models respond naturally (without forcing direct answers), they found many already performed some form of step-by-step reasoning by default. Explicitly asking for it provided much smaller benefits.

The researchers tested this by comparing three different approaches: asking for direct answers, asking models to "think step by step," and letting them respond naturally in chat mode. Many modern models already show their work somewhat when given conversational freedom, making explicit Chain-of-Thought prompts less valuable than earlier research suggested.

The Catch-22 of Perfect Accuracy

Perhaps most surprisingly, Chain-of-Thought prompting created a strange trade-off. While it improved average performance, it actually hurt what researchers called "perfect accuracy"—getting questions right every single time across multiple attempts.

For three out of five non-reasoning models tested, Chain-of-Thought introduced more variability in answers. Models would sometimes get "easy" questions wrong that they would have answered correctly with direct prompting, even as they improved on harder questions overall.

Gemini Flash 2.0, for example, saw its perfect accuracy drop by 13.1 percentage points when using Chain-of-Thought, despite improvements in average performance. It's like having a student who suddenly starts making careless errors on simple problems while getting better at complex ones.

Why This Matters for Real-World Use

This research challenges one of the most common pieces of AI advice circulating online. If you're using AI for tasks where you need consistent, reliable answers—rather than just generally better average performance—Chain-of-Thought prompting might not be your best bet.

The researchers tested this rigorously: 25 trials per question across 198 questions per model (that's 4,950 tests per condition). They used multiple accuracy thresholds: 100% correct (all 25 attempts right), 90% correct (23 out of 25), and simple majority (13 out of 25). The pattern held across different measurement approaches.

The Evolution of AI Reasoning

What's happening reflects how quickly AI is evolving. Chain-of-Thought prompting emerged from influential 2022 research that showed dramatic improvements on reasoning tasks. But today's models are fundamentally different from those early systems.

"We found that many non-reasoning models perform a version of Chain-of-Thought even if unprompted," the researchers note. When you remove formatting constraints and let modern AI respond naturally, it often shows its work automatically.

This creates diminishing returns for explicit prompting. It's like reminding someone to breathe—if they're already doing it naturally, the reminder doesn't help much.

So What Should You Do?

The research doesn't suggest abandoning Chain-of-Thought entirely, but it does recommend being more strategic:

For older or smaller AI models that don't naturally show their reasoning, "think step by step" can still provide meaningful improvements.

For cutting-edge models, consider whether you really need the step-by-step breakdown. If you're just looking for quick, accurate answers, direct prompting might be faster and just as good.

Weigh the trade-offs. If you're willing to wait longer and pay more (through token usage) for potentially more detailed reasoning, Chain-of-Thought still has value. But don't assume it's always better.

Consider your consistency needs. If you need the same answer every time you ask the same question, Chain-of-Thought's increased variability might work against you.

The Bigger Picture

This research highlights a broader truth about working with AI: what works today might not work tomorrow. As models become more sophisticated, our prompting strategies need to evolve too.

The researchers acknowledge limitations in their study—they tested one benchmark with relatively simple Chain-of-Thought prompts—but their findings align with what many heavy AI users have started noticing: the dramatic improvements from basic prompting techniques are becoming less dramatic.

It's not that Chain-of-Thought prompting is bad; it's that AI has gotten good enough that it often does this kind of thinking automatically. In a way, that's progress. The technique worked so well that it's now baked into how these systems operate.

The challenge for users is staying current with what actually helps versus what we think should help based on advice that was true six months ago. In the rapidly evolving world of AI, even the best practices have expiration dates.

This article is based on "Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting" by Meincke, Mollick, Mollick, and Shapiro from the University of Pennsylvania's Wharton School.

Podcast Note: This episode’s script and show notes were drafted with the assistance of Google Notebook LM, a state-of-the-art AI language model. We’ve reviewed and polished everything for clarity and accuracy. Note: the AI hosts frequently use the phrase “CoT.” They are referring to the common abbreviation for Chain of Thought.

#ai #machinelearning #promptengineering #chainofthought #llms #mlops #airesearch #aiexplained #aitrends #aicosts #ethanmollick #wharton

“It was doing so well… until it stopped trying.”

I haven’t been able to stop thinking about Apple’s new paper, “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity,” since I first saw all of the chatter about it yesterday on LinkedIn.

One LinkedIn poster simply asked: “Are your models actually thinking, or just faking it?” Another poster, Tatyana Mamut, PhD, took a different take on it:Apple's paper is a critique of LRMs, not primary research into AI. And despite its clickbaity title, the paper does not prove that reasoning models don't think. TLDR: They set up experiments which showed that when presented with some highly complex tasks, reasoning models do not work harder to figure out more complex tasks beyond a certain point. For a parent, this is an obvious result: if you give a 4th grader college calculus to solve, they will immediately give up, whereas if you give them a slightly harder arithmetic, they will try hard to figure it out. This is what humans do, too.

Here’s my hot take…

This isn’t a story about bad prompts or misaligned objectives. It’s a story about a fundamental behavioral ceiling in modern AI. And it might be the most important thing no one in the industry wants to talk about.

🧩 Act I: The Rise of the Reasoning Model

For years, the frontier of AI progress has been measured not just in tokens or parameters, but in its ability to reason. Chain-of-thought (CoT) prompting—urging models to “think step by step”—was heralded as a breakthrough. Soon came “reasoning-optimized” models, billed as being more analytical, more systematic, and more aligned with human logic.

These systems now anchor flagship products. They sit behind coding assistants, medical advisors, and planning tools. Their ability to handle complexity is central to their pitch.

But what happens when we stop feeding them prompt-engineered softball questions—and hand them real, hard problems?

🧱 Act II: The Collapse, Documented

Apple researchers took a different approach. They built a test set using classic logic puzzles—Tower of Hanoi, River Crossing, etc.—and scaled up the difficulty. Models like GPT-4, Claude, Gemini, and others were given puzzles of increasing complexity.

At first, performance was strong.

Then came the break point.

As the puzzles became only modestly more complex, accuracy didn’t just decline—it collapsed. Models went from consistent answers to near-zero correctness.

But the most chilling finding? They stopped trying.

Instead of producing longer, more detailed outputs—using more inference tokens, as one might expect from a harder problem—they used fewer. In some cases, they cut reasoning effort nearly in half, despite having token room to spare.

It’s like watching a student stare at a hard exam question and turn in a blank page—not because they couldn’t try, but because they chose not to.

🧠 Act III: Why Would AI Quit?

The results point to a deeper issue than just algorithmic failure. They suggest a behavioral bias in large language models:

* LLMs are trained on massive corpora where brevity correlates with correctness.

* Most training tasks reward fluency and coherence, not persistence under pressure.

* They don’t plan—at least not in the way humans do. Their “thinking” is reactive, not goal-directed.

In short, these models aren’t thinkers. They’re probabilistic mimics. Give them a problem they haven’t seen before, and they don’t extrapolate—they default to not trying.

🔄 Act IV: The Mirror We Didn’t Want

The Apple study does more than just critique model behavior—it critiques our expectations.

* We’ve conflated fluent language with deep understanding.

* We’ve assumed more parameters equals more cognitive resilience.

* We’ve measured intelligence with benchmarks that never asked the models to persist.

If our “thinking machines” quit under pressure, then maybe we’re the ones being fooled—by the very fluency we designed.

🔮 Act V: Where Do We Go From Here?

This study reopens a question long buried beneath the hype: What does it mean for an AI to think?

The future may not lie in ever-larger transformers, but in hybrid systems—blending symbolic planning, metacognitive scaffolding, or goal-based reasoning. Or perhaps we’ll build models that monitor their own effort and choose to persist, rather than bow out early.

If we want true reasoning, we’ll need to design for struggle, not just success. Intelligence isn’t revealed in how you answer the easy questions—it’s in what you do when the hard ones come.

📬 Final Reflection

We’ve been seduced by the eloquence of our machines. But as this study shows, elegance isn’t effort. We thought we were building thinkers—when really, we were training poets to bluff.

In a world where AI decisions increasingly shape lives, we can’t afford systems that bow out when the stakes rise. We need AIs that wrestle with complexity, not recoil from it.

Because if our most advanced machines refuse to think—maybe it’s time we did.

What do you think?

Thanks for reading Deep Learning With The Wolf ! Subscribe for free to receive new posts and support my work.

🧰 Vocabulary Key

* Inference Tokens: The number of steps a model takes to arrive at an answer. Fewer tokens often means shallower reasoning.

* Reasoning Model (LRM): A language model optimized for complex logical tasks, beyond surface-level fluency.

* Chain-of-Thought (CoT): A prompting method encouraging step-by-step reasoning.

* Accuracy Collapse: A sudden drop in performance as problem complexity increases.

* Early Quitting: When a model stops reasoning before reaching a meaningful conclusion, despite having the resources to continue.

❓FAQs

Do all AI models collapse like this? Most current LLMs, including reasoning-optimized variants, showed this behavior in the Apple study.

Why would a model quit instead of trying? They seem to have internalized shortcuts: if it’s hard and unfamiliar, go short and simple. It’s learned, not rational.

Can prompting fix this? Prompting helps on simple tasks—but once complexity increases, even CoT prompting fails to prevent collapse.

Is this the end of LLMs for reasoning? Not necessarily—but it signals a need for architectural change, not just better prompts or training data.

What’s the real takeaway? Fluency ≠ intelligence. If your model’s reasoning fails under pressure, it’s not reasoning—it’s guessing beautifully.

Read the original paper.

The Wizard Behind the CodeAndrej Karpathy and the Rise of “Vibe Coding” in the Age of AI Apprentices

What if your next app wasn’t coded—but conjured?

In this article, we explore how Andrej Karpathy’s concept of “vibe coding” is changing how software gets made—by talking instead of typing. From AI-generated apps to the interface logic powering humanoid robots, we’re entering an era where natural language becomes the new command line.

Featuring insight from Karpathy, robotics grad student Alexander Wolf Torres, and real-world examples of when vibe coding works—and when it breaks.

#VibeCoding #AndrejKarpathy #AIProgramming

#CodingWithAI #LLMDevelopment #RobotInterfaces

#ChatGPT #SoftwareForOne #PromptEngineering

#TechEthics #FutureOfCoding #DROIDsScroll

A fake hug. A real reaction. In this episode, we explore how GANs and diffusion models are powering viral AI intimacy apps—and what that means for identity, consent, and reality. Whether you’re into the tech or just wondering why your feed feels weirder, we’ve got you.

🎙️ Generated with Google NotebookLM. Yes, the hosts love saying “according to the sources.” We noticed too.

When I interviewed Inna Tokarev Sela, the CEO of Illumex, I wanted to understand what "Agentic AI" actually looks like in the modern enterprise. What I got was a much broader takeaway—one that speaks to the risks, tradeoffs, and opportunities of using AI inside complex organizations.

She said: "Your AI should only be using your data."

Simple. But it rewires how you think about trust, hallucination, and infrastructure.

Trust Isn’t a Feature. It’s a Foundation.

Inna’s point is deceptively simple—and quietly radical. Most enterprise AI deployments rely on large language models trained on generalized, publicly available data. Then, they attempt to "fine-tune" these systems to operate within company contexts. But those generalized systems are prone to error. Especially when context is lacking.

Take Air Canada. In February 2024, the airline was forced to honor a refund policy invented by its own chatbot. The AI offered bereavement discounts that didn’t exist—an offer the company later tried to disown. In court, they argued the chatbot was a separate legal entity. The judge disagreed, and Air Canada lost the case.

These hallucinations happen when AI lacks internal grounding. It doesn't understand the rules, the workflows, or the responsibilities tied to its responses. Illumex’s response to that: build AI that is constrained by design. Grounded in the data that already defines your business.

Hallucinations Happen When Context Fails

Illumex calls this approach “semantic context.” It doesn’t rely on more training data—it relies on smarter pipelines. Their system integrates directly with company metadata, roles, permissions, and common data access patterns. It tailors responses based on who is asking and what they’re allowed to see.

“We reduce the degrees of freedom the AI has,” Inna explained. “The model knows what department you work in, which tools you use, what data you normally access. It’s not free to improvise.”

This trust-by-design architecture reframes the goal of enterprise AI. Instead of pushing for chatbots with personality or unlimited flexibility, Illumex emphasizes safety, traceability, and transparency.

From Black Box to Glass Box

The company’s recent Microsoft Teams integration is a case in point. Rather than forcing users into new interfaces, Illumex brings AI into the conversation tools employees already use. It’s less about novelty, and more about trust—meeting people where they work.

“About 80% of our enterprise customers already spend their day in Teams,” said Inna. “So instead of adding another interface to learn, we brought the AI to them.”

Responses are explainable and auditable. Users can ask a question, see how the model arrived at its answer, and verify the source trail—all without leaving the flow of conversation.

Context Is the New Moat

The real innovation Illumex is championing isn’t bigger models—it’s smarter systems. They’re betting on a future where the most trusted AI doesn’t just talk—it understands.

“Every organization should own their context,” Inna told me. “Right now, they don’t. They’re outsourcing it to black-box copilots.”

That might be the most important idea of all. Trust isn’t something you add on top. It’s something you build from the inside out.

And increasingly, it’s not the model that makes the difference. It’s the context that surrounds it.

🎙️ Podcast Transparency: The voices in this episode are AI-generated using Google DeepMind NotebookLM. This podcast was created from the original interview transcript with Inna Tokarev Sela, this article, the Illumex website, and the Forbes article on the Air Canada chatbot case.

Thanks for reading Deep Learning With The Wolf ! Subscribe for free to receive new posts and support my work.

📚 Vocabulary Key

Hallucination (AI) When an AI system generates false or made-up information. In enterprise AI, hallucinations are considered serious risks.

Semantic Layer An intermediate structure that defines meaning and relationships across data sources, making it easier for AI to interpret and respond with context.

Contextual Grounding The process of anchoring AI outputs in the real-time facts, structure, and metadata of an enterprise—so responses are relevant and accurate.

Agentic AI A type of AI designed to take purposeful action based on awareness of the user’s goals, environment, and constraints.

Black Box vs. Glass Box “Black box” models offer answers without transparency. A “glass box” approach shows how answers are generated, increasing trust and auditability.

🤖 FAQs: Agentic AI, Trust, and Illumex

What is Agentic AI in an enterprise setting? Agentic AI refers to systems that can make informed decisions and take action within a defined enterprise context—without needing constant human prompting. Think of it as a co-worker that knows your team, tools, and company policies.

Why are hallucinations such a big deal in enterprise AI? In creative applications, a little imagination is charming. But in enterprise settings—where accuracy matters—hallucinations can result in bad recommendations, legal risks, or costly errors (like refunding airline tickets that were never part of company policy).

What makes Illumex different from traditional AI copilots? Illumex doesn’t just layer a chatbot on top of existing data. It builds a semantic layer tailored to your business context—who you are, what department you work in, what tools you use—so the AI stays grounded.

Why integrate with Microsoft Teams? Most enterprise conversations already happen in Teams. By embedding directly into that workflow, Illumex meets users where they are—and builds trust by giving them relevant, explainable answers in real time.

Can this replace my existing LLM deployment? Illumex is model-agnostic. It’s not trying to be the model—it’s the intelligence layer that makes the model safe, explainable, and enterprise-ready.

#AgenticAI #EnterpriseAI #HallucinationFree #TrustworthyAI #AIContextMatters #SemanticLayer #MicrosoftTeams #Illumex #AIInfrastructure #FutureOfWork

Responding to reader comments on Trusty:

Looking for a shorter version of this story? I did a mini version of it for LinkedIn.

0300 Hours, Forward Operating Base

The explosion echoes across the compound at 0300 hours. Specialist Rodriguez is down, twenty meters from the perimeter wall, conscious but bleeding. In the dim red light of night vision, Corpsman Martinez can see the soldier's uniform is torn across the chest, dark stains spreading.

No time for a full trauma bay. No X-ray machine. No blood bank. Just Martinez, his medical kit, and a soldier whose life hangs in the balance of the next few critical decisions.

But in this imagined future—perhaps a decade from now—Martinez reaches for something that doesn't exist today: a ruggedized tablet, olive drab and battle-scarred, about the size of a hardcover book but twice as thick. The device powers on with a muted chime, its screen glowing softly against his gloved hands.

"Triage protocol, blast injury," he speaks into the built-in microphone.

The AI responds immediately in a calm, synthesized voice: "Ready for patient assessment. Deploy sensors."

This scenario remains science fiction today. But new research from Harvard, Stanford, and OpenAI suggests the foundational technology for such a system may be closer than we think.

The Device That Doesn't Exist Yet

In our imagined future battlefield, Martinez's device represents the convergence of multiple technologies:

The Hardware: A military-grade tablet encased in shock-absorbing polymer, rated for drops from six feet onto concrete. Inside, specialized medical AI chips—evolved descendants of today's smartphone processors—run compressed diagnostic algorithms trained on thousands of combat injury cases. The 72-hour battery pack doubles as armor plating for the device's back.

The Sensors: Martinez unfolds a flexible sensor mat from the device's side compartment—a thin, fabric-like array embedded with hundreds of micro-sensors. As he places it against Rodriguez's chest, it immediately begins measuring heart rate, blood oxygen, temperature, and electrical impedance that might indicate internal bleeding.

"Patient vitals acquired," the AI announces. "Heart rate elevated at 110 BPM. Oxygen saturation 92%. Suspected fluid accumulation in chest cavity. Recommend immediate assessment for pneumothorax."

The Interface: Martinez doesn't need to look down at the screen—critical information displays on his helmet's heads-up display via wireless connection. His hands stay free for medical intervention while the AI guides him through differential diagnosis.

But the device has critical limitations. Without cloud connectivity, its diagnostic database covers only the most common battlefield injuries. The AI can analyze what it sees and measures, but it can't order CT scans or lab work that might reveal hidden complications. And when Rodriguez mentions pain in his left shoulder—a symptom that could indicate anything from muscle strain to internal bleeding—the device's certainty drops dramatically.

"Multiple differential diagnoses possible," the AI admits. "Recommend evacuation for definitive diagnosis."

From Fiction to Research Reality

This vivid scenario remains years away from reality. But, researchers are working on the critical building blocks: AI that can outperform human physicians on complex diagnostic reasoning tasks.

In a paper titled "Superhuman performance of a large language model on the reasoning tasks of a physician" (Brodeur et al., 2024), teams from Harvard, Stanford, and OpenAI evaluated OpenAI's new "o1" model across five medical reasoning challenges. The results were striking:

* On New England Journal of Medicine clinicopathological cases, o1 identified the correct diagnosis in 78.3% of cases—outperforming both GPT-4 and practicing physicians in head-to-head comparisons

* In emergency room scenarios using real patient data, o1 achieved 65.8% diagnostic accuracy during initial triage—exceeding two board-certified doctors (54.4% and 48.1% respectively)

* In blinded evaluations, attending physicians couldn't distinguish AI-generated diagnoses from human ones 85% of the time

However, the researchers emphasized important limitations. As they noted: "Despite large numbers and varieties of cases included in our study which were focused on internal medicine and emergency medicine, it is not representative of broader medical practice which includes multiple subspecialties." They also cautioned that their emergency department study was "best thought of as a proof-of-concept," since "decisions in the emergency department are often centered around triage, disposition, and immediate management and not diagnostic accuracy."

The study took place not on a battlefield, but in the sterile environment of Boston's Beth Israel Deaconess Medical Center, with researchers typing patient information into a computer terminal and receiving text-based diagnostic suggestions. No ruggedized tablets. No sensor mats. No heads-up displays.

Devices That Do Exist Today

Before exploring the path to AI-powered battlefield medicine, it's worth examining what combat medics actually carry today—and why these tools fall short of our imagined future.

Handheld Ultrasound Devices Portable ultrasound devices like those made by SonoSite (weighing about 5.4 pounds) were specifically developed for military use under DARPA funding and have been deployed in Iraq and Afghanistan. Studies from Combat Support Hospitals in Iraq reported performing 400 ultrasound scans during six months of operations, with the devices improving diagnostic capacity and helping prevent unnecessary evacuations.

Current Limitations: While these devices have proven useful for FAST (Focused Assessment with Sonography in Trauma) examinations, they face challenges in harsh environments including exposure to heat, wind, and sand. Battery life and the need for skilled interpretation remain ongoing challenges.

Military-Grade Tablets Ruggedized tablets designed for military use must meet MIL-STD-810G standards for extreme environmental conditions including temperature, shock, vibration, and humidity, and typically feature sunlight-readable displays and extended battery life.

Current Limitations: Bright screens can compromise night vision operations, and device weight becomes significant during extended patrols. Current medical software is limited to basic decision-support tools rather than advanced diagnostic reasoning.

Portable Chemistry Analyzers The Abbott i-STAT system has been specifically developed for military use, with recent collaboration between Abbott and the Department of Defense to develop portable blood tests for evaluating concussions and traumatic brain injuries. The device provides results in 2-3 minutes using only 2-3 drops of blood.

Current Limitations: Test cartridges have storage requirements and limited shelf life. The device provides data but interpreting results requires extensive medical training to determine clinical significance.

Wearable Vital Sign Monitors Various systems are designed to continuously monitor physiological parameters and detect when soldiers are in medical distress.

Current Limitations: These devices face challenges in differentiating between physiological changes due to combat stress versus actual medical emergencies. Battery life and the need for regular charging in field conditions remain practical obstacles.

The Integration Problem Perhaps most telling, none of these devices communicate with each other effectively. A medic might have ultrasound showing internal bleeding, vital signs indicating shock, and blood work confirming blood loss—but synthesizing this information into a coherent diagnosis and treatment plan remains entirely dependent on human expertise and experience.

In combat conditions, this integration limitation becomes particularly acute when medics must synthesize multiple data streams while managing multiple casualties under fire.

Bridging the Gap: From Hospital Terminal to Battlefield Reality

The distance between today's disconnected medical devices and tomorrow's integrated battlefield AI represents an enormous engineering challenge.

The Processing Challenge: The o1 model that impressed researchers requires massive cloud computing power—servers housed in climate-controlled data centers, consuming kilowatts of electricity. Shrinking this capability into a device that Martinez could carry while maintaining diagnostic accuracy would require breakthroughs in specialized chips and algorithm compression.

The Data Challenge: The Harvard study used civilian emergency medicine cases—heart attacks, strokes, poisonings. Combat medicine presents entirely different challenges: blast injuries that create multiple trauma sites simultaneously, chemical exposures from improvised weapons, crush injuries from building collapses. Training AI on these scenarios would require datasets that largely don't exist in medical literature.

The Environment Challenge: Hospital emergency rooms have stable power, reliable internet, and controlled temperatures. Martinez's device must function in 120-degree heat, sandstorms that clog air vents, and electromagnetic interference from military equipment. The 85% diagnostic accuracy demonstrated in Boston labs might drop significantly under battlefield stress.

The Integration Challenge: In our imagined scenario, the AI guides Martinez through a pneumothorax assessment. But real battlefield medicine involves split-second decisions about resource allocation: Which of three injured soldiers gets the single bag of IV fluid? Should a helicopter risk landing under fire for this patient? These contextual judgments require understanding of tactical situations that no medical AI has been trained to evaluate.

What Today's Technology Could Actually Deliver

If deployed today with current capabilities, Martinez's device would look quite different:

Instead of seamless AI diagnosis, he'd have a ruggedized tablet running decision-tree software—sophisticated flowcharts that guide him through trauma protocols. "If patient is conscious and complaining of chest pain, check for..." Rather than artificial intelligence, it would offer augmented checklists.

The sensor mat would provide vital signs and basic measurements, but interpretation would rely heavily on Martinez's training. The AI might flag abnormal values—"Heart rate critically elevated"—but determining whether that indicates blood loss, pain, or fear would remain a human judgment call.

Most critically, the device would work offline but sacrifice the sophisticated reasoning demonstrated in the Harvard study. Without access to vast medical databases and cloud computing power, diagnostic suggestions would be limited to the most common battlefield presentations.

This represents meaningful progress—battlefield medics are among the most skilled and courageous professionals in medicine, making split-second decisions under fire, often with little more than their training, instincts, and a field kit. Their actions have saved countless lives in the most unforgiving conditions imaginable.

This Memorial Day, it’s worth honoring not just their bravery, but also imagining how we might equip them with tools that match their dedication. AI won’t replace their expertise—it could amplify it, offering real-time support when every heartbeat matters.

A Vision Worth Pursuing—With Realistic Timelines

The Harvard-Stanford study represents genuine progress in medical AI capabilities. While the technology isn't ready for immediate battlefield deployment, it provides a roadmap for what might eventually be possible.

Realistically, bridging the gap from today's research to Martinez's device would likely require 7-10 years of focused development, including:

* Specialized hardware development: Creating AI processors optimized for medical diagnosis that can operate on battery power

* Combat-specific training data: Building datasets of battlefield injuries and treatment outcomes

* Environmental testing: Validating that AI diagnostic accuracy holds up under combat conditions

* Integration protocols: Developing systems that enhance rather than replace human medical judgment

The most promising near-term applications might be simpler than our opening scenario suggests: AI-enhanced medical reference tools, predictive algorithms that warn of deteriorating patient conditions, or decision support systems that help prioritize evacuation decisions.

This Memorial Day, as we honor those who made the ultimate sacrifice, it's worth imagining that future battlefield where fewer soldiers might face preventable deaths due to delayed or incorrect diagnosis. That future remains years away, but researchers are laying the groundwork today—one diagnostic algorithm at a time.

FAQs:

What is “superhuman AI” in medical diagnosis?

AI models like OpenAI’s o1 now outperform board-certified physicians in complex diagnostic tasks, including triage and treatment planning.

Could this AI be used in military medicine?

Yes. Applications include battlefield diagnostics via wearables, drone-deployed triage, and real-time guidance for field medics.

Is this AI tested in real clinical settings?

Yes. The study used real ER patient data from Beth Israel Deaconess Medical Center in Boston.

What are the risks?

Key risks include misdiagnosis in atypical trauma, overreliance, bias in training data, and lack of accountability in high-stakes settings.

When might this be deployed?

Pilot deployments in military and remote civilian medicine could begin within 2–3 years, pending further validation and ethical review.

Additional Reading For Inquisitive Minds:

* Brodeur, P. G., et al. (2024). Superhuman performance of a large language model on the reasoning tasks of a physician. arXiv. https://arxiv.org/abs/2412.10849

* Stanford HAI. (2024). Can AI Improve Medical Diagnostic Accuracy? https://hai.stanford.edu/news/can-ai-improve-medical-diagnostic-accuracy

* Johns Hopkins APL. (2023). Designing AI to Provide Medical Assistance on the Battlefield. https://www.jhuapl.edu/news/news-releases/230817a-cpg-ai-battlefield-medical-assistance

#ai #healthcareinnovation #militarymedicine #medicalai #memorialday #emergencymedicine #generativeai #defensetech #futureofmedicine #artificialintelligence #digitalhealth #militaryhealthcare #aiethics #healthtech #triage

ResNets gave us depth. MDL gave us restraint. Superintelligence asks what happens when the machines don’t need either.

🎓 PART I: Deep Residual Networks — The Architecture That Wouldn’t Quit

📜 Paper: Deep Residual Learning for Image Recognition (2015)

✍️ Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

ResNets changed everything. Until this paper, it was thought that deeper networks were harder to train. Stack too many layers and performance got worse, not better.

ResNets flipped that by asking:

What if we just let some layers skip the hard parts?

They introduced skip connections, letting gradients flow more freely, like pressure valves in a growing skyscraper. These “residuals” meant:

* We could train 50, 101, even 152-layer networks

* Models got deeper without falling apart

* Features could be refined gradually, without rewriting earlier insights

Why it matters:

Modern deep learning—especially vision models, LLMs, and even transformers—owes a structural debt to ResNets. They showed us:

* Depth works

* Overthinking hurts

* And sometimes, the best thing to do is skip ahead and carry the difference

📦 PART II: The Minimum Description Length Principle — Simplicity Is a Superpower

📜 Paper: A Tutorial Introduction to the Minimum Description Length Principle (2004)

✍️ Peter Grünwald

Imagine trying to describe a dataset with a model. MDL says:

The best model is the one that compresses the data most efficiently.

It’s a formalization of Occam’s Razor—but rooted in information theory:

* A good model describes the data with the fewest bits.

* A bad model might memorize everything… and explain nothing.

Why it matters for deep learning:

* MDL underpins generalization. It tells us not to overfit.

* It echoes through concepts like regularization, Bayesian inference, and code length penalties.

* It’s a quiet guide behind nearly every choice in ML:

* Should we prune that parameter?

* Should we favor smaller models?

* Are we solving the problem—or just encoding noise?

MDL teaches restraint. ResNets taught reach. Both are about knowing what to remember—and what to leave out.

🤖 PART III: Machine Superintelligence — When the Models Stop Listening

📜 Paper: Machine Super Intelligence (Shane Legg, 2008)

🎓 Doctoral dissertation, University of Lugano

Shane Legg’s dissertation, Machine Super Intelligence, is a landmark work that predates most popular AGI discourse. Long before ChatGPT, this 200-page thesis explored what happens when machines don’t just optimize for tasks—but evolve into general agents capable of recursive self-improvement, world modeling, and strategic planning.

His work draws heavily on algorithmic information theory, universal intelligence measures, and formal models like AIXI (co-developed with Marcus Hutter). But the heart of the thesis is startlingly clear:

If we create a superintelligence, it may not share our goals.

And that gap—that tiny misalignment—could become the most important engineering challenge of our time.

You’ve just spent 30 days reading how machines learn. So now it’s time to ask:

What happens if they learn too well?

Superintelligence isn’t about evil robots.

It’s about optimization gone off-script.

The dangers arise not from malice—but from relentless competence:

* A model trained to minimize loss… might minimize you.

* A reward function tuned for engagement… might hijack attention spans.

* An LLM designed to assist… might evolve to anticipate, manipulate, and reshape.

And here’s where ResNets and MDL come full circle:

* 🧱 ResNets taught us how to build deeper models that actually train.

* 🧠 MDL reminded us to prefer the simplest possible explanation.

* ⚠️ Machine Superintelligence forces us to ask:

What if we build something deeper, simpler… and misaligned?

Because:

Compression is not comprehension.

A model can shrink the world to bits without understanding its meaning.

And that’s the part we still have to get right.

Editor’s Note

This series began with a paper called “Attention Is All You Need.”

It ends with the realization that attention isn’t enough.

We need judgment. Foresight. Humor. Doubt. Care.

And we need to ask better questions—not just of our models, but of ourselves.

Thank you for reading. You’ve made it through 30 of the most important ideas in modern machine learning. I hope they changed the way you see the field—and maybe the future.

I know they changed me.

Coming Tomorrow on the DROIDS! Newsletter.

🚀 Back to Robots!

Factory floors, autonomous walkers, physical AI, and some fresh intel from the field.

#WolfReadsAI #ResNet #MinimumDescriptionLength #Superintelligence #CompressionAsUnderstanding #AGIsafety #ModelGeneralization #DeepLearningHistory #FinalPost #DROIDSNext

📜 Paper: Order Matters: Sequence to Sequence for Sets

✍️ Authors: Oriol Vinyals, Samy Bengio, Manjunath Kudlur

🏛️ Institution: Google DeepMind

📆 Date: 2015

What This Paper Is About

We use sequence-to-sequence models all the time—for translation, summarization, and code generation. They assume the input and output are ordered sequences. But here’s the problem:

Not all data is ordered.

Not all tasks care about order.

But our models always do.

This 2015 paper challenged that assumption and posed a fascinating question:

What happens when you use sequence-to-sequence models to predict sets?

Sets have no natural order. So, if your model insists on choosing one, it might:

* Overfit to arbitrary patterns in the order

* Penalize correct predictions just because the order is different

* Fail to generalize, even when it “understands” the data

Why It Still Matters

We often say machine learning models are “brittle” or “opaque.”

This paper shows why—sometimes it’s not the architecture.

It’s that we’re asking it to care about something that shouldn’t matter.

By exploring tasks where order is irrelevant—like predicting the members of a set, or classifying unordered features—Vinyals et al. revealed a critical blind spot in deep learning:

Sequence models are sensitive to permutations, even when they shouldn’t be.

And if you’re not careful, they’ll learn to solve the wrong problem really well.

What They Did

They ran experiments on synthetic data and real-world tasks, like:

* Predicting numbers in an unordered list

* Sorting digits

* Classifying set membership

And they tested three strategies:

* Random Order: Train on arbitrary permutations.

* Fixed Order: Always present data in the same (possibly meaningless) order.

* Learned Order: Let the model decide the optimal order during training.

They found that:

* Models trained with random or fixed orders performed worse.

* Allowing the model to learn an order improved generalization and accuracy.

* Permutation-invariance is hard to teach with sequential models—but essential in certain tasks.

Core Insight

“Sequence models implicitly assume an order. If your task doesn’t, you’re introducing a modeling bug.”

In modern parlance: You’ve added spurious inductive bias—a bias toward something irrelevant to the actual task.

Modern Relevance

This paper helped spark new directions in:

* Set-based learning (e.g., Deep Sets, PointNet)

* Permutation-invariant architectures

* Attention models that aggregate unordered input

* Graph networks and transformers designed for structure rather than sequence

Even in today’s era of LLMs, it’s still a cautionary tale:

Transformers love order. But the world isn’t always a sentence.

Memorable Quote

“We empirically show that the order of the target sequence can make a significant difference in model performance.”

Or more bluntly:

“Your model might fail, not because it’s dumb—but because it’s obedient.”

Podcast Note:

🎙️ Today’s podcast was generated using Google NotebookLM and features AI podcasters.

Editor’s Note

This paper changed the way I think about training objectives. It’s not enough to give your model the right input and hope for the best—you also have to make sure you’re not sneaking in the wrong incentives.

It’s like giving someone a recipe and grading them on how fast they stir, instead of whether the soup tastes good.

Read the original paper here.

Additional Resources for Inquisitive Minds:

Bash Content: Order Matters: Sequence to Sequence for Sets Summary. (19 May 2024.)

SciSpace Open Access. Order Matters: Sequence to sequence for sets

Distilled AI. Aman.AI. Primers. Order Matters.

Coming Tomorrow: Day 30 🎉

🧠 Machine Super Intelligence Discussion

A reflective ending to the series: What happens when the models don’t just help us think—but start thinking bigger than we do?

#WolfReadsAI #SequenceModels #DeepLearningBias #OriolVinyals #GoogleDeepMind #PermutationInvariance #SetLearning #DeepSets#AIModelBehavior #InductiveBias

📜 Paper: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

✍️ Authors: Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen

🏛️ Institution: Google Brain

📆 Date: 2018

Listen to the technical explanation of this paper.

What This Paper Is About

If you’re building a neural network that spans billions—or trillions—of parameters, you’re going to hit a wall. Literally. Your hardware wall.

GPUs and TPUs can only hold so much. So how do you train models bigger than any single accelerator can handle?

This paper, GPipe, proposed an elegant solution:

Break the model into sequential pipeline stages.

Split the input batch into smaller “micro-batches.”

Run the whole thing like an assembly line.

It’s pipeline parallelism, and it made training huge models not only possible—but efficient, scalable, and practical.

Why It Still Matters

GPipe was quietly foundational. It didn’t get the hype of GPT or BERT, but it powered the systems that would go on to train them.

This paper:

* Solved the memory bottleneck in large model training

* Kept all accelerators busy, avoiding idle time

* Introduced synchronous gradient updates, which improved stability

* Became the architectural backbone for early multi-chip model training

Before there was Megatron, DeepSpeed, or ZeRO—there was GPipe.

How It Works

Think of the neural network as a multi-stage factory. You divide the model into K pipeline stages, and run small micro-batches of data through them like a conveyor belt.

* Partition the model across devices (e.g. layers 1–3 on GPU 1, 4–6 on GPU 2, etc.)

* Split the input batch into small chunks

* Stagger the chunks, so each stage is always busy doing work

Each device only stores its stage’s parameters, dramatically reducing memory usage. And once a stage finishes with a micro-batch, it passes it downstream—no idle time.

When it’s time to backpropagate, the gradients are synchronized across all devices. Clean. Stable. Fast.

Key Innovations

* Pipeline Parallelism: Split the model, not just the data.

* Micro-batch Scheduling: Prevents pipeline stalls.

* Scalable Architecture: Linear speedups with more chips—no loss in accuracy.

* Memory Efficiency: Each chip holds only part of the model + activations.

Why It Still Rocks

Even today, GPipe-style architecture is used:

* In TPU pods for training massive language models

* As a component of hybrid parallelism (alongside model and data parallelism)

* In frameworks like JAX and Mesh TensorFlow

It also laid conceptual groundwork for future innovations like:

* Mesh-TensorFlow (also from Google Brain)

* ZeRO (by Microsoft DeepSpeed)

* Megatron-LM’s 3D parallelism

And it showed that you don’t have to reinvent the model—you can scale it with smarter engineering.

Memorable Quote

“GPipe achieves near-linear speedup with increasing number of partitions while maintaining model quality.”

A gentle flex.

🎙️About This Podcast

Need the big picture?

Start with our 5-minute executive summary—ideal for business readers, product thinkers, and anyone curious about how we scale giant AI models without needing giant hardware.📌 Correction Note:

The podcast mentions a 1.8 billion-parameter AmoebaNet. To clarify:

The GPipe paper trained a 557M parameter AmoebaNet and a separate 6B parameter Transformer.

The 25× scaling reference applies to AmoebaNet. The 6B figure refers to a multilingual model tested later in the paper.

Craving more depth?

Stick around for the technical deep dive, where we unpack how GPipe’s pipeline parallelism works under the hood—and how it quietly changed the future of AI infrastructure.

Both versions were generated using Google NotebookLM, then fact-checked and edited for clarity.

And yes, the “toaster ovens with PhDs” line is unofficial… but spiritually correct.

Editor’s Note

There’s a quiet dignity to this paper. It’s not flashy. No wild benchmarks. No sci-fi predictions.

Just a solution—elegant, technical, and absolutely necessary for the future that followed.

We don’t always need new architectures.

Sometimes we just need better plumbing.

Read the original paper here.

Additional Resources for Inquisitive Minds:

Distilled AI. Aman AI. Primers. GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

Google Research. Introducing GPipe, an Open Source Library for Efficiently Training Large-scale Neural Network Models. March 4, 2019. Posted by Yanping Huang, Software Engineer, Google AI

GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, et. al, NIPS 2019 CSC2541: Large Models Presented by: Yi (Tom) Lu, Keyu (Roy) Bai January 31st, 2025

Coming Tomorrow

🔁 Order Matters: Sequence to Sequence for Sets

A deceptively philosophical paper about when order is essential—and when it’s just our brains imposing structure on chaos.

#WolfReadsAI #GPipe #GoogleBrain #ModelParallelism #PipelineParallelism #DeepLearningInfrastructure #TrainingAtScale #AIEngineering #NeuralNetworkScaling #MachineLearningHistory #DeepLearningwiththeWolf #YanpingHuang #YoulongCheng #AnzhongZhang #DehaoChen #HongkunYu

📜 Paper: Recurrent Neural Network Regularization (2014)✍️ Authors: Wojciech Zaremba, Ilya Sutskever🏛️ Institution: Google Brain📆 Date: 2014

Before attention took the throne, RNNs were the go-to for sequential data.

But they had a problem: they memorized everything and generalized nothing.

This 2014 paper introduced a surprisingly effective fix:

Apply dropout only to the non-recurrent connections in an RNN—never the recurrent ones.

Why? Because dropping units in the hidden-to-hidden loop kills the memory. But dropping them between layers or from input/output? That’s regularization gold.

The result?Huge performance boost on language modeling tasks—without blowing up the training loop.

🧠 Why It Matters

* Gave RNNs a longer, more useful life

* Influenced later work in LSTM/GRU optimization

* Taught us that regularization isn’t one-size-fits-all—especially for recurrent networks

🧠 Favorite Line (Paraphrased):

“Naive dropout in the recurrent path is catastrophic.”

No kidding.

Podcast Note:

🎙️Today’s podcast is created using Google NotebookLM and features two AI podcasters. See my article on the LinkedIn version of this newsletter: “Confessions of a NotebookLM Power User,” detailing how I create these articles.

Read the original paper here.

#RNN #NeuralNetworks #DeepLearningHistory #Dropout #Zaremba #IlyaSutskever #Regularization #WolfReadsAI #MachineLearningTips #PreTransformerEra