Token Engineering

Token Engineering

By NeuroMetric AITechnology
Download on the App Store

Token Engineering episodes

  • From MIT Decoding Research to Today’s Inference Tradeoffs

    Check out the latest episode of Inference Time Tactics. Our guest is Pawan Deshpande, founder, product leader, and angel investor in companies like Anthropic and Toast, with roles at Google, Scale AI and Domino Data Lab. 

     

    Hosts Rob May & Calvin Cooper sit down with Pawan to cover:

    • Early MIT NLP research applied to today’s inference-time tradeoffs
    • How to evaluate enterprise agents in practice
    • Training data plus inference filtering in real deployments
    • Open source adoption realities in the enterprise
    • Where durable value lives in the stack

      • Connect with Pawan Deshpande

      Website: https://pawandeshpande.com/


      Academic / Research Works & Thesis
      Decoding Algorithms for Complex Natural Language Tasks (MIT thesis, 2007)
      Randomized Decoding for Selection-and-Ordering Problems


      Connect with Neurometric:
      Website: https://www.neurometric.ai/ 

      Substack: https://neurometric.substack.com/ 

      X: https://x.com/neurometricai/ 

      Bluesky: https://bsky.app/profile/neurometric.bsky.social 

       

      Hosts:

      Rob May

      https://x.com/robmay 

      https://www.linkedin.com/in/robmay

       

      Calvin Cooper

      https://x.com/cooper_nyc_ 

      https://www.linkedin.com/in/coopernyc

      31 min
    • Drag, Drop, and Deploy: Rethinking How We Build AI Systems

      In this episode of Inference Time Tactics, Rob, Cooper, Byron, and Dave share product updates for Neurometric’s Inference Time Compute Studio and what they reveal about the shift from single models to full AI systems. They discuss why wiring models together at scale is so challenging, how a drag-and-drop interface can make experimenting with inference strategies easier, and why open source, benchmarking, and community feedback are key to building the next generation of composable AI systems.

       

      We talked about:

       

      • Why AI is shifting from single models to full systems and what that means for builders.
      • The challenges of wiring multiple models together at scale and running them in production.
      • How Neurometric’s drag-and-drop interface simplifies testing inference strategies without code.
      • Why open-source models are becoming increasingly competitive with commercial solutions.
      • The lack of standardization in AI stacks and why the industry still feels like the “early web” era.
      • How inference-time compute can balance performance, cost, and latency across different tasks.
      • Why benchmarks alone are insufficient and how domain-specific evaluations can fill the gap.
      • The role of community feedback in shaping priorities for benchmarks and new primitives.



        • Connect with Neurometric:

        Website: https://www.neurometric.ai/ 

        Substack: https://neurometric.substack.com/ 

        X: https://x.com/neurometricai/ 

        Bluesky: https://bsky.app/profile/neurometric.bsky.social

         

        Hosts:

        Rob May

        https://x.com/robmay 

        https://www.linkedin.com/in/robmay

         

        Calvin Cooper

        https://x.com/cooper_nyc_ 

        https://www.linkedin.com/in/coopernyc

         

        Guest/s:

        Byron Galbraith

        https://x.com/bgalbraith 

        https://www.linkedin.com/in/byrongalbraith 

         

        Dave Rauchwerk

        https://x.com/elevenarms 

        https://www.linkedin.com/in/dave-rauchwerk-0ba82822

        21 min
      • Beyond Vibe Testing: Smarter Eval for Agentic AI

        In this episode of Inference Time Tactics, Rob, Cooper, and Byron explore Salesforce’s CRMArena-Pro benchmark and what it reveals about the limits of enterprise AI agents. They share why benchmark scores often fail in production, how inference-time tactics like best-of-N can improve reliability, and what NeuroMetric is building to make eval easier—from an ITC Test Engine to a drag-and-drop interface for rapid visualization and experimentation.

         

        We talked about:

         

        • Why Salesforce’s CRMArena-Pro benchmark highlights the gap between lab benchmarks and real-world agent reliability.
        • How leading models perform inconsistently across single-turn and multi-turn enterprise tasks.
        • Why benchmark scores are weak predictors of operational success in production.
        • The role of inference-time tactics in reducing variance and improving stability.
        • NeuroMetric’s new platform: ITC Test Engine and drag-and-drop interface for experimentation.
        • Challenges in building agentic systems, from database integration to managing multi-prompt complexity.
        • Why large language models’ stochastic nature conflicts with business demands for reliability.
        • Latency, cost, and rate limits as major bottlenecks in scaling agentic workflows.
        • The limits of “vibe testing” and why rigorous evaluation frameworks are essential.
        • How Google’s Stacks tool speeds up evaluation with LLM-as-judge, and why it still falls short for enterprise needs.



          • Resources Mentioned:

          CRMArena-Pro from Saleforce:

          https://www.salesforce.com/blog/crmarena-pro/  

           

          Connect with Neurometric:

          Website: https://www.neurometric.ai/ 

          Substack: https://neurometric.substack.com/ 

          X: https://x.com/neurometricai/ 

          Bluesky: https://bsky.app/profile/neurometric.bsky.social

           

          Hosts:

          Rob May

          https://x.com/robmay 

          https://www.linkedin.com/in/robmay

           

          Calvin Cooper

          https://x.com/cooper_nyc_ 

          https://www.linkedin.com/in/coopernyc

           

          Guest/s:

          Byron Galbraith

          https://x.com/bgalbraith 

          https://www.linkedin.com/in/byrongalbraith

          23 min
        • GPT-5, The $100B Gap, and The Economics of Inference

          In this episode of Inference Time Tactics, Rob and Cooper unpack the launch of GPT 5.0 and what OpenAI’s new routing layer signals about the shifting AI landscape. They explore the tradeoffs of cost, latency, and accuracy, zoom out to programmable inference in an agent-driven world, and track the ripple effects on chips, data centers, and energy use.


          We talked about:

           

          • Why GPT 5.0’s launch felt more like refinement than a revolution in AI progress.
          • How OpenAI’s new routing layer reframes the race around inference control.
          • The tradeoffs routing enables between cost, latency, and accuracy across models.
          • Why the “one model to rule them all” view is giving way to multi-model orchestration.
          • The strategic role of programmable inference in an agent-driven world.
          • How router companies are becoming a strategic layer in the AI technology stack.
          • The impact of inference compute on chips, accelerators, and data center design.
          • Why energy use at scale is driving a push for more efficient AI systems.
          • Why inference optimization may be the next big competitive edge.

            •  

            Connect with Neurometric:

            Website: https://www.neurometric.ai/ 

            Substack: https://neurometric.substack.com/ 

            X: https://x.com/neurometricai/ 

            Bluesky: https://bsky.app/profile/neurometric.bsky.social

             

            Hosts:

            Rob May

            https://x.com/robmay 

            https://www.linkedin.com/in/robmay

             

            Calvin Cooper

            https://x.com/cooper_nyc_ 

            https://www.linkedin.com/in/coopernyc

            26 min
          • When AI Overthinks: Lessons from the Illusion of Thinking Paper

            In this episode of Inference Time Tactics, Rob, Cooper, and CTO Byron unpack Apple’s “Illusion of Thinking” paper—why it split the AI community, what it reveals about reasoning model limits, and how hidden thinking traces shape performance. They share insights from building an open-source tool to reproduce the study, explain why models loop, overthink, or stall, and outline what it will take to build more reliable reasoning systems for real-world use.


            We talked about:

             

            • Why Apple’s Illusion of Thinking paper sparked heated debate in the AI community.
            • How reasoning models work, including hidden “thinking” phases and token budget limits.
            • Key findings on when reasoning improves results, when it degrades them, and where it stalls.
            • Reasons models loop, overthink, or abandon tasks.
            • Building an open-source tool to replicate the study and test local reasoning models.
            • What real-time reasoning traces reveal about model behavior and limits.
            • Challenges in scoring reasoning quality and treating “I don’t know” as a valid output.
            • Why reasoning models must be matched carefully to specific tasks.
            • The ongoing debate over scaling vs. new architectures for advancing reasoning.
            • Developing a benchmarking platform to help enterprises choose models for IP-sensitive applications.



              • Resources Mentioned:

              Illusion of Thinking Paper

              https://machinelearning.apple.com/research/illusion-of-thinking 

              Neurometric Illusion of Thinking Tool

              https://github.com/NeurometricAI/illusion-of-thinking 

               

              Connect with Neurometric:

              Website: https://www.neurometric.ai/ 

              Substack: https://neurometric.substack.com/ 

              X: https://x.com/neurometricai/ 

              Bluesky: https://bsky.app/profile/neurometric.bsky.social

               

              Hosts:

              Rob May

              https://x.com/robmay 

              https://www.linkedin.com/in/robmay

               

              Calvin Cooper

              https://x.com/cooper_nyc_ 

              https://www.linkedin.com/in/coopernyc

               

              Guest/s:

              Byron Galbraith

              https://x.com/bgalbraith 

              https://www.linkedin.com/in/byrongalbraith

              24 min
            • The Strategic Trade Offs Behind Inference Time Compute Decisions

              In this episode of Inference Time Tactics, Rob and Cooper dig into the strategic trade-offs driving a major shift in AI: why some enterprises start with closed models like OpenAI or Anthropic, then move to open-source stacks. The team breaks down the challenges of switching and how inference-time compute is becoming a competitive differentiator. They also unpack why pricing is shifting, how governance will evolve for this new layer, and what Rob learned from reviewing 250 research papers on reasoning algorithms.


               

              We talked about: 

               

              • Insights from reviewing 250 research papers on reasoning algorithms. 

              • Why enterprises start with closed models like OpenAI or Anthropic before moving to open-source stacks. 

              • Challenges of switching stacks, including model fragmentation, capability gaps, and hardware choices. 

              • Cost-performance trade-offs when choosing inference architectures. 

              • How inference-time configuration can become a competitive differentiator. 

              • The role of pricing shifts and vendor lock-in in AI adoption. 

              • Emerging governance considerations for inference workflows. 

              • The growing variety and complexity of inference-time techniques.. 

              • Benchmarking challenges for multi-step and reasoning tasks. 

              • Why the lack of best practices makes inference optimization harder to operationalize. 

                 

                 

                Connect with Neurometric:

                Website: https://www.neurometric.ai/  

                Substack: https://neurometric.substack.com/  

                X: https://x.com/neurometricai/  

                Bluesky: https://bsky.app/profile/neurometric.bsky.social  

                 

                Hosts: 

                Rob May 

                https://x.com/robmay  

                https://www.linkedin.com/in/robmay 

                 

                Calvin Cooper 

                https://x.com/cooper_nyc_  

                https://www.linkedin.com/in/coopernyc Comment end  

                20 min
              • Why Inference Time Compute Is the Future of AI

                Welcome to the very first episode of Inference Time Tactics — the podcast for builders, researchers, and engineers pushing the limits of AI performance.

                In this kickoff conversation, hosts Rob May and Cooper (co-founders of Neurometric AI) break down why inference time compute is emerging as the third scaling law of AI — and why it matters more than ever.

                They unpack:

                • What “inference time compute” really means (and how it differs from training and fine-tuning)

                • Why reasoning algorithms like best-of-N, chain of thought, and beam search are reshaping performance

                • How recent research — and OpenAI’s 2024 reasoning model — sparked an explosion of interest

                • The challenge of reliability (“three nines” and beyond) in multi-step agent workflows

                • Why open-source models may win big, and where inference fits at the edge

                  This is a technical, tactical deep-dive — but without the heavy math. If you’re building the next generation of AI systems, or just want to understand where the field is really headed, this episode is your starting point.

                  🔗 Learn more at neurometric.ai

                  22 min

                About Token Engineering

                From the publisher's feed

                A show about AI economics and the cost of thinking. From Neurometric, on the tokenminning movement, the discipline emerging to make every unit of AI work cheaper, faster, and more reliable.