Sign up to save your podcasts
Or
Share Week of 2025-10-19
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Week of 2025-10-19
Alex: Hello and welcome to The Generative AI Group Digest for the week of 19 Oct 2025!
Maya: We're Alex and Maya.
Alex: We had a busy week in the group — lots of practical problem-solving threads. Let’s jump straight into the highlights. First up: memory and summarization for RAG and multi-agent systems. Arpan Paul brought a great real-world pain: “objective is summarising large databases of tickets… number of tickets are 10k,” and he’s hitting LLM context limits while trying to do multi-turn memory. What stood out to you, Maya?
Maya: This is so common — you can’t shove 10,000 short-ticket summaries into one prompt. The essentials here are chunking and orchestration. Arpan was using langgraph for orchestration and Sonnet 4 as the LLM. People suggested everything from stepwise summarization, clustering like k‑means to sample representative tickets, to tagging summaries and using session/memory services. G Kuppuram and Sushanth emphasized defining the entities to extract — category, date, domain, priority — and making those part of the prompt.
Alex: Right. For non-technical listeners: RAG means you pull documents into the model’s context to answer a query. The “context window” is how much the model can read at once. When that’s smaller than your dataset, you need strategies. Practical pipeline idea: chunk documents, create short summaries, tag them with categories or issue-types, index those summaries in a vector DB, and at query time retrieve only the most relevant clusters then do a final summarization pass. That’s the hierarchical summarization approach.
Maya: Non-obvious takeaway: use parallel LLM calls but treat them like workers — run many small summarizers (possibly cheaper models), then merge results with a higher-quality model. Also consider session/memory services like Google’s ADK for conversation memory. If you’re using Sonnet, try splitting work across Sonnet 4 for heavy summarization and a lighter model like Nova for simple tasks to optimize cost and latency.
Alex: Good ops tip: precompute monthly or weekly summaries and tag them. Query-time work then becomes combining a handful of relevant summaries, not scanning 10k tickets live.
Maya: Next topic — AI in the offline world, factories, and “dark factories.” Paras Chopra asked about offline automation and whether physical bottlenecks make fast takeoff unlikely. The group had a measured conversation: Suryansh described it as a “fast gradient” — compute and software scale fast, atoms and supply chains do not. Pratik Desai said, “knowledge discovery is the first use case, actuation will take time.”
Alex: Why this matters: people imagine full robot-run factories overnight. In reality, many factories have great potential for AI, but the bottleneck is data capture, integration, and supply-chain constraints. Srihari highlighted IT/OT convergence problems and siloed data. Cheril pointed out China’s faster adoption due to clustered factories and scale.
Maya: Practical ideas: start with pockets that are high-volume and low-mix — tasks that are repetitive and standardized — and automate inspections or process optimization first. Invest in unified namespaces and standards (ISA-95, CESMII APIs) or a pseudo-standardization layer as Pratik mentioned. Use RAG and knowledge discovery to surface insights before you try to automate actuations; build digital twins for simulation where feasible.
Alex: A non-obvious point: you can get serious value by optimizing engineers’ discretionary parameters using historical data. Hadi Khan’s uncle — a boiler consultant — said engineers often set different parameters to reach the same outcome. Simple data-driven optimization on those parameters can yield fast ROI without heavy robotics.
Maya: Let’s move to agent skills and composability. Pratyush Choudhury highlighted Anthropic’s Skills and people compared that to an “app store for skills.” Simon Willison and tp53 posted useful links. What are “skills” in plain language?
Alex: Skills are pre-built tools or sub-agents that a main model can call for specific tasks — think of them as plugins or microservices the model invokes when needed. Pratyush quoted that “Claude will only access a skill when it's relevant,” which helps make agent behavior more modular and predictable.
Maya: Why it matters: skills reduce ad‑hoc prompt engineering and let you stitch capabilities — search, code execution, domain logic — together. Practical moves: try Claude Code’s skills or the Superpowers skills repo to prototype. Think about modular design: build small, testable skills, and keep the state and I/O explicit. Also consider governance early — enterprise sharing vs public marketplaces is a real business decision.
Alex: On that marketplace thought — tp53 and others asked whether skills should be remixable or monetizable. If you build a great “literature review for biology” skill, could you monetize or fork it? For now the emphasis seems enterprise-first, but that’s a design consideration if you’re building reusable assets.
Maya: Next up: tools, model tradeoffs, and infra updates. Pulkit flagged LangSmith supporting JS evals now — useful if your evaluation stack is JavaScript-native. Karthik Sashidhar found Sonnet 4 faster than Sonnet 4.5 in Bedrock, and Varun recommended Haiku 4.5 for Sonnet-level quality at better speed. There was also chatter about Broadcom partnering with OpenAI and inference chips.
Alex: The big insight: latency and stack fit matter as much as raw quality, especially for interactive agents. If your app is Node.js-heavy, using LangSmith’s JS evals is a small change with big developer ergonomics wins. Benchmarks are your friend: measure median latency, cost per call, and effect on UX. Try smaller, faster models like Sonnet 4 or Haiku 4.5 for interactive paths and reserve bigger models for offline batch reasoning.
Maya: And watch infrastructure news — partnerships that diversify chips (Broadcom etc.) can change cost and supply dynamics for inference in the months ahead.
Alex: Let’s cover practical product adoption and wins. Vaibhav asked about AI business analysts — not many teams are using them beyond text-to-SQL. Hadi Khan replied with a concrete win: a senior accountant using Cursor/Claude Code on 200–1000 row Excel files, automating invoice processing and reconciliation without being a coder.
Maya: That’s a lesson: pick a high-frequency, repetitive workflow (invoicing, reconciliation, QA QA) and ship a narrow assistant. Tools to try: Cursor, Claude Code, LangChain for text-to-SQL, and local tool integrations. Don’t aim for a general “business analyst” out of the gate — aim for a concrete automation that saves a few hours per person per week.
Alex: Deployment tip: keep audit logs, let people verify outputs, and offer humans-in-the-loop for the first months so the assistant earns trust.
Maya: Two quick operations notes from the thread that deserve repeating. Yashwardhan had a concurrency OOM problem on a 2-core AWS host with Qdrant for PDF ingestion. His solution: stream the PDF during ingestion, free memory as you go, and add aggressive cleanups. He said this worked. Also, for Indic ASR evaluations, Prashant is building his own eval pipeline — measure WER and SWER and compare against Gemini/Google ASR and Indic-specific models.
Alex: And if you’re parsing documents, PaddleOCR‑VL is out — Nabeel flagged it as a compact vision-language model that handles tables, formulas, and handwriting. Worth testing if you need industrial-grade document parsing without huge models.
Maya: Quick note on personalization: Akshat asked about turning Gmail history into a persona. People suggested recent messages, classifier tagging into scenarios, and procedural memory via few-shot examples. Key caveat: privacy, data handling, and whether personalization truly improves output — often recent context is most valuable. Avoid wholesale fine-tuning unless you need sustained persona changes.
Alex: Time for our Listener Tips. My tip: if you’re building summarization over thousands of short tickets, start with hierarchical summarization — cluster or tag your tickets, precompute short summaries per cluster, and at query time retrieve and synthesize only the top clusters. How would you apply that to a support queue, Maya?
Maya: I’d take your hierarchy, run a weekly k‑means on embeddings to create cluster summaries, tag each cluster with issue categories, and then expose a “Top Issues this week” API to product managers. My tip: when you see memory/context limits, add a lightweight “info-gatherer” skill — a small model whose reward is to ask the three questions that would most reduce uncertainty for the main task, then pass its concise answers to the performer model. How would you use that in an invoice-reconciliation flow, Alex?
Alex: I’d have the info-gatherer extract vendor, amount ranges, and anomalies, then feed those to the reconciliation model so it focuses only on flagged rows. That reduces tokens and mistakes.
Maya: Great. Final wrap-up: thanks to everyone who shared practical fixes this week — Arpan, Hadi Khan, Pratyush Choudhury, Karthik Sashidhar, and many others in the thread. We’ll keep watching skills, agent orchestration, and real-world automation.
Alex: That’s it from us for the week. See you next week — stay curious and keep building.
Maya: Bye for now — and send us interesting threads for the next digest!
View all episodes
By Alex: Hello and welcome to The Generative AI Group Digest for the week of 19 Oct 2025!
Maya: We're Alex and Maya.
Alex: We had a busy week in the group — lots of practical problem-solving threads. Let’s jump straight into the highlights. First up: memory and summarization for RAG and multi-agent systems. Arpan Paul brought a great real-world pain: “objective is summarising large databases of tickets… number of tickets are 10k,” and he’s hitting LLM context limits while trying to do multi-turn memory. What stood out to you, Maya?
Maya: This is so common — you can’t shove 10,000 short-ticket summaries into one prompt. The essentials here are chunking and orchestration. Arpan was using langgraph for orchestration and Sonnet 4 as the LLM. People suggested everything from stepwise summarization, clustering like k‑means to sample representative tickets, to tagging summaries and using session/memory services. G Kuppuram and Sushanth emphasized defining the entities to extract — category, date, domain, priority — and making those part of the prompt.
Alex: Right. For non-technical listeners: RAG means you pull documents into the model’s context to answer a query. The “context window” is how much the model can read at once. When that’s smaller than your dataset, you need strategies. Practical pipeline idea: chunk documents, create short summaries, tag them with categories or issue-types, index those summaries in a vector DB, and at query time retrieve only the most relevant clusters then do a final summarization pass. That’s the hierarchical summarization approach.
Maya: Non-obvious takeaway: use parallel LLM calls but treat them like workers — run many small summarizers (possibly cheaper models), then merge results with a higher-quality model. Also consider session/memory services like Google’s ADK for conversation memory. If you’re using Sonnet, try splitting work across Sonnet 4 for heavy summarization and a lighter model like Nova for simple tasks to optimize cost and latency.
Alex: Good ops tip: precompute monthly or weekly summaries and tag them. Query-time work then becomes combining a handful of relevant summaries, not scanning 10k tickets live.
Maya: Next topic — AI in the offline world, factories, and “dark factories.” Paras Chopra asked about offline automation and whether physical bottlenecks make fast takeoff unlikely. The group had a measured conversation: Suryansh described it as a “fast gradient” — compute and software scale fast, atoms and supply chains do not. Pratik Desai said, “knowledge discovery is the first use case, actuation will take time.”
Alex: Why this matters: people imagine full robot-run factories overnight. In reality, many factories have great potential for AI, but the bottleneck is data capture, integration, and supply-chain constraints. Srihari highlighted IT/OT convergence problems and siloed data. Cheril pointed out China’s faster adoption due to clustered factories and scale.
Maya: Practical ideas: start with pockets that are high-volume and low-mix — tasks that are repetitive and standardized — and automate inspections or process optimization first. Invest in unified namespaces and standards (ISA-95, CESMII APIs) or a pseudo-standardization layer as Pratik mentioned. Use RAG and knowledge discovery to surface insights before you try to automate actuations; build digital twins for simulation where feasible.
Alex: A non-obvious point: you can get serious value by optimizing engineers’ discretionary parameters using historical data. Hadi Khan’s uncle — a boiler consultant — said engineers often set different parameters to reach the same outcome. Simple data-driven optimization on those parameters can yield fast ROI without heavy robotics.
Maya: Let’s move to agent skills and composability. Pratyush Choudhury highlighted Anthropic’s Skills and people compared that to an “app store for skills.” Simon Willison and tp53 posted useful links. What are “skills” in plain language?
Alex: Skills are pre-built tools or sub-agents that a main model can call for specific tasks — think of them as plugins or microservices the model invokes when needed. Pratyush quoted that “Claude will only access a skill when it's relevant,” which helps make agent behavior more modular and predictable.
Maya: Why it matters: skills reduce ad‑hoc prompt engineering and let you stitch capabilities — search, code execution, domain logic — together. Practical moves: try Claude Code’s skills or the Superpowers skills repo to prototype. Think about modular design: build small, testable skills, and keep the state and I/O explicit. Also consider governance early — enterprise sharing vs public marketplaces is a real business decision.
Alex: On that marketplace thought — tp53 and others asked whether skills should be remixable or monetizable. If you build a great “literature review for biology” skill, could you monetize or fork it? For now the emphasis seems enterprise-first, but that’s a design consideration if you’re building reusable assets.
Maya: Next up: tools, model tradeoffs, and infra updates. Pulkit flagged LangSmith supporting JS evals now — useful if your evaluation stack is JavaScript-native. Karthik Sashidhar found Sonnet 4 faster than Sonnet 4.5 in Bedrock, and Varun recommended Haiku 4.5 for Sonnet-level quality at better speed. There was also chatter about Broadcom partnering with OpenAI and inference chips.
Alex: The big insight: latency and stack fit matter as much as raw quality, especially for interactive agents. If your app is Node.js-heavy, using LangSmith’s JS evals is a small change with big developer ergonomics wins. Benchmarks are your friend: measure median latency, cost per call, and effect on UX. Try smaller, faster models like Sonnet 4 or Haiku 4.5 for interactive paths and reserve bigger models for offline batch reasoning.
Maya: And watch infrastructure news — partnerships that diversify chips (Broadcom etc.) can change cost and supply dynamics for inference in the months ahead.
Alex: Let’s cover practical product adoption and wins. Vaibhav asked about AI business analysts — not many teams are using them beyond text-to-SQL. Hadi Khan replied with a concrete win: a senior accountant using Cursor/Claude Code on 200–1000 row Excel files, automating invoice processing and reconciliation without being a coder.
Maya: That’s a lesson: pick a high-frequency, repetitive workflow (invoicing, reconciliation, QA QA) and ship a narrow assistant. Tools to try: Cursor, Claude Code, LangChain for text-to-SQL, and local tool integrations. Don’t aim for a general “business analyst” out of the gate — aim for a concrete automation that saves a few hours per person per week.
Alex: Deployment tip: keep audit logs, let people verify outputs, and offer humans-in-the-loop for the first months so the assistant earns trust.
Maya: Two quick operations notes from the thread that deserve repeating. Yashwardhan had a concurrency OOM problem on a 2-core AWS host with Qdrant for PDF ingestion. His solution: stream the PDF during ingestion, free memory as you go, and add aggressive cleanups. He said this worked. Also, for Indic ASR evaluations, Prashant is building his own eval pipeline — measure WER and SWER and compare against Gemini/Google ASR and Indic-specific models.
Alex: And if you’re parsing documents, PaddleOCR‑VL is out — Nabeel flagged it as a compact vision-language model that handles tables, formulas, and handwriting. Worth testing if you need industrial-grade document parsing without huge models.
Maya: Quick note on personalization: Akshat asked about turning Gmail history into a persona. People suggested recent messages, classifier tagging into scenarios, and procedural memory via few-shot examples. Key caveat: privacy, data handling, and whether personalization truly improves output — often recent context is most valuable. Avoid wholesale fine-tuning unless you need sustained persona changes.
Alex: Time for our Listener Tips. My tip: if you’re building summarization over thousands of short tickets, start with hierarchical summarization — cluster or tag your tickets, precompute short summaries per cluster, and at query time retrieve and synthesize only the top clusters. How would you apply that to a support queue, Maya?
Maya: I’d take your hierarchy, run a weekly k‑means on embeddings to create cluster summaries, tag each cluster with issue categories, and then expose a “Top Issues this week” API to product managers. My tip: when you see memory/context limits, add a lightweight “info-gatherer” skill — a small model whose reward is to ask the three questions that would most reduce uncertainty for the main task, then pass its concise answers to the performer model. How would you use that in an invoice-reconciliation flow, Alex?
Alex: I’d have the info-gatherer extract vendor, amount ranges, and anomalies, then feed those to the reconciliation model so it focuses only on flagged rows. That reduces tokens and mistakes.
Maya: Great. Final wrap-up: thanks to everyone who shared practical fixes this week — Arpan, Hadi Khan, Pratyush Choudhury, Karthik Sashidhar, and many others in the thread. We’ll keep watching skills, agent orchestration, and real-world automation.
Alex: That’s it from us for the week. See you next week — stay curious and keep building.
Maya: Bye for now — and send us interesting threads for the next digest!