Sign up to save your podcasts
Or
Share Contextual Blocks: Implicit Weight Updates and Federated Learning
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Contextual Blocks: Implicit Weight Updates and Federated Learning
We compare and contrast the math behind two recent research papers which we have covered individually before on this podcast:
July 2025:
Learning without training:
The implicit dynamics of in-context learning
https://arxiv.org/pdf/2507.16003
September 2025:
Federated Learning with Ad-hoc Adapter Insertions: The Case of
Soft-Embeddings for Training Classifier-as-Retriever
https://arxiv.org/pdf/2509.16508
The first source explores the concept of **In-Context Learning (ICL)** in neural networks, proposing that the effect of context on a token's output is equivalent to an **implicit weight update** in the neural network, specifically in the MLP layer, generalizing the transformer block using a **contextual block** notion. This work provides an explicit low-rank update formula for this implicit weight modification and mathematically demonstrates that token consumption aligns with an implicit **gradient descent learning dynamics** on the network weights. The second source introduces a novel **retrieval-augmented generation (RAG)** architecture called **Classifier-as-Retriever (CaR)** for memory-constrained edge devices, proposing to use a frozen Small Language Model (SLM) augmented with a small trainable **adapter network** to generate "soft embeddings" and a trainable **classifier head** instead of conventional similarity functions. Crucially, this architecture is designed for distributed training using **Federated Learning (FL)**, incorporating **Differential Privacy (DP)** techniques to ensure client-side data protection and demonstrating significant speedup advantages over centralized training.
View all episodes
By mcgrofWe compare and contrast the math behind two recent research papers which we have covered individually before on this podcast:
July 2025:
Learning without training:
The implicit dynamics of in-context learning
https://arxiv.org/pdf/2507.16003
September 2025:
Federated Learning with Ad-hoc Adapter Insertions: The Case of
Soft-Embeddings for Training Classifier-as-Retriever
https://arxiv.org/pdf/2509.16508
The first source explores the concept of **In-Context Learning (ICL)** in neural networks, proposing that the effect of context on a token's output is equivalent to an **implicit weight update** in the neural network, specifically in the MLP layer, generalizing the transformer block using a **contextual block** notion. This work provides an explicit low-rank update formula for this implicit weight modification and mathematically demonstrates that token consumption aligns with an implicit **gradient descent learning dynamics** on the network weights. The second source introduces a novel **retrieval-augmented generation (RAG)** architecture called **Classifier-as-Retriever (CaR)** for memory-constrained edge devices, proposing to use a frozen Small Language Model (SLM) augmented with a small trainable **adapter network** to generate "soft embeddings" and a trainable **classifier head** instead of conventional similarity functions. Crucially, this architecture is designed for distributed training using **Federated Learning (FL)**, incorporating **Differential Privacy (DP)** techniques to ensure client-side data protection and demonstrating significant speedup advantages over centralized training.