Recent research advances attention distillation to optimize transformers. HAD binarizes keys/queries for efficiency, while SHD aligns varying head counts. CompoDistill improves multimodal reasoning via visual alignment, and new losses transfer visual characteristics in diffusion models.Sources:1)February 3 2025Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context TransformersMark Horton, Tergel Molom-Ochir, Peter Liu, Bhavna Gopal, Chiyue Wei, Cong Guo, Brady Taylor, Deliang Fan, Shan X. Wang, Hai Li, Yiran Chenhttps://doi.org/10.48550/arXiv.2502.017702)February 11 2025Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment BarriersZhaodong Bing, Linze Li, Jiajun Lianghttps://doi.org/10.48550/arXiv.2502.074363)February 27 2025Attention Distillation: A Unified Approach to Visual Characteristics TransferYang Zhou, Xu Gao, Zichong Chen, Hui Huanghttps://doi.org/10.48550/arXiv.2502.202354)October 14 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMsJiwan Kim, Kibum Kim, Sangwoo Seo, Chanyoung Parkhttps://doi.org/10.48550/arXiv.2510.12184