These five papers from 2022 up to 2025 discuss various knowledge distillation techniques aimed at transferring the capabilities of large language models (LLMs) to smaller, more efficient models, often without the need for explicit context during inference. One paper introduces Contextualization Distillation (CD) for Knowledge Graph Completion (KGC), demonstrating that utilizing LLMs like PaLM2 to generate descriptive context for triplets significantly enhances the performance of smaller, specialized KGC models, often outperforming direct use of LLMs for the task. Another source proposes Context Distillation as a general method for language models to internalize abstract instructions, step-by-step reasoning (scratch-pads), and concrete examples, effectively eliminating the need for lengthy prompts and improving inference efficiency. The third document details In-context Learning Distillation, a framework that combines in-context learning objectives with traditional language modeling to effectively transfer few-shot learning abilities from large to smaller models under different tuning paradigms. Finally, Generative Prompt Internalization (GenPI) is presented as a method to fully embed long, complex prompts into a smaller model by training it to generate the prompt content and the reasoning for its corresponding behavior, greatly increasing efficiency in agent-based applications. 2022: Learning by Distillation Context https://arxiv.org/pdf/2209.15189 2022: In-context Learning Distillation: Transferring Few-shot https://arxiv.org/pdf/2212.10670 2024: Contextualization Distillation from Large Language Model for Knowledge Graph Completion https://aclanthology.org/2024.findings-eacl.32.pdf May 12, 2025: Efficient LLM Context Distillation https://arxiv.org/pdf/2409.01930 March 25, 2025: Generative Prompt Internalization https://arxiv.org/pdf/2411.15927