This September 2025 paper introduces SearchInstruct, a novel framework designed to enhance Supervised Fine-Tuning (SFT) of large language models (LLMs) by constructing high-quality, domain-specific instruction datasets. This approach overcomes challenges like data scarcity and outdated model knowledge by dynamically retrieving external, up-to-date documents to generate accurate, context-grounded answers for augmented questions. SearchInstruct operates via a four-stage pipeline: starting with a small set of human-generated questions, expanding them using an LLM, retrieving relevant documents (via RAG or web search), and synthesizing context-aware responses. Experimental results confirm that the method significantly boosts LLM performance in specialized domains like Iranian culture and efficiently facilitates model editing for factual updates, creating a measurable improvement over baseline models.

Source:

https://arxiv.org/pdf/2509.10708