"WebGPT: Browser-assisted question-answering with human feedback"

Core Concept The paper introduces WebGPT, a system developed by OpenAI that improves long-form question answering by fine-tuning GPT-3 to interact with a text-based web-browsing environment. Instead of relying solely on its pre-trained internal knowledge, the model is equipped to send queries to the Microsoft Bing API, scroll through web pages, and extract specific quotes to answer open-ended questions.

Training Methodology To teach the model how to effectively search and synthesize information, the researchers relied heavily on human guidance:

• Behavior Cloning (Imitation Learning): The model was initially fine-tuned to mimic how human demonstrators used the text-based browser to find information and write answers.

• Human Feedback and Rejection Sampling: The researchers collected human comparisons of different model-generated answers to train a "reward model." The final system uses rejection sampling, meaning it generates multiple possible answers and selects the one that scores highest according to the reward model.

A crucial feature of WebGPT is that it must collect references (quotes and links) while browsing to support its final answers. This design choice allows human evaluators to easily check the factual accuracy of the model's claims without having to conduct subjective, independent research.

Key Results The model was primarily evaluated on the ELI5 ("Explain Like I'm Five") dataset, yielding highly competitive results:

• The best model (175 billion parameters, using behavior cloning and rejection sampling) produced answers that were preferred over human demonstrators 56% of the time.

• Its answers were preferred 69% of the time over the highest-voted human answers from Reddit.

• When tested on the adversarially-constructed TruthfulQA dataset, WebGPT produced truthful and informative answers 54% of the time, which outperformed the base GPT-3 model, though it still fell short of human-level truthfulness.