Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing AlignmentSearch: An AI Alignment-Informed Conversional Agent, published by BionicD0LPH1N on April 1, 2023 on LessWrong.

Authors: Henri Lemoine, Thomas Lemoine, and Fraser Lee

Note: Throughout our development process, we benchmarked with smaller scale vector embeddings, assuming performance would scale linearly. It appears that we were incorrect. It turns out the entirety of our augmented alignment database is too slow to be practically run on the servers we’re using without more specialized hardware, which we plan to migrate to over this weekend. As a stop gap, the live published version of AlignmentSearch is running with just over 11% of our embeddings - every piece of content in our dataset explicitly tagged with “AI”. From our testing, expect somewhat worse results than the full version.

We are excited to introduce AlignmentSearch, an attempt to create a conversational agent that can answer questions about AI alignment. We built this site in response to ArthurB’s $5k bounty for a LessWrong conversational agent calling for the creation of a chatbot capable of discussing bad AI Alignment takes, and guiding people new to the field through common misunderstandings and early points of confusion.

Tl;dr

AlignmentSearch uses a dataset about AI alignment to construct a prompt for ChatGPT to answer AI alignment-related questions, while citing established sources. We qualitatively observe a massive boost in the quality of answers over ChatGPT without any specialized prompt, with results either on par or better than those given by much stronger LLMs (GPT-4 and Bing Search).

Overview

AlignmentSearch indexes the Alignment Research Dataset, generating vector embeddings to enable nearest-semantic-neighbor search. We take a user query to find the top-k most semantically close “paragraphs” (text blocks of around 220 tokens, plus some padding). The size of the dataset means we generally have access to a few paragraphs that are very semantically similar to the user’s question, and are likely to contain information that is useful in producing an answer. We form a prompt that gives these paragraphs to ChatGPT along with the user’s question and instructions on citation, and retrieve an answer somewhere between summary and synthesis, all with accurate inline citations linking to the source material. We’ve created a website that lets the user interact easily with this process: ask questions, dive into the sources used in an answer, and ask for further clarifications. We also have an alternative mode that exposes the raw results of the semantic search.

Alignment Research Dataset

The alignment research dataset was announced on LessWrong in June 2022, with a related paper accessible here. It contains posts and papers from a wide range of websites and books about alignment, including LessWrong, stampy.ai, arXiv, the alignment newsletter, etc. As it was created in June ‘22, recent posts are absent. We are in the process of updating the dataset to contain recent AI alignment content.

Variable Source Quality

Not all sources are made equal. Posts on LessWrong with negative karma were removed, as to reduce the chances of the website parroting back foolishness. Moreover, stampy.ai contains a highly curated dataset of common question-answer pairs related to AI safety, and we plan on favoring this source in semantic searches. A more nuanced and optimization-driven prioritization system, a better curated dataset, and a better formatted prompt may further improve the quality of the answers.

Embeddings

We currently use OpenAI’s text-embedding-ada-002 embedding model. The motivation behind this decision comes from the recent price drop in OpenAI ada, now roughly 500x less expensive than its predecessor. Embedding the entire 1.2 GB dataset cost ~$50.

One might think embedding the entirety of L...