Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GPT-4: What we (I) know about it, published by Robert AIZI on March 15, 2023 on LessWrong.

OpenAI released a press release, research statement, and system card about GPT-4 approximately one eternity (24 hours) ago. The general public can’t use it yet, but it’s in the process of being rolled out to paid subscribers of ChatGPT, and via a waitlist to the API. We also got confirmation that the Bing AI (also currently rolling out via waitlist) is based on GPT-4.

Here I’ll try to summarize the news and boil down what we (I) know about GPT-4. Many points lifted from the discussion at lesswrong.

My main takeaways:

Capabilities progress is continuing without slowing.

OpenAI spent a lot of time on RLHF/fine-tuning to prevent unethical use (facilitating crime, generating hate speech, etc), and they behave as if this is sufficient to solve alignment.

OpenAI is no longer so open - we know almost nothing about GPT-4’s architecture.

Previously from OpenAI.

(Just recapping the progress of the GPT series of models, feel free to skip.)

AIs advance very quickly. The most impressive AI these days are large language models, including the GPT series, and they are all based on the transformer, an architecture introduced in 2017.

In 2018 OpenAI released the Generative Pre-Trained Transformer (GPT), which approached natural language tasks by predicting the next token. It was especially evaluated on narrow tasks (e.g. “Is the sentiment of this user review positive or negative? [user review]. The sentiment is.”). A key technique for GPT (and all its successors) was the eponymous “pre-training”, where the AI is trained not on any particular task, but just to predict the next token in a text. This gives you access you a huge volume of training data (literally all text), while building general understanding of the world - answering factual questions is a form of token completion, so the AI needs to be able to answer those questions, etc. This pre-training built a general knowledge base, and then GPT was “fine-tuned” to individual tasks with additional training on those datasets.

We know from the GPT-4 press release that OpenAI trained GPT-3.5 “a year ago”, using the same architecture as GPT-3 but with a custom-designed supercomputer and a better “deep learning stack”. While I’m not aware of publicly available comparisons of GPT-3 and 3.5, some users reported that 3.5 felt smarter, and I’m inclined to believe them.

During this time, OpenAI also became interested in Reinforcement Learning on Human Feedback (RLHF). In RLHF, a human evaluates the output of the AI, and rates it on some objectives (such as “helpful and honest”), and this is used to train the AI. An RLHF'd version of GPT 3.5 was released in November 2022 under the name ChatGPT, which became somewhat popular.

GPT-4 Timeline

According to the research statement, GPT-4 “finished training” in August of 2022. It’s not entirely clear what they mean by this, because they say they’ve been “iteratively improving” it since then - was this RLHF, fine-tuning, or something else? If they mean it finished pre-training, why didn’t they use that term?

Capabilities Improvements

GPT-4 continues to improve capabilities over GPT-4 and GPT-3.5. The raw numbers are available in the paper, but I think in the long run what matters is what GPT is being evaluated on. Now, in addition to AI benchmarks like “MMLU” and “HellaSwag”, GPT-4 is being evaluated on exams that humans take.

GPT-4 scored a 1410/1600 on the SAT and a 4/5 or 5/5 on the AP Art History, Biology, Calculus BC, Chemistry, Environmental Sciences, Macroeconomics, Microeconomics, Physics 2, Psychology, Statistics, US Government, US History, and US World History exams (a 3/5 is passing. GPT-4 scored only a 2/5 on {English Language and Composition} and {English Literature and Composition}). We’re now in ...