I see people use "in-context learning" in different ways.

Take the opening to "In-Context Learning Creates Task Vectors":

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework, where one uses a training set _S_ to find a best-fitting function _f(x)_ in some hypothesis class.

In one Bayesian sense, training data and prompts are both just evidence. From a given model, prior (random weights), and evidence (training data), you get new model weights. From the new model weights and some more evidence (prompt input), and a distribution of output text. But the "training step" _(text{prior}, text{data}) rightarrow text{weights}_ and "inference step" _(text{weights}, text{input}) rightarrow text{output}_ could be simplified to a single function:_(text{prior}, text{data}, text{input}) rightarrow text{output}_. An LLM trained on [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.