GenAI Level UP

Titans: Learning to Memorize at Test Time

Listen Later

Are current AI models hitting a memory wall? Join us as we delve into the fascinating research behind "Titans: Learning to Memorize at Test Time," an innovative approach to AI learning.

The podcast covers key concepts from the paper, including:

  • The challenges of long-term memory in AI, noting that models like Transformers are good at understanding immediate relationships but struggle with retaining information from the past.
  • How the Titan model addresses these limitations by equipping AI with both short-term and long-term memory.
  • The concept of "learning to memorize at test time", where the model figures out what is important to remember as it encounters new information.
  • The use of a surprise-based approach, where the model prioritizes information that is most surprising or unexpected.
  • The combination of surprise-based long-term memory with a more traditional short-term memory.
  • The way long-term memory is stored, which is within the parameters of a deep neural network.
  • The use of a technique similar to gradient descent with momentum for efficient memory formation.
  • The model's built-in forgetting mechanism to manage memory capacity and prioritize important information.
  • The use of attention to guide the search for relevant information in long-term memory.
  • The ability of Titans to handle longer sequences of information by using long-term memory to free up short-term memory.
  • The advantages of Titans in real-world applications such as language modeling, common sense reasoning, and the needle in a haystack problem.
  • The three variants of the Titan architecture: Memory as a Context (MAC), Memory as a Gate (MAG), and Memory as a Layer (MAL). Each variant uses long-term memory differently.



    • ...more
    View all episodesView all episodes
    Download on the App Store
    GenAI Level UPBy GenAI Level UP