In this episode:
• Opening Up a New Chapter: Apertus: Linda introduces the Apertus paper, highlighting its focus on data compliance and extreme multilingualism. Professor Norris is intrigued but skeptical about what 'fully open' and 'compliant' truly mean in practice.
• Clean Data, Clear Conscience?: The hosts discuss Apertus's novel approach to data compliance, including retroactively honoring robots.txt opt-outs. They debate the ethical implications and the performance trade-offs of training on a more restricted, 'cleaner' dataset.
• Speaking Over 1800 Languages: Linda explains the massive scale of Apertus's multilingual training, with 40% of its data being non-English. Professor Norris questions the depth versus breadth of language understanding, especially for the thousands of low-resource languages included.
• Forgetting for a Better Future: The Goldfish Loss: The conversation turns to the technical recipe, focusing on the 'Goldfish objective' designed to prevent memorization. Professor Norris finds the name amusing and probes whether this technique genuinely reduces copyright and privacy risks without harming the model's capabilities.
• The Verdict on Apertus: Linda and Professor Norris wrap up by evaluating Apertus's position in the LLM landscape. They conclude that its commitment to full transparency—releasing code, data scripts, and checkpoints—sets a new, important standard for the open-source community.