By Alessio + swyx
4.8
4646 ratings
The podcast currently has 78 episodes available.
If you see this in time, join our emergency LLM paper club on the Llama 3 paper!
For everyone else, join our special AI in Action club on the Latent Space Discord for a special feature with the Cursor cofounders on Composer, their newest coding agent!
Today, Meta is officially releasing the largest and most capable open model to date, Llama3-405B, a dense transformer trained on 15T tokens that beats GPT-4 on all major benchmarks:
The 8B and 70B models from the April Llama 3 release have also received serious spec bumps, warranting the new label of Llama 3.1.
If you are curious about the infra / hardware side, go check out our episode with Soumith Chintala, one of the AI infra leads at Meta. Today we have Thomas Scialom, who led Llama2 and now Llama3 post-training, so we spent most of our time on pre-training (synthetic data, data pipelines, scaling laws, etc) and post-training (RLHF vs instruction tuning, evals, tool calling).
Synthetic data is all you need
Llama3 was trained on 15T tokens, 7x more than Llama2 and with 4 times as much code and 30 different languages represented. But as Thomas beautifully put it:
“My intuition is that the web is full of s**t in terms of text, and training on those tokens is a waste of compute.”
“Llama 3 post-training doesn't have any human written answers there basically… It's just leveraging pure synthetic data from Llama 2.”
While it is well speculated that the 8B and 70B were "offline distillations" of the 405B, there are a good deal more synthetic data elements to Llama 3.1 than the expected. The paper explicitly calls out:
* SFT for Code: 3 approaches for synthetic data for the 405B bootstrapping itself with code execution feedback, programming language translation, and docs backtranslation.
* SFT for Math: The Llama 3 paper credits the Let’s Verify Step By Step authors, who we interviewed at ICLR:
* SFT for Multilinguality: "To collect higher quality human annotations in non-English languages, we train a multilingual expert by branching off the pre-training run and continuing to pre-train on a data mix that consists of 90% multilingualtokens."
* SFT for Long Context: "It is largely impractical to get humans to annotate such examples due to the tedious and time-consuming nature of reading lengthy contexts, so we predominantly rely on synthetic data to fill this gap. We use earlier versions of Llama 3 to generate synthetic data based on the key long-context use-cases: (possibly multi-turn) question-answering, summarization for long documents, and reasoning over code repositories, and describe them in greater detail below"
* SFT for Tool Use: trained for Brave Search, Wolfram Alpha, and a Python Interpreter (a special new ipython role) for single, nested, parallel, and multiturn function calling.
* RLHF: DPO preference data was used extensively on Llama 2 generations. This is something we partially covered in RLHF 201: humans are often better at judging between two options (i.e. which of two poems they prefer) than creating one (writing one from scratch). Similarly, models might not be great at creating text but they can be good at classifying their quality.
Last but not least, Llama 3.1 received a license update explicitly allowing its use for synthetic data generation.
Llama2 was also used as a classifier for all pre-training data that went into the model. It both labelled it by quality so that bad tokens were removed, but also used type (i.e. science, law, politics) to achieve a balanced data mix.
Tokenizer size matters
The tokens vocab of a model is the collection of all tokens that the model uses. Llama2 had a 34,000 tokens vocab, GPT-4 has 100,000, and 4o went up to 200,000. Llama3 went up 4x to 128,000 tokens. You can find the GPT-4 vocab list on Github.
This is something that people gloss over, but there are many reason why a large vocab matters:
* More tokens allow it to represent more concepts, and then be better at understanding the nuances.
* The larger the tokenizer, the less tokens you need for the same amount of text, extending the perceived context size. In Llama3’s case, that’s ~30% more text due to the tokenizer upgrade.
* With the same amount of compute you can train more knowledge into the model as you need fewer steps.
The smaller the model, the larger the impact that the tokenizer size will have on it. You can listen at 55:24 for a deeper explanation.
Dense models = 1 Expert MoEs
Many people on X asked “why not MoE?”, and Thomas’ answer was pretty clever: dense models are just MoEs with 1 expert :)
[00:28:06]: I heard that question a lot, different aspects there. Why not MoE in the future? The other thing is, I think a dense model is just one specific variation of the model for an hyperparameter for an MOE with basically one expert. So it's just an hyperparameter we haven't optimized a lot yet, but we have some stuff ongoing and that's an hyperparameter we'll explore in the future.
Basically… wait and see!
Llama4
Meta already started training Llama4 in June, and it sounds like one of the big focuses will be around agents. Thomas was one of the authors behind GAIA (listen to our interview with Thomas in our ICLR recap) and has been working on agent tooling for a while with things like Toolformer. Current models have “a gap of intelligence” when it comes to agentic workflows, as they are unable to plan without the user relying on prompting techniques and loops like ReAct, Chain of Thought, or frameworks like Autogen and Crew. That may be fixed soon? 👀
The whole podcast was a lot of fun to record, as usual you can find show notes and chapters below. Make sure to also subscribe on YouTube! 🙏
Full Video Podcast
Show Notes
* Thomas Scialom
* Recital
* Galactica
* Lucas Beyer - Citation Generator
* Llama 2 paper
* Guillaume Lample
* Hugo Touvron
* April 2023 Llama 3 release
* Llama3 Repo
* Chinchilla trap
* Agents research
* Thomas’ paper: Augmented Language Models: A Survey
* GAIA: Gaia General Assistant Benchmark (we interviewed Thomas at ICLR on this)
* Toolformer paper
* JEPA
* Clementine Fourrier episode
* Nathan Lambert episode
* Noam Shazeer
* Optimizing AI Inference at Character.AI aka Shazeer et al 2024 - we misspoke and said “native FP8” when we meant INT8
* The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
* Mentioned Papers
* MobileLLM
* SmolLM
* Overleaf
* AlphaGo
* Lindy AI
Timestamps
* Song credit: Code of the Future via Udio
* [00:00:13] Introducing Thomas
* [00:03:18] BLOOM and Meta Galactica
* [00:06:33] Leading Llama 2
* [00:09:56] Going 100x Chinchilla Scaling Laws
* [00:12:15] Open Sourcing Llama 3 405B
* [00:14:29] Quantization with INT8 / FP8 / Ternary (1.58 Bits)
* [00:16:58] MobileLLM, SmolLM, On Device Models
* [00:17:36] Llama 3 Architecture
* [00:18:33] Llama 3 Tokenizer: 128k and beyond
* [00:23:12] Synthetic Data for Pretraining
* [00:25:08] Synthetic Data from Augmented Language Models
* [00:27:19] Data Mix and Continual Pretraining
* [00:29:16] Adding Code, Reasoning, Multilinguality to Llama 3
* [00:30:39] Nvidia Nemotron and dedicated SynData Models
* [00:31:30] Why no MOE?
* [00:32:23] RLHF: Humans as Discriminators > Annotators
* [00:38:37] Teacher Forcing/Critique
* [00:42:02] Llama 3 Benchmarking
* [00:45:24] Llama 3 Arena ELO
* [00:47:27] Calibration Evals
* [00:49:23] Function Calling
* [00:50:17] Llama 4's plan for Agents
* [00:55:09] The State of Variable/Long Inference Research
* [00:57:19] Llama 4 Focus
* [00:59:15] AI Startups
* [01:03:34] Call to Action - Hiring
Transcript
Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO-in-Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI.
Swyx [00:00:13]: Hey, and today we have a very special episode with Thomas Scialom. I don't know how to describe, you've done so much work in a very short amount of time at Meta, but you were most notably leading Llama 2 and now today we're also coordinating on the release of Llama 3. So welcome.
Thomas [00:00:28]: Thanks for having me.
Swyx [00:00:29]: So let's play obviously the Llama 3 405B. Is that the official size number that we're going with, or do we just say 400B?
Thomas [00:00:37]: For the text model only, yes. A bit of additional parameters for the multi-model version that will come later.
Swyx [00:00:44]: Awesome. Just to quickly go over your background, actually we had a slightly similar past. I was also a quantitative trader and it looks like you did five years in QuantFinance, working a trading timer in SockGen, and then you transitioned into natural language, getting a PhD at Sorbonne. Working on Recital as well. And then right after your PhD, joining Meta.
Thomas [00:01:04]: No, it's exactly that, but basically I think it's at the AlphaGo moment where I was doing some trading. I say like, what I need to understand, what's the technology behind that? And I wanted to study machine learning. I did first some training, like six months degree, executive degree, at the end of which I knew like what XGBoost at the time, and nothing about deep learning at all. And most of the people around were like PhD people, and I was like, okay, PhD seems pretty cool, deep learning seems pretty cool, so I want to do a PhD in deep learning. That's where I joined, we have this PhD program in France within a company and academia. And so I did my PhD with Recital and Sorbonne University on natural language generation reinforcement learning. I guess it was a good topic. I was not like a visionary. It was very random. I've had a company that offered me this topic, and it was something like I started two weeks before BERT. Excellent timing.
Swyx [00:02:03]: Yeah. We actually also just released our episode with Clementine Fouquier, who also did her PhD with a company in kind of like a very similar format. I think, yeah, very underrated, very underrated, this sort of PhD with industry expertise, because you're also publishing papers the whole time. I looked at your publishing history, you were doing summarization work, you're doing factual consistency work, you released some benchmarks, and then you worked on language GANs before the transformers took over.
Thomas [00:02:31]: We can come back to that later, but I should have, I mean, papers have like 10, 50 citations. If I'm pretty sure that if I call them like, RLHF without human in the loop, but like a discriminator which is synthetic human in the loop, I will have get much more citations today. And all the inspiration for this paper were from actually the original open-air paper of RLHF. But at Academia, we don't have the way to pay annotation online like that. So how to simulate it? Yeah.
Swyx [00:03:06]: A lot of these ideas are repeated, like discriminator, generator, we just call them different names now, like verifier, whatever. Well, I think your progress into NLP was like really strong, because like the first thing you worked on at Meta was Bloom.
Thomas [00:03:17]: Yeah, actually, I started to work on that before joining Meta. I was not like one of the main contributors, but it was at the intersection of multilinguality, which was very important to me, large language modeling. And that's why actually my first big project at Meta and the team I was working on was Galactica. And actually, an interesting step back from Bloom was like, we did a lot of mistakes, but it was expression that's expected, and we learned a lot. But like trying to scale towards like multilinguality, in fact, we learned later that multilinguality almost emerged naturally with very, very few data, which was really surprising and not expected at all for us at the time.
Swyx [00:03:57]: I mean, my learning from that is just there's a natural harmony of language that is abstract from English. When you learn English, you learn language, and then language just translates to other forms of languages, especially if they're the same family, right? So maybe we should get right into Llama 2, spend a little bit of time there, and then we'll go into Llama 3. So like, what is the story of Llama 2 from your point of view?
Thomas [00:04:19]: Yeah. So as I was saying, I started to Meta on Galactica, that was one of the first large language model at Meta. It's a language model for science. We released it in, I think, December or November, I don't remember, one year and a half ago. I don't know if people remember, but it was huge on Twitter, both with people like thinking it's the end of science, and like that with a lot of hallucination papers, all those were like, it's super awesome. I still think it was super awesome, but, you know, we didn't do like instruction tuning or LHF techniques at the time. It was a weird moment because two weeks later, ChatGPT came out. And that's a moment where like, I think all the thing companies went upside down and where we had a huge traction from leads to now work on that and make a ChatGPT as soon as possible. So we had this one, two months of like, what to do, actually was working on Galactica Instruct, which basically you could connect it, we had a partner with Overleaf, the Google Doc of like scientists, where you can write papers. And you're right there in LaTeX, you have to do a lot of citations. So the idea was that you can just like ChatGPT or GPT Instruct, ask or swap two columns in a LaTeX table. That's something very, very time-consuming, I can promise. You could like say, oh, find me a citation about LLMs and bias, we'll find you some papers, insert automatically the bib in LaTeX. So that was pretty cool. But because of the backslash, we never like opened it in the end.
Swyx [00:05:49]: Oh, because the Galactica backlash. Oh yeah. Yes. Like I was just saying like, today it's not solved because Lucas Bayer is still asking for this citation generator.
Thomas [00:05:57]: I saw this tweet, I was, dude, we had that two years ago. And I promised, I tested it, it works so well. I had it on Overleaf Integrated. I tested it.
Swyx [00:06:07]: Wow.
Thomas [00:06:08]: Okay. Yeah, yeah, yeah. No, it went quite far, in fact. And actually about citations, like it's anecdotical, but because the way Galactica was trained to cite papers with all the references in paper, that's what made it emerge so easily at instruction time. Actually, Galactica Instruct was the first annotation project for RLHF at Meta. It was a follow up of Galactica that we were preparing. And at the same time, my friends from Paris office created Llama1. It's like to connect the dots with what we said before, the last author was Guillaume Lample, who founded Mistral. The first author is Hugo Touvron, who worked with me on Llama2, still at Meta, and both did a PhD program within Meta as a company and an academia. So that's a pretty good program indeed. And so we worked on Llama2 from that point. We had all the support from the company leadership. That was one of the main priority. We had Llama1 and Galactica as like backbone of good language model. We started from Llama1 and we worked mainly with Guillaume on how to make instruction following and chat models that will follow instructions. So all the supervised fine tuning stage, then the LHF, there are some papers. So you had some intuition from there we could use. But in fact, at large scale, and that was probably the most challenge for us, there's no research anymore. We don't know how much to scale.
Swyx [00:07:34]: Can you describe what scale you're talking about?
Thomas [00:07:36]: Yeah, yeah. To what level of annotation to scale is annotation like, do you need 100,000, 1 million, 10 million annotations of supervised fine tuning, of LHF preference? We had no idea. What is the actual algorithm to do? How often to retrain the models? You have just the basic, but then when it comes to like chat GPT or GPT instructor cloud, no one published the details there. And so we had to reinvent the wheel there in a very short amount of time.
Alessio [00:08:03]: And what about parameter size? This is one question that a lot of folks had about LlamaTree. So Llama1, you had 7b, 13b, 33b, 65b model sizes, and then Llama2, 7, 13, 70. How do you kind of evaluate what's worth training, especially when you think about data? Maybe 100,000 is enough for like a 7b model, but it's not enough for a 70b model. How do you decide model size, especially when you're maybe annotation constrained on some of these things?
Thomas [00:08:32]: That's a very good question, and there's no good answer. There's so many parameters to take into account from the scaling loss, training time to get the best performance, the GPU constraint, and on what different hardwares, and we think about meta, but also of the community, and people are not just using 800, but there's 800, there's different size of GPUs memory. So which size will fit in what, and what is the most useful? Also at inference time, not just at fine tuning time, then you can maybe do some tricks at inference time to quantize it a bit, or FP16 or FP8 now. All those constraints makes it very, very challenging. At inference time, you have a lot of costs. So how to trade off between inference costs and training costs? It's a very challenging problem. In general, we tend to think, in particular for Llama 3, Llama 2 maybe I would say it's like Llama 1, we had a flagship model which was 70b, it's also because the project was taking some routes to reproducing Chinchilla, which was a 70b. For Llama 3, we also moved to one size more, the flagship model for 0.5b. I think there was also the question of, we want a model at this time, we have this amount of compute, given the scaling laws and the amount of tokens we have to train it. What would be the right balance to still fit in at inference time? So we try to have some trade-offs like that. Yeah.
Alessio [00:09:57]: You mentioned Chinchilla is the best way to go, but then you tweeted recently, don't fall into the Chinchilla trap if you want your model to be used by billions of people. So what's the updated state of scaling loss? I think there was obviously the Kepler, and then there was Chinchilla, and then people kind of got the Llama scaling law, like the 100 to 200x parameter to token ratio. What's your updated thinking on how to think about scaling loss when you get model size and training data?
Thomas [00:10:24]: Right. So, you know, as you said, this Kepler paper with scaling laws, but they figured out, basically they tried two dimensions, the model weights and the number of training time, like number of steps, training tokens, epochs. And for that, they figured that model size is what matters. So GPT-3 was way too big compared to the actual number of training tokens because they did a mistake, not adapting the scheduler. That's what Chinchilla emphasized and discovered. To be fair, I think OpenAI knew that at the time of Chinchilla paper, but yeah, basically Chinchilla said we have to revisit the scaling laws originally published by Kepler and emphasize much more the importance of training tokens. And they did like some really good scaling laws showing that there's an optimal, basically you need to double the number of training tokens every time you double the training weights to get an optimal ratio so that for a finite number of compute, you will end with the best results in your paper. And what I call the Chinchilla trap is that, that's good if you want the best flagship model that obtains the highest performance on your paper. But if you want to use your model at inference time, inference, the two dimensions, one remains the model weights, but one drops the number of tokens you train it, number of steps. And so to be compute efficient at inference time, it's much better to train it much longer training time, even if it's an effort, an additional effort, than to have a bigger model. That's what I call, I refer to the Chinchilla trap. Not that Chinchilla was wrong, but if you can see your inference time, you need to go beyond Chinchilla. And in fact, that's what Llama1 folks did by overtraining in the sense they could have get a better performance in paper, but they prefer to create the best artifact that will be used by the community.
Alessio [00:12:15]: So that's the skinny thinking. What other went into LlamaTree kind of planning, you know, so LlamaTree, you have a pretty good model. People really liked it. So you drop like the intermediate weight. So it's a 870 and now 405B. What was the thinking behind going so large? I mean, you talked about the hardware capabilities at inference. Like I can now run a 405B model at home for sure. And it might be hard to even get the cloud resources to do it. What was the decision there?
Thomas [00:12:43]: The decision is super simple. We want the best model. We want to be number one and number two. We started one year and a half ago and we did quite some journey. We filled the gap with GPT-4. So that will be the first open source model that actually compares to GPT-4. There's now GPT-4o, of course. And we're close, but we're not there yet, not in all capabilities, but the gap is getting smaller and smaller. There's also like what compute we had at the time when we started to run in January. We put a lot of effort there, but as like Mark announced, we have more and more GPUs. So the next generation will be bigger. So that's what drives the decision. Now, maybe let me reflect two things he said. You cannot use it at home. That's probably true, but quantizing it to FP8 can run on Node, even with a long contact of 128K tokens. Second thing is I'm hopeful that the community will lead to a lot of findings by open sourcing it and there is a smart way to actually make you use it on your computer. If you remember Llama 1, Llama 2, like when we published models, people were saying it's too big. And after two weeks, it was running on a Raspberry. I don't know if it will be the same, but I hope it's the same kind of trend. And by releasing those models, we are enabling that. Now, the last thing I want to add is having bigger models enables us to collect better data, for instance, at LHF stage, because that's the model we use for the annotation. And so we distillate straightforward, like this annotation from this better model to the other models. So I can guarantee you that the quality of the smaller models we are releasing with Llama 3 are also thanks to having these artifacts where we can collect and train.
Swyx [00:14:27]: Yeah, there's a lot of really good info there. One thing I'll just briefly touch on for quantization. There was a recent Noam Shazir blog post. Noam is writing again for some reason, and he was talking about native FP8 training. It seems like that is most useful for inference. That is what you expect the open source community to do with your weights once you release them anyway. Is there any movement or thinking about just moving to FP8 or whatever other new format is in vogue these days?
Thomas [00:14:59]: Also, these papers like to train like some, I forget the name, but like there's two follow papers on like just a zero one or minus one weights. And like, there's a lot of work there. I think it's promising directions of all regarding FP8 in particular, those are the possibility for the community to try FP8 or other methods that are very easy at fine tuning time. So I'm really looking forward to what the community can do there. Overall, like scaling, I don't know if it's all you need, but I will not bet against scaling. And one of the ways to get more scale is by having better algorithms that we can train for the same level for less compute.
Swyx [00:15:40]: Less compute and less memory. Yeah, like inference time memory is becoming a real constraint.
Thomas [00:15:46]: Yeah, but also training with FP8. If you're not training with FP8 or I mean, FP0 is probably nonsense, but to what extent, how far we can go, you know? And every time like you unlock compared to what we had two, three years ago on a 32 or 64, it's like huge progress in terms of scaling.
Swyx [00:16:05]: For me, it's interesting to say, to see you mention the ternary quantization, like the 1.58 bit thing. Because I didn't know that, I don't know how much to believe, you know, like there's a lot of these kinds of papers where it makes a lot of noise, but it doesn't actually pan out.
Thomas [00:16:20]: It doesn't scale. I totally agree with you. It's so hard for researchers, at least for me, to see all those papers published, all those cool ideas, all those results that are preliminary. And in all this massive amount of research, what will scale or not? What will resist the test of time or not? And are we like losing maybe some gems that are not just, people are not working on them, but because there's too much research around, I don't know, maybe. And that's like some problems to have. That's cool to have these problems nowadays compared to probably what Yann LeCun and the others had 30 years ago, but still it's a problem.
Swyx [00:16:58]: You know, for what it's worth, like I do think that FAIR is putting out like incredible research, you know, probably it doesn't seem like it's your group, but you know, you also recently published Mobile LLM, which on the small model side is a really good research on just small model architecture that it looks like Hugging Face is also replicating it and it's doing quite well. Like, you know, there's a lot of ideas on shared weights and shared matrices and, you know, model architecture stuff that we can talk about for smaller scale models. Like Llama is not at that scale, but it seems like one of the big themes of this year is like on-device, in-browser, small models that are like good enough for daily use. I do want to talk about architecture, right? Like I'm not sure when you're releasing the Llama 3 research paper, but in Llama 2, you talked a little bit about the architecture choices, like in any...
Thomas [00:17:45]: It will be released the day I think of the release.
Swyx [00:17:48]: Okay. What should people know? What are the major choices of Llama 3 versus Llama 2?
Thomas [00:17:53]: There's not like a lot of changes in terms of architectures. I think we can do a lot better in the future and not just like with transformers, but for instance, to me, like it doesn't make sense to use the same amount of compute per token for every token. Like there's architecture lack of flexibilities. There's a lot of research to go there, but still that's the best thing we have for now. And so it's the same recipe than in terms of architectures and training than Llama 2, but we put so much effort on scaling the data and the quality of data. There's now 15 trillion tokens compared to 2 trillion. So it's another venture there as well, including for the smaller models.
Alessio [00:18:33]: One of the things I noticed on the paper is that you use Llama 2 to do the data cleaning for what went into Llama 3. I think there's a lot of chatter obviously about synthetic data and like there was the Rephrase the Web paper that came out maybe a few months ago about using, you know, Mastral to make training data better. Any learnings from that? It's like, is there, how much can you rewrite with the models? Like I'm sure people would love to hear more about it.
Thomas [00:18:58]: Right. So it's very interesting, the research direction. Synthetic data in general, synthetic data for pre-training. My intuition is that the web is full of s**t in terms of text and training on those tokens is a waste of compute. Just having a good classifier that labelize that is cool. And Llama was at the time, before Llama 3, the best model we had access to legally to labelize the web and select what are the good tokens and the bad tokens. The additional thing is that it also enabled to have a topic tag, like, is it about law? Is it about politics? Is it about chemistry, math, reasoning? So that you can also adapt a bit the mixture to like balance a bit more the diversity.
Swyx [00:19:48]: To me, you know, I'm not exactly sure what you guys did, but like, I feel like when people say synthetic data, there needs to be different categories of synthetic data now, because I think there's so many different usage of this thing. But specifically synthetic data for pre-training, it feels almost like you're running multiple epochs on the raw data while it's rephrased or reformatted by a language model, right? And in my mind, it's very similar to computer vision, where you do data augmentation on an item, right? Like we're doing data augmentation. That's the less cool name for synthetic data.
Thomas [00:20:23]: That's very interesting. I totally agree with you related to pre-training, totally stamp what you said. I think it's very different though for post-training and the future direction on synthetic data that I'm personally excited. Like for instance, what I'm excited about is we had this survey on augmented LLM a year ago. And all the idea is like, if you augment your LLM with something else, it can be a retriever. It can be search. It can be a tool. It can be a calculator. It can be a code execution. Then you are not just doing some data augmentation with your model, but you're actually adding some expert skills that possibly goes beyond the model weights. For instance, if your model can calculate something it was wrong before and now it has access to a calculator and you can retrain your model on that, then you're learning something new. If your model didn't know something about LLM 2, probably doesn't know a lot about LLM 3. You can search online about it and then you train the model on that. Then you have a positive feedback loop, like what we call expert direction, targeting directly the weakness of the model. It's like continual augmentation of the language model, much beyond just data augmentation.
Swyx [00:21:35]: How related is this to tool use? Are you teaching it to use tools to augment the model or are you saying, do active learning, where it's weak, go augment the model with extra data and then memorize that new data?
Thomas [00:21:50]: What I said is more like in terms of directions, not for LLM 3, but when it knows how to use a tool and correct itself, this is a very promising direction that goes much beyond augmentation in the future. To keep collecting new data and new tokens, people are saying we are lacking of tokens, but if you think about those kinds of tokens, where the model always goes to correct its own weakness, it can say, that's 10 plus 10, that's an easy example, probably the model knows, but imagine for something more complex, 10 plus 10, I expect this to be 20. Let's verify with a calculator, which is easy for a basic agent now, powered by LLM. And then you verified with respect to what you expected, that it's correct. If it's not, you can back propagate this example directly to the weights and so they will keep learning new things. It makes sense.
Swyx [00:22:40]: What have been your insights? You know, you mentioned about just like using calculators. What have been your insights? I think just in general, a lot of that is just driven using code generation and apart from just tool use. What have been your insights on just like the data mix of how much code, how much multilinguality, which is something that you're also passionate about? We know that that's changed between LLM 2 and LLM 3. Is it changing for different stages between the different sizes of LLM 3? Like, you know, anything like of that sort?
Thomas [00:23:08]: No, it didn't. For the different size, we use the same mostly. What happened is we changed the data mix during the training of LLM 3 with some findings that happened. I mean, training is long, so you have to do something while it's training. And what the team did, I was working on my side of multi-motion post-training, but so the pre-training team did quite a lot of work to have some new findings, improve the data mixture along the way, and they intersected before the end of the training.
Swyx [00:23:35]: I sense a movement in terms of like the curriculum that people are adopting during pre-training and even post-training about, you know, what the mix should be. Like Snowflake is doing some interesting work with enterprise intelligence or whatever they call it. What are your goals with post-training? Like just at a high level, you know, like what do you work with like the pre-train team?
Thomas [00:23:55]: I think it's quite easy for now because there's not yet like this kind of continual augmentation where it could feedback like pre-training, things like that. One of the big continuum between pre-training and post-training in particular is continual pre-training, where you actually continue the pre-training before RLHF in a self-supervised way but on expert level domains, like to have an expert in code, an expert in like reasoning or an expert in multilinguality that enables to collect even better RLHF notation after. So that's one thing. And then you start from those models to actually do the RLHF stage. And goal about your question, like goal was to get the best model in those dimensions. That's actually one thing very different to, I can comment, compared to LlamaT-II. LlamaT-II, you know, as I said, we were nowhere. We build entirely end-to-end all the stack from data notation, contract, methodology, protocol, algorithms for RLHF at Meta. And we had to limit our scope. We were like not allowed to work on that. We focus mainly on helpfulness, following instructions for LlamaT-II. And you can see that as in the following months after LlamaT-II, a lot of open source models came, distillating GPT-4 mainly, but obtaining better reasoning, math, coding, chat models. And we didn't annotate at all for code, neither for reasoning or multilinguality. And one thing I'm quite proud is with the early preview release we did of LlamaT-III back in February, May or March, I don't remember, it led quickly to instantly to state-of-the-art results for the model size, almost competing with GPT-4 on the Arena leaderboard, where humans fight each other, compare two models and select their preference. And no one since then had been able to put a LlamaT-III model better than what we did on most of the domains, from code, reasoning, multilinguality, helpfulness. So that's the sign that this time, as opposed to LlamaT-II, we tackle all those different aspects.
Alessio [00:26:01]: Talking about model distillation, this is the million dollar question. Can people train on the LlamaT-III outputs? And do you think, especially at this size, you know, maybe people will not be able to run inference at scale, but you can use it to improve some of the smaller models?
Thomas [00:26:14]: I don't think I can answer. There's, it might be, no, but it might be MIT license. It's not decided yet. I just don't know. Yeah.
Swyx [00:26:22]: Yeah. It used to be like a special LlamaT license. And then now there's like this restriction on like, if you would have a derivative model, you must call it like LlamaT-III as a prefix or something.
Thomas [00:26:32]: Right. Yeah. If you want, I can answer that. But if it's, I can re-answer that if you want to, but if it's MIT, it changes a lot. Cool.
Swyx [00:26:41]: Yeah. We love just Meta's commitment to open source and, you know, you do what you need to do to make it work for your organization.
Alessio [00:26:48]: Do you have any other thoughts on the more synthetic data focused models, kind of like a Nemotron? I think folks were asking if you see that as an interesting direction to kind of having specific synthetic data generation things.
Thomas [00:27:02]: I don't know about this model exactly, but I think like LlamaT had better performance overall. I'm very bullish on synthetic data generation, but I think just gets better when you have a better model. I'm not really bullish on having like a model only for synthetic data generation. I understand the need of having like bigger models, but then you can rationalizing, yeah, maybe people will not use them for inference, but to distillate some specific knowledge of synthetic data. That narrative is, I think I totally agree with that, but having a model purely for that and not like good at other things, I don't think it's the case.
Swyx [00:27:39]: That makes sense. One of the architecture questions that I forgot to mention in there was, so just the architecture choice of like a very big, you know, 400B dense model, I actually honestly thought that maybe 175 or like, you know, was kind of the peak, you know, whatever can fit on like an H100. So basically I think the common question that people have is like, why no MoE? In a way that Mistral and the others have gone and, you know, it seems like the trend has been MOEs and you guys have bucked the trend there.
Thomas [00:28:06]: I heard that question a lot, different aspects there. Why notMoEin the future? The other thing is, I think a dense model is just one specific variation of the model for an hyperparameter for anMoEwith basically one expert. So it's just an hyperparameter we haven't optimized a lot yet, but we have some stuff ongoing and that's an hyperparameter we'll explore in the future.
Alessio [00:28:31]: Let's make sure we run through everything on post-training. You also had a recent tweet about RLHF versus imitation learning explained in one tweet. So we'll put this in the show notes, but it's basically like two charts about a doctor opinions. On one side, there's like whether or not the suggestion is good from like a content perspective and the chatbots rank really highly and the physicians are kind of like, you know, a bell curve as you might imagine. But then the empathetic voting, most physicians are rated not empathetic or slightly empathetic versus all the model responses are rated very empathetic and empathetic at worst. You know, most people might look at it and not really get much from it, but obviously it resonated with you. Can you run people through like some of the choices you make in post-training to like optimize for one of the two and getting the best responses?
Thomas [00:29:20]: I think the tweet was about like the intuition of why reinforcement learning with human feedback works. When we started Llama2, I had like this budget of annotations in millions of dollars and okay, what to do? I'm responsible of that, I'm accountable for a model at the end that can follow instructions and compete with GPT-3.5 at the time, what to do? You can annotate supervised fine-tuning data, which refers to a human to create a prompt and to also write himself the answer expected by the model. So then you train on that and in a supervised manner, that's like very classic and standard on fine-tuning machine learning. The other thing is reinforcement learning with human feedback where the annotators type a prompt, but this time you sample two different answers from your model and you ask the annotator which one he prefers and then you will train on the preference basically to simplify. When you ask to train on the preference of the model, that seems very weird and not really robust training on synthetic model by the model. So I was like, let's annotate 100,000 more of supervised fine-tuning data and let's annotate a bit of preference to do a relationship because everyone is doing it. And we had this human evaluation after a few weeks in a Llama2 project where our model was already better than the annotation from the humans. So you'd get a prompt, you check what the human will have annotated as an answer, you check what the model generates and most of the time the model was better. I was like, oh maybe the annotators are pretty bad, let's look at that and no, like the model was pretty good. So I understood the intuition behind LHF, like those models are already super good at some tasks and with LHF then what you have is, imagine a distribution, a Gaussian distribution which was like basically the tweets and you have on the left like bad outputs and on the right good outputs and the same like medical diagnostics from a doctor. You have good outputs on the right and the bad diagnostics on the left, but you have the distribution then when you collect all the diagnostics from doctors, hopefully it's mostly on the right, there's better, a lot of time good diagnostics, but human makes mistakes, right? So there's bad diagnostics. On the left you have still a bit of examples which makes like curves not at zero, the distribution. And the same way for humans, like they make mistakes when they annotate and so training on behavioral cloning to reflect humans, the model will learn to do also some mistakes just like humans. And so you will have some bad outputs from the model time to time reflecting humans and you cannot go beyond that if you train on human outputs. But now if I ask a doctor to check a sample from my model or a sample from two doctors, one diagnostic and another diagnostic, one is better than the other, it's easy for a doctor to say which one is better. The same way if I sample from my model that learns a human distribution of answers and there's one bad time to time like humans but most of the time good answers. And I ask a human to choose which one he prefers. Personally I'm really bad at creating poems, the example I give a lot of time, try to write a haiku in three lines of about language models. I don't know you, take like five seconds to think what you could come up with, I'm terrible. But yet if I check two poems generated by a model or human, I can tell which one I prefer. I'm good at discriminating. And because of that you can have a model that flats the bad outputs and learns to only shift towards the best and better and better outputs. And you can even end to superhuman abilities since that I'm bad at writing a poem but I'm good at judging which one is better. So I can actually annotate data beyond my own skills at creating them. That's the magic of RLHF.
Alessio [00:33:07]: We have one episode, RLHF 201, with Nathan Lambert from the Allen Institute who was at HuggingFace leading RLHF before. And he mentioned one of the things that makes RLHF work is that humans are not maybe great at creating a lot of things, but they're usually very good at giving an opinion on which one to they prefer. So they're able to actually annotate data of things they would never create from scratch. One question actually that he asked me to ask you, how much in post-training you attribute improvement to the RLHF side versus the instruction fine-tuning side and maybe how you think about prioritizing the two and what areas they impact the most?
Thomas [00:33:44]: You mean between supervised fine-tuning like supervised fine-tuning annotation and preference annotation? Yeah. So 100% to RLHF. In fact, that's quite interesting. You start for Llama 2 with a pre-trained model and you have to have an instruction model to chat model. Otherwise, like the model is just like continue finishing sentences. So you need that to start RLHF. So we had to annotate like 10,000 examples. What did we do for Llama 3? You start with a new pre-trained model and then you want, before starting the RLHF, to have now a chat model, which is not too bad. The option one was, let's do human annotation again, like SFT stage. But in fact, by the principle I said before, the annotation would be actually worse than Llama 2. So what we did is that we generated all the data on the prompts with Llama 2 and we applied like basically the last round of Llama 2 we had to kick off and start Llama 3 post-training. So Llama 3 post-training doesn't have any like human written answers there basically, almost. It's just leveraging pure synthetic data from Llama 2.
Alessio [00:34:45]: Do you have an intuition on which areas work better for which? For example, you mentioned the physicians are expert. What about maybe like code or, yeah, you also have a multi-model working on, so like image generation is like, or does this apply to any modality, any subject?
Thomas [00:35:00]: That's an open research question. The intuition in general is like, for instance, for code, because this is factual, you can check if the code is correct or not, RLHF is not the way to go. You prefer to do like supervised fine tuning as a human to write the code. But in fact, because humans make mistakes, because actually even in code, there are some preferences that emerge like that. And maybe for some other reasons that we don't know, RLHF is so much more scalable. It costs less, it's easier, that it leads in general to just better performance. And maybe we can come with a compromise. We actually suggested teacher forcing in Llama 3, a new method that kind of fills the gap between, not teacher forcing, sorry, teacher critic. Teacher forcing is a good way to train the models. Teacher critic where it reconciliates and unifies supervised fine tuning and RLHF, so that when you do human preference, and you have two outputs, but both are very bad in the code, for instance, you will ask the human to edit the best answer to make it correct now. So now you are doing SFT when all the answer was really bad, so that you can get out from the local minimum of your model.
Swyx [00:36:05]: I think this is like super promising and it seems like there's just, well, do you have an idea? You know, you started with this question of how much scale you need, do you now have a better idea?
Thomas [00:36:15]: No. What we know is it's not plateauing yet.
Swyx [00:36:19]: It's not plateauing yet, yeah. So just infinite amounts more, well, you know, scale AI and all the annotation providers are very happy to hear that. So we mentioned at the start of the conversation about the AlphaGo moment, and I feel like this is very interesting to reflect on, right? We're basically saying that, I think that one of the lessons from AlphaGo is that people thought that human interest in Go would be diminished because computers are better than humans. But then we have this sort of centaur model where humans and computers are actually doing better than either humans and computers would be alone. And I think we're seeing that with this, what are you talking about, this RLHF improvement, right? That we're kind of building human preference into the model and the blending of the human preference and the model capability is actually doing better than we could on our own. I just think it's pretty fascinating.
Thomas [00:37:11]: It is fascinating.
Swyx [00:37:12]: The other thing is RLHF came from the alignment community. And I think there's a lot of conception that maybe it's due to safety concerns, but I feel like it's really over the past two, three years expanded to just this produces a better model period, even if you don't really are not that concerned about existential risk. I always feel like it's so interesting to see this, like people who take alignment super seriously, they're the first to consider super alignment. And now we're considered like, I'm almost thinking about this as like super quality, that we are training models that are higher quality than humans. And it's not really about alignment so much as like, we now see that this is actually possible. Yeah. And it's not even for alignment purposes. We just think it's better at reasoning, better at knowledge, better at everything.
Thomas [00:37:59]: Well, I don't know how much better yet it is on those, but clearly it's super human on some writing skills and it's super useful. I think that's great, to be honest.
Swyx [00:38:08]: Yeah. Perhaps we can transition to evals. We've had some questions about the 400B details that we want to disclose, you know, by the time this podcast comes out, you know, we'll have disclosed them. Yeah. I think last time you disclosed like the evals while you were still training, what should people know about the high level headlines for the new Llama 3?
Thomas [00:38:30]: At a high level, it's the best open source model ever. It's better than GPT-4. I mean, what version, but by far compared to the version originally released, even now, I think there's maybe the last clouds on a 3.5 and GPT-4.0 that are performing it. And that's it. Period. For the 405B, that's a flagship, that's a pretty good model. Not yet the number one. We still have a journey to get there. For the 7TB and 7B, they are like world-class models for this size, for general models.
Alessio [00:39:05]: And are the benchmark numbers from the initial checkpoint still right? So the April 15 checkpoint, MMLU on Instruct is like 86, GPUA 48, HumanEval 84, GSMAK 94, and that's 57.8. Is this still roughly the same performance or, you know, I haven't seen the numbers yet either. We're just breaking the news right now.
Thomas [00:39:28]: No, it's roughly that. Awesome.
Alessio [00:39:30]: So talking about evals, we just had an episode with Clementin from Hugging Face about leaderboards and arenas and evals and benchmarks and all of that. How do you think about evals during the training process? And then when the handoff happens, do you already know exactly what you want to improve? I know that, for example, to improve like maybe an arena score, you need different than like an MMLU score. How do you think about prioritizing the post-training improvement based on benchmarks?
Thomas [00:39:58]: That's a super hard and good question. There's no good answer. I mean, evals is an open research problem, like in particular when you're trying to tackle so many capabilities. And you know, it's also like as soon as a benchmark, you're trying to push numbers on a benchmark, it stops to be a good benchmark because then you don't know if you're overfitting it and it will transfer to similar capabilities. So evaluation for language models, in particular on post-training, is a very hard problem. We tackle that by playing with different methods like reward models, evaluation, model-as-a-judge, having a diversity of prompts, diversity of benchmarks as well for a lot of different capabilities. That limits the possibility of hacking them, of course. We do also a lot of human evaluation. I do also a lot of model test quality analysis, like testing myself some prompts. I feel it was much easier during Llama 2 when the model was like worst than today. Now the models are getting so good that it's hard to get to some prompts to break them and to compare models and see their edge cases. So it's getting harder. And a great way also to compare models is, you know, truth, the different rounds we have done for RHF. Every time we upload a new model, for all the annotation we are doing, we have the win rate between the previous model and the new model by just sampling for every prompt we annotate, sample A with the old model, sample B with the new model. So we can calculate automatically a win rate.
Alessio [00:41:33]: Interesting. What are areas that you had to work the hardest to catch up to like the private models? Maybe like there's, you know, not as good public data or whatnot, or is performance improvement just kind of even across the spectrum?
Thomas [00:41:46]: Honestly, all of them, we are behind all of them with between Llama 2 and GPT-4. I mean, it's different challenges every time. Like being good at code or reasoning is something we didn't do at Llama 2. So we had to build everything from scratch. Improving on helpfulness, which is one of the main dimensions that people look at, I think, in the arena, which is, by the way, a very interesting evaluation. Because when we did the preview, and I don't know yet what will be the results for this new Llama 3, but we ended very high in this blind test leaderboard. And to be honest, I didn't expect that. I knew we had good results internally, but how that will transfer to perception from the community, people like using it in practice and comparing it to the other models, I didn't expect that positive feedback. That's high ELO score on this benchmark. It doesn't say like everything, as I said before, which is also interesting, because it's a community that judge the prompts and create the prompts and judge the answers. We are limited. We are not like good to do that. And so it gives you a very good indicator of how good, helpful, how on the main core of the distribution, simple prompts about the tone of the model compared to the others. But for much more complex prompts, much more intelligent reasoning, coding of complex stuff, it doesn't tell the full story. You know, like while we had 7TB preview at the level of GPT-4, even better at the time, I think it was partly true. But clearly we were not at like GPT-4 level in code or reasoning, we are now.
Swyx [00:43:24]: There's some conversation about like the math score. I think the next GPT next or whatever has reached 90, which is a big, big jump from the current state of the art. It will be interesting. One of our previous guests, rounding out the topics on potential models, areas of development and evals, Clementine is looking for a confidence estimation or uncertainty benchmark. One of our previous guests, Brian Bischoff, is also asking about like, how do we think about evals for practical things like confidence estimation, structured output, you know, stuff like that.
Thomas [00:43:59]: Yeah, I think we lack actually of such evaluations. One of the numbers I was suggesting like two days ago to the team to report at some point is, okay, we have this accuracy on MMLU, on whatever, on math and JSM84. What if we change a bit the prompt and instead of telling the model you have this question, you have to answer A, B, C, or D? What if we tell the model you have to answer A, B, C, or D, or you don't know? And maybe the accuracy will be a bit lower, but I'm curious to see if some models we have different calibrations where maybe model A have 50% correct, model B has 50% correct, but model A answered 100% of the questions, so 50% are not correct. Model B actually said like, answered only 60%, so for 40% of the time he said, I don't know. I prefer model B. And we are not like reflecting that in evaluations.
Swyx [00:44:51]: I think this is very relevant for post-training in particular, because it seems that the general consensus is that base models are more calibrated than post-train models, right? Something like that. Exactly. That seems to be the research from OpenAI as well. I don't know the degree of this and maybe we can invert it, right? Maybe post-training can help to increase calibration rather than decrease it. I feel like this is a little bit of being too similar to humans because humans are not calibrated very well.
Thomas [00:45:20]: Yeah, and that's the goal of post-training, I think, to make models more calibrated, to not be biased to answering A, B, C, or D as often as possible, to follow the uniform distribution.
Swyx [00:45:32]: On the structured output tool calling side, do you think that it's not an explicit part of the evals? Obviously, you worked on tool former and the language augmentation, do you encourage the open-source community to fine-tune Llama3 to do tool calling, or do you want to just have that in the model from day one?
Thomas [00:45:52]: We have that from day one, good news for the community. We are state-of-the-art there. I think the model will be pretty good at that. We have a lot of gems about tools in the paper, but the model is fine-tuned to do tool usage, to zero-shot function calling. There are some system prompts if you tell the model to do, it can use a search and imagination, can do a lot of stuff like code execution as well, even in a multi-message way. So almost multi-step agents, which kind of sparks our agents. Okay.
Swyx [00:46:26]: You talked about agents. So I guess we should probably mention the work on agent stuff. And you also, in our pre-conversation, mentioned that you're already starting work on Llama4. What does agents have to do with Llama4? How does your work on Gaia inform all this work?
Thomas [00:46:39]: Yeah, you know, so we published one year ago, Gaia General Assistant Benchmark. That followed a direction I really like pursuing, I mean, everyone passionate about AI and trying to build Jarvis will go there. So I did Toolformer and the survey on augmented models. In fact, you know, reflecting back, I was, okay, we have Galactica, we have Llama1, we have Toolformer, and there's like GPT 3.5 at the time and Llama4. If you don't have a good instruct model to follow instructions, the extension and the future of Toolformer is limited. So we need to work on that. And we did Llama2 and then now Llama3. And it's very interesting. On General Assistant Benchmark, so Gaia, agents powered by language models perform to zero with GPT 3.5 and to something very significant, like 30, 40%, 60% with GPT 4. So there's a gap of intelligence here. And I think this gap of intelligence, this threshold that you pass in terms of zero-threat function calling, following complex instructions that can span over a page of constraints, those things that make nowadays agents with React loops, pre-planning, multi-steps reasoning, function calling, work in practice is like this gap of intelligence. So now that we have Llama3, I'll be back to agents, I expect some incremental and significant progress on pre-planning, post-planning, but I'm really hopeful that we can gain some order of magnitude of scaling by interconnecting well models into agents as a more complex system that can do planning, that can do backtracking, that can take actions, navigate the web, execute code.
Swyx [00:48:25]: Okay. There's a lot there. When you say integrating world models, is there anything from JEPA? Is that something that we're talking about, or is that a different line of research?
Thomas [00:48:36]: No, not directly. That's the same goal, I would say, but JEPA is very, very fundamental research, which has some promising early results. And what I was looking right now on state-of-the-art results on Gaia, there's a leaderboard, by the way, you mentioned Clementine before, she contributed to Gaia as well, and Huggingface puts a leaderboard there on their website. There's some state-of-the-art results. What is interesting is like GPT-4 alone has 0%, or like 5%, I think, on level one, that's three level of difficulties. But OSCOPILOT then, and Autogen from Microsoft, and recently Huggingface agent, obtains on level one up to 60%. So connecting an LLM to an agent that can do all those things moves much forward new capabilities. This is kind of a breakthrough. And those models are purely based on instruction tuning models, following instructions, where you have an orchestrator and you say to your LLM, okay, this is your task, you have access to these tools, you can navigate the web, can you do a plan of what you should do? And then, okay, that's the plan. Now execute the first step. Did you manage to succeed for the first step, or do you want to rethink your plan because you enter in a dilemma? And you have kind of all this orchestration by system prompting, instruction following, and just that, which is quite suboptimal and probably you need to go later in latent space and more JPAS time. But just that is getting us to some really impressive results already.
Alessio [00:50:15]: And do you see the planning and review to always be needed in the future? This is kind of like Andrej Karpathy's idea of like more tokens equal more thinking. But the more you're having it write tokens and think about the outcome and the better result you're probably going to get to, do you think that's always going to be the case? Or that in the future, the model, you can just say, this is the task, and then I'll just return the answer directly and do all of that in the latent space, so to speak?
Thomas [00:50:42]: Right. I think in the future, it should hopefully go more as this is a task and I return it. But we need to teach that to the model to train that, which is far from now. Every medium long-term direction that could be relevant here is thinking into latent space. I know some early works are doing that. And that's a way probably to move to first you think, and then you don't have to write all the tokens. Like it's in your head. It doesn't have to be as constricted than a plain text BLM. And once you have done your thoughts, you can just write the final answer or take an action.
Swyx [00:51:18]: Just a commentary on that. Anthropic actually cheats at this right now. If you look at the system prompt in Claude Artifacts, I actually have a thinking section that is explicitly removed from the output, which is, I mean, they're still spending the tokens, but before training it, at the prompting level, you can simulate this. And then at iClear, there was the pause token, the backtrack token. I feel like all these are token level stopgap measures. I feel like it's still not the final form. We still need to have, at the architecture level, some kind of variable inference length thing that lets you actually think in latent space, like you're talking about. I don't know if there's any papers that you're thinking about.
Thomas [00:52:01]: No, but that's interesting because that's what we said at the beginning of the discussion. If you remember, we are lacking flexibility for pre-training architecture transformers, where we spend the same amount of compute per token. And so because of that, how can you mitigate this? By generating more tokens, so more thoughts, more compute, because you have only access to this dimension. Ideally, you want an architecture that will enable, naturally, to make this emerge, basically.
Swyx [00:52:30]: Any papers come to mind there that you would recommend people read, or this is like completely new science that we have to do?
Thomas [00:52:37]: No, I mean, it's earlier science. I don't know any work that managed to get there. I know, for instance, Universal Transformer had this idea of a number, and you can compute on the layer n times, n being decided by the architecture itself with respect to the complexity of the token. I think there's a paper from DeepMind on a mixture of experts with a key player, a mixture of... Is it this one?
Swyx [00:53:05]: A mixture of depths.
Thomas [00:53:06]: I'm not sure if it's this one, maybe. But basically, the idea was that with a mixture of experts, you have an expert that is an identity matrix that you can skip. And so you can... But that's early works, very preliminary works. For instance, I haven't seen yet a lot like putting the compute, generating a token into the loss. That's going to be interesting when we start to do that.
Alessio [00:53:28]: I know we're getting up on time, but we have just a few more questions we definitely want to ask you. So as you think about... There were reports about Llama4 started training again in June. If you think about the evolution of the models, I think up until Llama3, with Meta AI and some of these things, I'm like, it makes sense that they want to build their own models and they're multi-modal. It sounds like Llama4, maybe a lot of the focus will also be a more agentic behavior and have all of this. I'm curious at what point it's like, okay, this is a research direction that we still want to take, even though it doesn't fit right into the product. What's that discussion internally about what to focus on as you keep scaling these models?
Thomas [00:54:04]: Yeah. I think it's a balance between, well, we want to be number one, Mark wants to be number one there. And there's this understanding also that this is a critical technology in the future. And even if nowadays that research, if nowadays it's not directly intersecting product, we don't want to be late in the game as we had in the past. So that's the first thing. The second thing is, we think that this technology will change the world. We want to work towards AGI and AGI will change the world. And if Meta develop an AGI, it will probably intersect pretty easily the products. Now the third thing is, with that in mind, we have to balance with product needs. And there's always this ongoing discussion and this balance to find for like between a flagship model, between maybe a model that will be more adapted to product needs. And it doesn't have to be decorrelated. As I said before, like you can leverage also the big models to distillate some capabilities to a smaller one that will be maybe more suited like research. There's always this back and forth. There's also the fact that the product kind of ideas to the research evaluations that are grounded in actual use cases, that we can also measure ourselves with respect to is there some progress or is it just on an academic benchmark, you know?
Alessio [00:55:24]: So one, before we transition off, I think there's the hidden side maybe of these LLMs that most people don't think about, which is the tokenizer and the vocab size, especially of them. So LLAMA3 is 128k tokens, vocab tokenizer, GVD4 was 100k, 4.0 is 200k. How should people think about the impact that it has? So basically like, I mean, the TLDR is like in the vocab, you have this kind of like concepts represented as tokens. So usually the larger the vocab size, the more nuanced the model can be about thinking about different things. What are the scaling laws of those organizers? You know, is 120k kind of like very large and it doesn't really matter. Like do you want to double it? Like any thoughts there would be great.
Thomas [00:56:09]: There's a lot of dimensions to take into account here. I think the first thing obvious to say is LLAMA3 compared to LLAMA2 is multilingual, has multilingual capabilities. We worked on that. And so because you have languages that are not just Latin languages like English, there's a lot of different characters. You want to include them to represent like special word there. And so you need to have a bigger vocabulary size. But the obvious thing, which is also probably why GVD4.0 has a much bigger vocabulary as it's like naturally multilingual, multimodal in speech. So that's why we went to from 30 to 128 vocabulary size. The interesting thing I think to discuss about tokenizer is both scaling laws related to that. If you increase your vocab size, you have a bigger matrix, which takes longer to compute. It depends on the model size. But for a small model, it has a much bigger impact than a bigger model. So increasing that, basically saying otherwise, the number of vocabulary size for 128 is the same than the 8, 70, or 405b, but so relatively in percentage of the total number of weights for the 7 bits, much more than the 405b, but it's small compared to the total number of weights. So that has more impact in terms of training speed there. But what is interesting is with a bigger vocabulary, for the same text, you have less tokens, right? And so you can train your model on the same amount of knowledge with fewer steps. So for the same compute, you can see more knowledge if you don't epoch. That's one cool thing. The second thing is at inference time, you know that the context line is not in the size of the text, but the number of tokens. And so you can compress more such that now with a bigger tokenizer, 128 more vocabulary, you can get to longer text for the same number of tokens, 8k basically, or 128k. Now with this tokenizer means 30% about less text to encode.
Alessio [00:58:23]: How are tokenizer vocabs built? I actually don't know that. What's the work that goes into it? And then like, why are people using smaller ones? Is it harder to make them or is it just about some of the things you mentioned around scaling the training and all of that?
Thomas [00:58:36]: Oh, it's no, there's different methods, but it becomes quite standard, although it could change in the future. BPE. Yeah, exactly.
Swyx [00:58:44]: Well, BPE is for text. I don't know about multimodal vocab, that's, I haven't read anything about.
Thomas [00:58:50]: Yeah. I'm not an expert there and I don't remember exactly what they ended to do.
Swyx [00:58:56]: Now that you're saying this, right, okay, so now we have 100k vocab, 200k vocab. Do we see a million vocab? Do we see infinity, which is no tokenizer, you know, like what's the natural limit of tokenization?
Thomas [00:59:09]: Yeah. That's a good question. I don't know. I think there's a limit with respect that we grow with respect to the model size. So bigger models means possibly bigger vocabulary without affecting too much the training. But yeah, there's a lot of people, that's not my domain of expertise, but a lot of people are discussing the interest of having this kind of tokenizer, which doesn't fit like natural. Could we go to character level tokenizer? Could we go to actually multimodal tokenizer, which will like decompose at pixel level? I don't know. Future directions that could be very promising.
Swyx [00:59:46]: I would say the diffusion people have actually started to swing back to pixel level and probably that will presage the language people also moving towards, you know, 1 million vocabulary and then, you know, whatever the natural limit is for character level.
Alessio [01:00:03]: I think we can maybe transition towards some of your personal stuff. We kept you here for a long time. We also, this is a very distributed podcast, you know, I'm in the Bay Area, you're in France, Sean is in Singapore, so everybody is on a different time zone. You also do, you know, some startup investing and advising, you know, we also meet Chantal on the podcast. He also mentioned he always enjoys kind of working with founders and researchers. Any company you're involved with that you want to shout out that you think is super promising, requests for startups that you've had, anything around that space would be awesome.
Thomas [01:00:35]: Two cool companies I can think now is, one is Lindy, which is based in the Bay Area with Flo Crivello. Yeah, yeah. Very cool one.
Swyx [01:00:44]: Yeah, he's a good friend.
Thomas [01:00:45]: Flo.
Swyx [01:00:46]: Why do you like it?
Thomas [01:00:47]: Flo is really good. Like he's a French master, I guess. And number two, very recently, I really liked Open Devin, which is basically trying to reproduce Devin.
Swyx [01:00:58]: We interviewed him at ICLR. Both are agent startups. What do you think is like the direction that startups should be working on, you know, agent wise, and maybe what is not working?
Thomas [01:01:08]: That's a tough question. One thing I say quite often is deep learning has these very specificities that makes it challenging to predict that it's self-destructive, self-destructive technology, since that thing like, you know, Grammarly, this technology like where the startup, you plug play and it corrects your grammatical errors. Everyone told them, guys, deep learning creates a barrier to entrance, annotate data, create data. And they had a lot of data for that. And the next day, with the same exact technology, deep learning, someone comes with JGPT and tell them, yeah, I can do the same, better, and so many other things. This is your barrier to entry from yesterday to today. And what is crazy here is that it's based on the same technology. And so there's a lot of people working nowadays to try to mitigate issues with current generation of models. And I'm telling them, like, assume always the next generation will get better. So if your business will benefit from a new generation with better abilities, that's a good business. If your business may be replaceable, and if all the work you have done may vanish and be like wasted because there's better models, then maybe change.
Swyx [01:02:22]: Yeah, I mean, yes, but better is so unpredictable. Like if you asked me before, let's say March of this year, I would have said that maybe, you know, voice chat is still very defensible. And then suddenly, you know, OpenAI demoed their sort of real-time voice thing, sort of natively multimodal.
Thomas [01:02:42]: It's easy to not anticipate the dimension where it gets better, but find another one that resisted, it's harder. I would say in general, assume you will have progress everywhere. It may not be right, but it's a bit dangerous to bet against that.
Alessio [01:02:59]: Is there any space that you think is overrated by founders that are trying to build something that like, yeah, either, you know, the new models are just going to do or like you just don't think there's that much interest from folks?
Thomas [01:03:11]: It's a challenging time for founders. It's very exciting. There's a lot of funds, a lot of applications as well, a lot of stuff to build. That's pretty cool. But what is hard is because this technology is moving so fast, I see like now a lot of fundamental stacks that are like the unicorn of today, for national models, for national like clusters, data notations, things like that. There's a lot, but less successful yet for now, at least, application company. And it's hard to build an application when it's so fast, as we discussed before. So it is both crowdy and yet like we haven't found a good like use case that is like the new thing company there. I want to see it.
Alessio [01:03:53]: Yeah, we definitely see the same, you know, all of our agent companies, or at least, you know, building agents are the ones getting the most traction. Most companies are like, hey, I actually don't have that much expertise and I'm just waiting for the models to get better. So I'm not really sure if I need this now. So it's an interesting time to be investors. Anything else we missed? This was kind of like a masterclass in how to build state of the art LLM. So it's going to be a highly, highly played episode, I'm sure. Any final thoughts you want to share?
Thomas [01:04:23]: There's two things I can, I guess I can say one is LLM is hiring talents worldwide. And two, you can contact me, reach me out on LinkedIn, looking for Gen AI technology that and founders that will create the future.
Swyx [01:04:38]: Okay, hiring one role that you're like, man, like, we really need this, this kind of person. If you describe it, that person will be will be referred to you, right? Because we're, we're trying to broadcast it to the whole world.
Thomas [01:04:52]: Researchers with good common sense, first principle thinking, not necessarily like huge expertise on LLM, but more being super rigorous, meticulous, structured.
Alessio [01:05:02]: Azzaman, thank you again for coming on and hope everybody gets to enjoy LLMA3 today since it just came out. And we'll have you again for LLMA4.
The first AI Engineer World’s Fair talks from OpenAI and Cognition are up!
In our Benchmarks 101 episode back in April 2023 we covered the history of AI benchmarks, their shortcomings, and our hopes for better ones.
Fast forward 1.5 years, the pace of model development has far exceeded the speed at which benchmarks are updated. Frontier labs are still using MMLU and HumanEval for model marketing, even though most models are reaching their natural plateau at a ~90% success rate (any higher and they’re probably just memorizing/overfitting).
From Benchmarks to Leaderboards
Outside of being stale, lab-reported benchmarks also suffer from non-reproducibility. The models served through the API also change over time, so at different points in time it might return different scores.
Today’s guest, Clémentine Fourrier, is the lead maintainer of HuggingFace’s OpenLLM Leaderboard. Their goal is standardizing how models are evaluated by curating a set of high quality benchmarks, and then publishing the results in a reproducible way with tools like EleutherAI’s Harness.
The leaderboard was first launched summer 2023 and quickly became the de facto standard for open source LLM performance. To give you a sense for the scale:
* Over 2 million unique visitors
* 300,000 active community members
* Over 7,500 models evaluated
Last week they announced the second version of the leaderboard. Why? Because models were getting too good!
The new version of the leaderboard is based on 6 benchmarks:
* 📚 MMLU-Pro (Massive Multitask Language Understanding - Pro version, paper)
* 📚 GPQA (Google-Proof Q&A Benchmark, paper)
* 💭MuSR (Multistep Soft Reasoning, paper)
* 🧮 MATH (Mathematics Aptitude Test of Heuristics, Level 5 subset, paper)
* 🤝 IFEval (Instruction Following Evaluation, paper)
* 🧮 🤝 BBH (Big Bench Hard, paper)
You can read the reasoning behind each of them on their announcement blog post. These updates had some clear winners and losers, with models jumping up or down up to 50 spots at once; the most likely reason for this is that the models were overfit to the benchmarks, or had some contamination in their training dataset.
But the most important change is in the absolute scores. All models score much lower on v2 than they do on v1, which now creates a lot more room for models to show improved performance.
On Arenas
Another high-signal platform for AI Engineers is the LMSys Arena, which asks users to rank the output of two different models on the same prompt, and then give them an ELO score based on the outcomes.
Clémentine called arenas “sociological experiments”: it tells you a lot about the users preference, but not always much about the model capabilities. She pointed to Anthropic’s sycophancy paper as early research in this space:
We find that when a response matches a user’s views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time.
The other issue is that Arena rankings aren’t reproducible, as you don’t know who ranked what and what exactly the outcome was at the time of ranking. They are still quite helpful as tools, but they aren’t a rigorous way to rank capabilities of the models.
Her advice for both arena and leaderboard is to use these tools as ranges; find 3-4 models that fit your needs (speed, cost, capabilities, etc) and then do vibe checks to figure out which one is best for your specific task.
LLMs aren’t good judges
In the last ~6 months, there has been an increased interest in using LLMs as Judges: rather than asking a person to evaluate the outcome of a model, you can ask a more powerful LLM to score it. We covered this a bit in our Brightwave episode last month as well. HuggingFace also has a cookbook on it, but Clémentine was actually not a fan of this approach:
* Mode collapse: if you are asking a model to choose which output is better, it will just self-reinforce its own preferences. It will also prefer models from its own family (i.e. GPT models will prefer other GPT models over Claude outputs). If these outputs are then used to fine-tune the model, you will further mode collapse the model. Cohere for example has said they do not train on any model-generated data to avoid this.
* Positional bias: LLMs usually prefer the first answer, so you can’t naively give them options and ask them to rank them, but you also have to mix up the order in which they appear.
* Don’t score, rank: rather than asking a model to assign a score to each output, you should have it stack-rank them. The models aren’t trained to score things, so even though they might understand what response is better, assigning a score to it is hard.
If you do have to use LLMs as Judges (we aren’t all ScaleAI-rich!), she suggested using an open LLM like Prometheus or JudgeLM to make sure you can reproduce those rankings in the future.
Show Notes
* Clémentine Fourrier
* Hugging Face
* OpenLLM v2 Leaderboard
* Let’s talk about LLM Evaluation
* Leaderboard V2 Blog Post
* Latent Space Benchmarks 101
* Gradient AI epsiode on Long Context Evals
* Allen AI long context novel evals
Companies and Organizations
* Anthropic
* Cohere
* EleutherAI
* INRIA
* ICLR (International Conference on Learning Representations)
People
* Aidan Gomez
* Dan Hendrycks
* Edward Beeching
* Haley Sholkoff
* Lewis Tunstall
* Nathan Habib
* Thomas Scialom
Projects, Models, and Benchmarks
* LMSys Arena
* ARC AGI Challenge
* Allen Institute ARC Challenge
* BigBench
* GAIA benchmark
* GPQA
* GSM 8K
* IFEval
* LightEval
* ML perf
* MMLU
* JudgeLM
* Prometheus
* RavenWolf
* SWE-Bench
* Vantage
Timestamps
* [00:00:00] Introductions
* [00:02:32] How Clémentine went from geology to AI
* [00:05:52] Origin of the OpenLLM Leaderboard
* [00:09:06] How v1 Benchmarks Were Selected
* [00:10:49] The Problem with Current Benchmarks
* [00:13:45] Saturating benchmarks and the future of evaluation
* [00:16:14] Issues with human evaluations
* [00:24:07] AI girlfriends as the multi-turn benchmark
* [00:25:35] What's New in OpenLLM leaderboard V2
* [00:28:12] Benchmark Answers Black Market
* [00:30:21] The impact of prompt formatting on model evaluation scores
* [00:33:30] Difficulty and Computational Constraints of Evals
* [00:36:28] The Responsibility of Setting Standards
* [00:40:35] The Economics of OpenLLM
* [00:44:15] Long context reasoning benchmarks
* [00:46:34] Agent benchmarks, GAIA, and the ARC AGI challenge
* [00:50:43] Vibe check for benchmarks
* [00:53:16] Request for benchmarks
* [00:56:48] v3 predictions?
Transcript
Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO-in-Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI.
Swyx [00:00:13]: Hey, and today we have a super special guest that we've been trying to book on the schedule for a while. It's Clémentine Fourrier. I'm trying my best to do the French, but maybe you can do a better job of it than me.
Clémentine [00:00:26]: This was perfect. It's Clémentine Fourrier, but your pronunciation was really on point.
Swyx [00:00:31]: There was a Fourrier, which is very sort of French intonation, which I don't really understand. So I'll introduce you off of your LinkedIn and I would love for you to fill in the blanks. You are currently a research scientist at Hugging Face and the maintainer of the OpenLLM leaderboard, which we'll talk about very shortly. Previously, you were at INRIA as well, but then it looks like you also concurrently got your PhD at the same time. How does that work? Is that a very common thing?
Clémentine [00:01:01]: So I basically did my PhD at INRIA, technically. So INRIA funded my PhD and PhDs in France are three years, but I also worked as an engineer at INRIA before my PhD, hence maybe the confusion.
Swyx [00:01:14]: I think there's a rise in universities having sort of industrial attachments to these things. And I think it actually makes for a much more grounded study, especially if you're doing your sort of graduate studies and all these things. I think it's rising in North America as well with Berkeley and with Waterloo in Toronto. Cool. Like, you know, there's, there's a lot of other things we can, we can introduce. I can't really pronounce the name of the, the university you went to, but what else should people know?
Clémentine [00:01:44]: So I actually, technically I'm an engineer in geology. So I studied rocks and I graduated in 2015 after having done like extensive studies about rocks. And I discovered I was very bad at it, but I was very good at computer science. So I went to computer science. What stuck with me though is that geology is very much an experimental science. And I think that machine learning is very much an experimental science too, even though people want to claim that it's pure math. And I worked on several machine learning projects throughout the years, a bit of the prediction of illnesses in the brain at Brain and Spine Institute in Paris. I worked as an engineer in a research team in NLP where I did my thesis and then I joined Hugging Face.
Swyx [00:02:32]: Do you have a favorite rock fact or sort of rock story before we get into the NLP stuff?
Clémentine [00:02:38]: Okay. I was not expecting this question.
Swyx [00:02:43]: I did my geography A-levels and I always loved learning about like isostasy and stuff like that where you have different plates kind of up and down in the mantle. And I don't think people think about vertical dimensions to geographical plates, but it's real.
Clémentine [00:03:02]: Yeah, definitely. And like when you do geology, the time scale is just not the same. There is like one specific place in France where you can see rocks that are 1 billion years old and like the sheer scale of this is huge. Yeah, that's what I loved about geology, that the scale is completely different and it makes us see the rest of the world in perspective, I guess. We are like a blink in the length of time of the earth.
Swyx [00:03:31]: But a very significant blink. So you went from large monoliths to large language models. I don't know how to make that transition there. And yeah, so like maybe could you describe your journey into Hugging Face? Obviously, I think you're like our second or third person from Hugging Face on the podcast and it's like the definitional sort of open AI company, maybe the real open AI.
Clémentine [00:03:56]: Yeah, I did. So at the end of my PhD, I realized that I did not want to stay in academia. And I actually got contacted by Meta because they wanted to offer me an internship. And I was like, wow, I can do an internship during my PhD. Where do I want to do an internship? And so I applied to Hugging Face. Thank you Meta for like opening this door for me. And I actually was hired to work on pre-trained graph transformers. And we train foundational graph transformer models. And it was a very interesting project. But it was a bit hard to accomplish with the resources we had at the time. We tried it for three months, gave it three months more. So the first three months were my internship, three months more were my first three months at Hugging Face. And then we dropped it. We still left a lot of artifacts about graph machine learning that people can use. But we stopped trying to basically compete with Google on this specific topic. And then we had a team which was doing model, like literally LLM training at the time. And we made a list of the different topics. And one topic which people were not interested in was actually evaluation. So I spent a month just reading all the papers I could about evaluation. And I discovered that it was very interesting. And so we started setting up our own internal evaluation suite, which later became LightEval. And once Tom saw that we were interested in evaluation, he sent us the leaderboard, which was a completely different initiative at the time. And then we basically became a small team doing evaluation and leaderboards at Hugging Face. I'm saying we because I'm including Nathan Habib, who is the engineer working with me on evaluation and leaderboards at Hugging Face.
Alessio [00:05:52]: Just to set the stage, maybe back in April 2023, we did our Benchmarks 101 episode. And I think everybody was trying to figure out how do you actually evaluate these models. And the models were not very good. Well, first of all, there were not that many models. And the models were not very good at a lot of things back then. Can you maybe give people a bit of a background on how many models you're testing on the leaderboard? I know it's thousands and thousands of models now. And then how you're thinking about what benchmarks matter. And we can go into some of the details. But I think just explain the scale, how many models there are, how many people contribute to this community outside of the actual Hugging Face maintaining team. Okay.
Clémentine [00:06:33]: So very beginning, it was really just an internal research project, because our reinforcement learning team wanted to compare some results that they had with published papers, and they did not manage to. And so they opened a small leaderboard where they had manually evaluated a bunch of models. And the community really took over. People were super motivated. And after one month and a half, that's when it was given over to Nathan and I, so that we could make it into an actual engineering product, which could run an actual production rather than a research project. So at the moment, we have evaluated on the version one of the OpenLLM leaderboard, 7,400 models, which have been community submitted for most of them. I think around 800 discussion threads of users interacting with us, either for support or for suggestions. We have had several million visitors since the creation of the leaderboard. And yeah, the scale of it is quite huge. We actually have, from time to time, startups sending us thank you messages saying, oh, our model ranked so high on the leaderboard that we actually got a funding round thanks to you. And so they are very happy about this, and we get thank you messages. So it's quite used, especially by the community. A lot of the community is using it to test their methods and test how well their ideas perform against the SOTA models.
Swyx [00:08:11]: It's instructive to rewind back maybe one, maybe two years ago when this kind of leaderboard practice wasn't normal. It's not normal to have an independently validated leaderboard. Everyone just kind of runs their own evals and publishes their evals against their models on their own paper, and it's not reproducible. So I think it's really about reproducible science. I think the only other, before this, it was kind of like ML perf, that's the other big leaderboard that I can think about where, obviously, maybe AlexNet, specific competitions, specific benchmarks, but not something that aggregates across all the other benchmarks. Maybe HuggingFace was involved in BigBench in the earlier days.
Clémentine [00:09:03]: Maybe some of the other people in the team.
Swyx [00:09:06]: Anyway, so this is the first time getting everything together. And so what was your thinking around inclusion, right? Because I think that's another element that we'll talk about V2 later on, but V1 was your selection of here are the top benchmarks. I don't know if there was any story to tell behind that apart from, was it obvious? Were there any controversial choices?
Clémentine [00:09:29]: So for V1, Edward Beeching and Lewis Tunstall, who were our reinforcement learning team at the time, basically wanted to look at the scores which were there in every paper. So they took all the big RL papers of the time and they looked, and you had GSM 8K, you had MMLU, you had ArcChallenge, systematically. So they took those benchmarks because, yeah, they were kind of obvious which ones were the standards of the time. And when we added evaluation, I think we actually added GSM 8K later on. We also tried to add Drop, which we dropped because there was an implementation problem. When we did our round two, we based ourselves, like not the V2, but I'm going to say V1.5. We interacted a lot with the community to see what was missing in terms of evaluations, what were the capabilities that people wanted to see. And we added those datasets at the time. We keep interacting a lot with the RLHF team, like Lewis Tunstall specifically has been very helpful in helping us choose the last set of evaluations for the V2 also.
Alessio [00:10:49]: In the V2 announcement blog posts, you mentioned some of the issues that all their benchmarks had. I feel like it's funny to me that now everybody's bringing up these issues, but before they were just using the numbers for marketing and promotion saying, look how good we do. But now it's like, no, actually the benchmark is wrong, like it should be harder. Are people just finding out recently about these problems because now the scores are getting so high that you're actually inspecting the benchmarks and maybe in the past you were scoring so badly that maybe you weren't as worried about the overall quality? Or what do you think now, like, you know, the last maybe like two, three months have been really like where the leaderboard and like has been kind of like taken off as far as popularity and then why it's now the right time to do V2. And then we'll talk about what V2 actually is.
Clémentine [00:11:36]: For the first question, when you read evaluation papers, actually a lot of the datasets from, I'm going to say it's a pre-LLM period, are datasets which have been turked basically. So datasets which were made by people which are underpaid, where English is usually not the native language. And so a lot of them have a lot of mistakes and it's kind of obvious just from reading the paper that there are going to be issues because when you generate 10,000 samples, it's quite hard to manually verify each and every one of them. The datasets we've been using, such as MMLU, ArcChallenge and stuff, were of higher quality from the start, but the attention which was given to them forced people to at one point really explore the datasets to see where the scores were coming from. And yes, at the moment, like when a benchmark reaches saturation, so when models basically get the same performance as humans on a benchmark or go above human performance, which the press really likes to say, what it actually says is usually that the models are completely contaminated on said benchmarks and they are now doing errors which humans would not be doing. For example, for MMLU, human performance is at 80-something, also because a number of the questions that humans fail are actually wrong. So the correct answer to those questions is actually a wrong answer. And so the humans who got a wrong answer on this actually had the correct answer. And so if you're getting above humans on this, you have actually learned to predict very wrong stuff, which I find absolutely fascinating. So evaluation has good enough signal-to-noise ratio, if the quality is high enough, it's going to be useful enough for a period of time, and once you reach saturation, you want to inspect it more.
Swyx [00:13:45]: There's a three-way race condition as we all figure out who's going to go first. Yeah, so I really like this concept of evaluation. So actually, yeah, I think there's typically what I always say is like sort of 25 is random chance, 50 is average human, 75 is expert human, 90 is you're cheating. And the question now is that most models are high 80s in MMLU, and so it's not challenging anymore or we've sort of saturated it. It looks like some people have put out MMLU Pro. I've seen a few variations of what comes after MMLU. Dan Hendrycks, who came out with MMLU, has promised to make his own MMLU. To me, what I worry about MMLU Pro or any other MMLU variant is that this will last one year. Yeah.
Clémentine [00:14:39]: And then what? And then we do Leaderboard v3. Oh, okay. That's it?
Swyx [00:14:45]: I mean, yes.
Clémentine [00:14:46]: Sorry to disappoint, but like, we basically expect the scale of AI progress to go so fast that anyway, we will have to renew them. Of course, some of it will be for contamination issues, people trying to game the leaderboard, cheating and stuff. But a lot of it will just be because the benchmarks will have become just too easy, like the scale of the progress we did in just one year on those benchmarks is already huge. And I think that's also why the leaderboard was so important, because everybody wanted to climb the scores. And so those evaluations really saw a jump in the performance, like we've got curves on version one, which is archived, but still accessible, of model performance through time. And you can really see the steps which you had for each evaluation.
Alessio [00:15:47]: I think the other thing to talk about here is whether or not humans are good at judging and evaluating these models. We're kind of slacking about it over today, but I would love to get your thoughts. It's like, at what point are we not the best people to test these models anymore? And how do you kind of balance the machine benchmarks, the MMLUs of the world, the LMSIS kind of human-driven rating, and then the AI judges, so to speak?
Clémentine [00:16:14]: I have many opinions about human evaluation. But I think that...
Alessio [00:16:20]: We've got time, so...
Clémentine [00:16:23]: Basically, to go back to the initial separation, just to make it clear, so automated benchmarks, like the one we're using on the OpenLLM leaderboard, are usually fair and reproducible. Every model gets evaluated in exactly the same way, and you can really reproduce the scores you get. They tend to be also limited in the scope of what they allow to evaluate, because if you're looking at a multi-choice question, well, it's not telling you how good the model is at generating poetry, for example. So people have been using human evaluations to kind of go further in terms of the capabilities we can evaluate. We've got three types of human evaluations, in my opinion. We've got vibe-check evaluations. We've got Arena-type evaluations, like the LMsys Chatbot Arena. And then you've got human experts, so paid human annotators who will evaluate stuff, which is the approach that Scale has, for example. I think that paid human experts is a really good way to evaluate models, because you can actually give a proper grid of things you want people to check. And because they are actually paid to do so, you can hope for quite a high quality. But since human experts are expensive, people have tried to use model-as-judges, which is our third approach. Model-as-judges, I won't delve too much into this at the moment, but I think they are a problem for the field, actually. I think people should stop using LLMS judges, because they have a lot of subtle biases that they introduce in evaluation. They tend to prefer outputs from the same families. They tend to prefer first answers, which is called a position bias. They tend to prefer long and verbose answers. They struggle with evaluating models in a continuous range. So if you absolutely want to use a model-as-a-judge for your specific use case, do not use GPT-4 also because it's closed source and it will not be reproducible at all. Use a small model such as Prometheus or JudgeLM, and just use it to give you rankings, such as this option is better than this other option. Don't ask it to give you scores, because at the moment those models are not able to do this in a proper fashion. And I saw on Twitter a couple of days ago, Aidan from Cohere, who was saying that their models have a very distinct style, because they don't train with other models' outputs. They actually took the time to gather super high quality data. And other models kind of sound the same because of this. And I think for evaluation, it's going to be the exact same problem. If you choose your model based on model evaluation, you're going to make it kind of the same as all the other models. To go back on human evaluation, if we go on this distinction of VibeCheck versus R&R versus human experts, I think that VibeChecks are actually quite necessary. If you are an engineer and you want to know which model is best for your specific use case, please do a VibeCheck. You can look at a general leaderboard, like the OpenLLM leaderboard. It will tell you which model is best in a range of tasks. And for your use case, you need to test it yourself. For the R&R or R&R-like systems in general, they are trying to rely on wisdom of the crowd approaches. But the wisdom of the crowd tends to work for quantifiable things, right? So it was initially done to try to see if a crowd could average the weight of a pig at a farmer's market in the 18th or 19th century. And it's been reproduced by asking people to estimate a number of a marble in a jar. And for anything which is like super quantifiable, it works very well. But when you're just telling people what is a good output, it's much harder to get something reproducible and experimental science is based on reproducibility, like rigorous protocols. And when using an R&R, you're not getting that. I think that an R&R is a very good sociological experiment, however. I think it's telling you a lot about the users. It's telling you a lot about what are the prompts, how people interact with models. And I also think that you can crowdsource evaluations if you have clear metrics. For example, for red teaming. You can definitely crowdsource red teaming because whether the model gave you private information or whether the model was toxic is something you can have a strict like yes or no answer, in a sense. But for anything else, it's very limited. There were a bunch of papers which were very interesting about this at ICLR this year. There was the psychophancy paper of Anthropic, where basically they showed that humans tend to prefer models which go their way and which agree with them because we want people to like us and apparently we want models to like us too and to agree with us too.
Swyx [00:21:49]: Arguably, that's alignment, you know, we want models to like humans. Sometimes it's good.
Clémentine [00:21:56]: Yeah. But you also want humans to actually say the truth. Disagree. Yes. Exactly. To challenge you. Definitely. If what you're thinking is not factual. There was also this cool paper by Cohere and the University of Edinburgh, which was human feedback is not gold standard, I think. And where they actually established super interesting things such as humans prefer models which are over assertive. And if you have the choice between an answer which is false, but given super assertively and an answer which is right, but not as assertive, humans naturally will say the assertive but false answer is a better one. So basically, Arenas are not giving you factuality, which should be a super important aspect of LLMs, I think.
Alessio [00:22:52]: That's the same with everyday life. You know, people just trust the person saying the thing assertively, even though it's false, and then actually try and figure out what the truth is. So yeah, I think you mentioned that, you know, it's like a more social experiment. I think it's a good point. Like the same biases that people have interacting with humans, they kind of put in the models themselves.
Clémentine [00:23:15]: Yeah, definitely. In these things. But there's also the fact that some like the judgments and the likings that we have in real life do not necessarily have the same impacts as LLMs, which are used in production, right? So you don't want the best LLM, according to everybody, to be the one which is going to be the most psychophantic and then get propaganda chatbots or something. On anything like an Arenas, there can be also the problem of the lack of diversity of the annotators, because most of the users of the chatbot arenas, for example, tend to be, from what I gathered, men from the US. I'm sorry, but this is not a diverse demographic. So those are reasons for which human evaluations, in my opinion, are quite limited.
Swyx [00:24:07]: I'll throw in one more, which is, I think the sample of the chatbot arena data is actually out there. And most of them are single turn tests as well.
Clémentine [00:24:19]: Definitely.
Swyx [00:24:20]: So multi-turn is not tested at all.
Clémentine [00:24:22]: At the same time, I won't complain too much about this because we also tend to not evaluate multi-turn for automatic benchmark. So I cannot really say anything about this.
Swyx [00:24:32]: The AI girlfriend community has got you there. They're very good at the multi-turn and you just need to go to OpenRouter to see which the top trending bots are. For those who don't know, a lot of this is covered in your blog posts, which I think you wrote after ICLR, which is, let's talk about LLM evaluation. You cover sort of a top-down, what you think about evals, and you even point to RavenWolf for the vibe check, who apparently blogs a lot on HuggingFace, because HuggingFace is now a blogging platform and does really good vibe checks, apparently.
Clémentine [00:25:08]: He does. I actually found out about the guy on Reddit because he does extremely long threads about the different models he evaluates and the kind of questions that they get right or wrong. He does his evaluations in German, if I remember properly. So it's usually very interesting to see how he does it. He's super rigorous, but he does, I don't know, 15 prompts. So a rigorous vibe check.
Swyx [00:25:35]: So I've read those things on the local LLM subreddit and it's a little bit excessive. I don't know if I need all that, but I'm glad somebody does it. To me, he's my automated vibe eval. I don't know who he is, but he shows up, so that's about it. So we wanted to cover specific choices around the new leaderboard. So congrats on launching it. You corrected a bunch of very fundamental data science things, like the variance between the benchmarks, as well as selecting for better benchmarks. I think obviously MMLU Pro is the top one, just because that's the top number that a lot of people report. The headline figures are, for example, it's 10 choices instead of four, and it's actually reviewed by experts instead of just not reviewed by experts. Any other sort of special notes that you would, basically, I want to do a quick tour around the ones that you picked, right? MMLU Pro, GPQA, I think these two are very well regarded. I have Eval as well. I noticed that Apple Intelligence is the only benchmark that Apple Intelligence used. Everything else was their own internal evals, but Apple Intelligence picked IFEval as their benchmark. Anyway, so do you want to comment quickly on some of the ones that you picked? Yeah.
Clémentine [00:26:50]: So for IFEval, I think it's a very interesting one because it's like unit tests, but for language, right? When you evaluate coding LLMs, you give them a bunch of unit tests, and you see if the functions that the LLM has written is able to make all the unit tests work. And IFEval behaves in literally the same way. They are giving prompts with very strict instruction formatting, and they are only evaluating instruction following. And I find it very interesting because it's not a metric which is ambiguous at any step. A lot of evaluations which are looking at the content are going to be using bag of words or embeddings to try to get like semantic similarity. Here you don't care. You are literally evaluating on understanding instructions. And I think it's a very smart data set. I loved it. We also added GPQA, which I've wanted to add to the leaderboard since it came out. Basically MMLU, but PhD level. Super complex questions which have been written by PhD experts and which are easy kind of to answer. If you have a PhD in the field, but not if you don't. So I think those ones are super interesting. They are only in science.
Alessio [00:28:12]: Yeah, I wanted to know if there's a black market for like the actual data sets that go in the benchmark. I know you have a gating mechanism to get the actual questions to make sure that models don't get contaminated. Do you ever get people reaching out to you? They want to buy the question answers to get better scores on the model. I wonder if marketing budgets are being spent on that.
Clémentine [00:28:34]: So for GPQA specifically, anyone can have access to the answers. You just need to create an account and say yes to the gating system and you will have access. The gating system is mostly here for bots, basically to prevent bots passing the web from getting access. However, for the Gaia benchmark, which I was part of, which is a benchmark for agents, we actually got contacted by some institutions from some specific countries who were actually like, well, can you give us the answers to the test sets? We're going to keep it for our Intel benchmarks. And we were like, no, have you heard of what a test set is? But we actually got contacted and they were like, yeah, we think it would really help our safety for our use cases. It's funny.
Alessio [00:29:25]: Yeah. Well, I asked thinking that you would say no, nobody would ever ask that, but humans are humans. Also, I know you work closely with Haley Sholkoff from Flutter AI on this, so I DMed her. Last night I asked her some questions I should ask you. So thank you, Haley, for your help. She told me to ask you about MMLU prompt format choices and whether or not there's a right choice when building prompts for the benchmarks. And this is kind of like the GPQA example, you know, maybe you two are experts, so you're kind of having these discussions. For me, it's like, I don't even know what all the options are. So I would love for you to maybe break that down too, you know, there's the benchmark, which is like the questions and the answers and like how you evaluate them. But then there's also how do you prompt the model to actually ask them? So any insights you have, I'm sure would be fun to share.
Clémentine [00:30:21]: Okay, so for MMLU specifically, it's a multi-choice evaluation. So you have a prompt and you've got many ways to prompt a model. MMLU that we chose is the one which was used in the harness. So it's question, column, the actual content of the question, return to the line, choices, column, go to the next line, A dot first choice, B dot second choice, and then we return to the line, answer, column. And we did a bunch of experiments at some point by trying different methods, just removing question, removing choices, removing answer. And we got a variation of 30 points on 100, depending on the prompt choices, and 30 points is insane in terms of the variation of evaluations. So the smallest prompt we have was just asking the question, and then we look at the log probabilities of all the choices. So we select the good choice as the one with the best log probability. The more complex one that we had was questions, the question, choices, and enumeration of the choices, but prefixed with letters between parentheses and not letter and then a dot. And this one got the best scores across most models. And in terms of contents, both source prompts have the same, because if you look at the log probabilities, if the model actually has the knowledge, the best log probability should be the best choice, and giving it explicitly the choice should not change anything in terms of contents you're looking at. But yeah, we got 30 points of difference on this. And we actually partnered with Outlines to do a blog post about it on how structured generation can improve evaluations by a lot. And in terms of MMLU, you can also evaluate it in another way, which is what Helm does. And in this case, you do not look at the log probabilities of the choices, you actually ask the model to generate a letter. And you take the generated letter, even if it's not in the option spaces, let's say. So if you say I've got choices A, B, C, D, and the model answers cat, well, cat is wrong. And so, shame for the model, it is wrong. We chose to run multi-choice evaluations in a log-likelihood way because it's way less expensive than running evaluations in a generative way for most tasks. And it's also kind of easy to parallelize, usually, because if you're only looking at one token of generation, then you can batch it very easily.
Alessio [00:33:30]: Are the multiple choice benchmarks much easier for the models? Do you have any intuition on how would you stack rank? Because you have the MMLU, then you add GPUA, then you have the math benchmark, you have BBH. Then you run multi-choice, then there's open generation without formatting, then there's formatting-driven ones like IFEval. Which ones are hardest, most impressive? Which ones are easiest? And how did you pick this exact mix?
Clémentine [00:34:01]: The two hardest evaluations on our benchmark are math because we only selected the hardest questions. We selected the level five questions. This is a choice that we made because we wanted an evaluation which was discriminative to allow us to see which models were actually good or not. And also because it's very costly to run the full data set. We realized that it would take several hours for a 7b just for this specific data set. And we were like, no, we've got to cut stuff. What do we do? And so one of the reasons behind us using so many multi-choice evaluations is the fact that we are compute constrained. We are using nodes with H100 on them. So every evaluation we'd run on one node with 80 gigs of RAM. And if you look at, for example, Vantage, which shares prices of those kinds of instances, in terms of public price, we are at about $100 an hour. So if we evaluate a 7b model at the moment, it takes approximately two hours. If we evaluate a 70b at the moment, it takes around 20 hours. So there's a limit to how much compute and how much money we can spend on this, right? And this is also a reminder, which is important for the community, because sometimes we get some messages like, I submitted a 70b model yesterday, why was it not evaluated? And I'm like, first of all, do you think compute grows on trees? If you have an NVIDIA GPU tree, give it to me, right? I want more GPUs. And also, it takes a lot of time to evaluate models. And yeah, to go back to your initial question about the benchmarks, the two hardest are so math, and yes, it's a generative evaluation, and generative evaluations in general are harder than multi-choice, but they are also harder to get right because of the metrics. I can go back to this afterwards. And the second hardest evaluation we have is MUSR, multi-step soft reasoning. And it's hard because it's super long context. Basically, it's murder mysteries, and then the model needs to find who is the culprit. The murder mysteries are like rule-based generated, and few models do better than random on this one at the moment.
Alessio [00:36:28]: Yeah, great. Great to see benchmarks that models don't do well on. If you just look at the results, it's like, these models are amazing. And then you use them, and you can clearly see there's a lot of room for improvement. So that's great. How do you kind of take this, in a way, responsibility, right? For whether you want it or not, this is one of the lighthouse things that people look at when evaluating models, like your leaderboard. What are maybe some of the hard decisions that you have to make internally? Because you kind of have to balance how you can face the company, but also the scientific objectivity of these things. What are discussions that you had internally on how to pick this, and balancing the commercial side versus the more research side? And yeah, whether or not you had people reach out to you and say, hey, you got this completely wrong. This is actually what the leaderboard should look like. How do you deal with those disagreements from the community?
Clémentine [00:37:26]: We know that we have, as you mentioned, a huge responsibility towards the community, because this is a place where people can evaluate their models, and they can also compare and cut through all the marketing b******t, right? If you release tomorrow a model, and you're like, my model is the best model ever, we will actually evaluate it, and we will give you a number. We need to be very fair about our evaluations. This means that for the choice of evaluation, we discussed a lot internally with different people, so Louis, Tensel, Tom Wolfe, Nathan, and I, basically, and so we made short lists of which evaluations are relevant at the moment, both in terms of their contents, in terms of their stability, how well they are seen in the community. And then we spent, I'd say, about a month just running the evaluations on a wide variety of models to make sure that the implementations were absolutely correct and fair for all models. For example, when we were evaluating the version 1 of the leaderboards, we observed that Drop was using a dot as an end of sentence token, and so a lot of floating point answers would be cut off, and so would be incorrect. This was for the v1, and so we actually had scratched entirely this evaluation, because the implementation was incorrect. For v2, we spent much longer just looking at every nook and cranny, making sure the few short samples were fixed, making sure everything was properly formatted, that there were no backslash n running around or whatever. We also know that some models have issues with their tokenizers, so we made sure that they were still being evaluated properly on generative evaluations, because we know it's going to be used, and so numbers need to be as right as possible. And there isn't really a commercial aspect to the leaderboard, however, because basically we are just spending money on the thing, because we think it's a very useful resource for the community to have, but people are not paying for their evaluations to be there. It's a gift to the community, I guess.
Swyx [00:39:56]: I wonder about the compute, right? You have basically a standing H100 cluster, but the number of models grows every day. I think you cache them, you also remove models that are maybe contaminated. I think that this happens a lot, that some new model will suddenly show up at the top of the leaderboard, and then people will discuss, and they're like, oh yeah, it's contaminated, and you have to withdraw them. I just wonder about the economics of this thing. How much are you spending? You just have one standing cluster, and you just have a queue. Is that as simple as that?
Clémentine [00:40:35]: It's actually more complex. HuggingFace has one research cluster, and so the research cluster is used for every research experiment we have. If the FindWeb team is creating a new super cool dataset for you to train your model on, it's going to be on the cluster. If the IDFX team is creating a new multi-model model, it's going to train on the cluster. The OpenLLM leaderboard team is running on the spare cycles of that. We actually changed the way that our jobs are queued and launched. Basically, the leaderboard jobs are launched with the lowest priority of the cluster. Anything which is launched will kill our jobs if the cluster is too full. So that's why we can give it to the community, in a sense, because it's not costing us that much. It would be lost compute anyway. However, it means that sometimes the queue holds because the cluster is full, and users are not always super happy about it. But they get cool machine learning artifacts, so I think they should be happy.
Swyx [00:41:41]: Is there a way for the community to donate compute to you? Is there an interface that you can easily transfer your jobs to a different cluster?
Clémentine [00:41:51]: It's actually been discussed a lot, and we are thinking about adding the option to run evaluation on inference endpoint, where people would be able to pay for the compute of their evaluation. The thing is, at the moment, we really wanted to use a EleutherAI harness because it's a big stable library that everybody uses, and we think that Elusive is doing a great job at evaluations in general. But we have the functionality to run evaluations on inference endpoint in our own evaluation library, which is called Lightval. So we will have to port this functionality to the harness before being able to give it to the users. It's not been high on our priority list because then we will have to set up possibly another space where evaluation will run, or maybe people will have to duplicate some stuff. It's more engineering, and we've been a bit swamped with things to do.
Swyx [00:42:47]: I can imagine. Yeah, so hopefully when that opens up for inference endpoints, the only thing I'll caution is that all the inference providers write their own CUDA kernels and implementations of stuff. So sometimes you won't get one-for-one the same model, even though it's the same weights, but it's not exactly the same performance of the model because they quantize or do whatever they want to do with the shortcuts for attention.
Clémentine [00:43:17]: So regarding quantization, we usually indicate precisely what the precision of the model is. So you can find some models in several precisions. I guess this should be fixed by SART, but yeah, if evaluations run on different hardware with different batch sizes, results are going to be slightly different.
Swyx [00:43:38]: We're going to ask maybe three dimensions of benchmarks, and then we'll ask about missing benchmarks that you really want from the community. So the first one is something that the community is discussing a lot, which is long context. You already talked about Muser, but the other one that's popular is the very famous needle in a haystack. There are a lot of variations of needle in a haystack. We talked about this in a previous podcast with advanced needle in a needle stack and variable tracking and all that. Do you think there should be a long context version of the leaderboard, or how are you going to cut it such that you accommodate those things?
Clémentine [00:44:15]: For the leaderboard specifically, that's why we added Muser, because it's long context reasoning. In terms of high quality long context reasoning benchmarks, I can think of two which I really like. One is called a benchmark for learning to translate a new language from one grammar book, and it's actually a very fun data set where they basically provide the LLM with a grammar book written by a linguist on a small language, which is super low resource called Kalamang. Since it's so low resource, you're sure that there is no data about it anywhere on the web. Then they ask questions about the grammar, what would be the correct form, etc. This is reasoning, this is language skills, this is super long context because it's a book. I think this data set is very interesting in terms of long context. There was also LNAI, which made a benchmark which they called a novel challenge for long context model, where basically they took full-on novels published last year. They asked people who had read it to do summaries and to do adversarial descriptions of events happening in the book, which require you to have understood the full book to answer. They prompted models with that, so also a very long context evaluation because you've got a full book and then you've got those questions that you need to answer correctly and also not contaminated because hopefully the books are not in training data yet. Yeah, it's new novels. Yeah, definitely. So I think those kind of data sets are more interesting. Yeah, go ahead.
Swyx [00:46:03]: You just gave me an idea that Goodreads should be a data set because these are all novels that are commentaries about the contents of the novel.
Clémentine [00:46:12]: Definitely. There is definitely something to do about this.
Swyx [00:46:18]: Okay, that's long context. Sorry, go ahead, Alessio.
Alessio [00:46:21]: You mentioned the GAIA benchmark before that you worked on. What about agents, all of that part? Do you think we have good agent benchmarks? Do you think agent benchmarks are worth it? Yeah, curious for your thoughts.
Clémentine [00:46:34]: So for agent benchmarks, I haven't followed the literature so closely for this year, but when we did the GAIA benchmark, the main problem that we observed was that almost all agentic benchmarks would take LLMs, put them in a black box environment, which was absolutely not the real world, and then ask them to do things using very specific APIs. And that's kind of what started the GAIA project, actually, because we had this mental model of what agents could do, especially like AI assistants. We had this list of tasks of we expect them to be able to browse the web, we expect them to be able to extract information from structured places, from having access to modality, tools, etc. And from this, we built the GAIA benchmark. So really not from a capability standpoint, but more through, I'm going to call it proxy tasks, right? We expect agents to be able to do stuff and stuff. So reasoning on so many items, using so many tools. And that's how we built it. Instead of creating those boxed environments, which do not generalize well to the real world, GAIA basically tests your model on the real world. So I hope we get more datasets like GAIA. We basically provided the full recipe, and I really think that anyone could contribute or create similar datasets. So that would be one of the directions I would be excited about to see GAIA 2, GAIA 3, people thinking about creating tools also. Depending on which tools are created, some tasks are going to become way easier. So how do you add complexity to that, etc?
Swyx [00:48:28]: I interviewed Thomas Scialom at the ICLR poster session on GAIA, and for people who want to know more about GAIA, they can refer to our ICLR episode. The other big agent benchmark of this year has been SweetBench, much more coding oriented. I'm just curious if you have any thoughts or if you've looked at SweetBench at all.
Clémentine [00:48:49]: I remember going to the poster actually, but no, out of the blue, I wouldn't be able to give you feedback on it right now.
Swyx [00:48:55]: Just poking. Okay, then we have a question about ARC.
Alessio [00:49:00]: Yeah, just curious to get your thoughts. You know, obviously the ARC challenge got a new million dollar boost to get it solved a couple of weeks ago, so a lot of eyes on it. I think maybe some people are saying...
Clémentine [00:49:14]: Because we've got two ARC challenges. Like we've got challenge, which is a subset of the LNAI ARC dataset, and then you've got the Cholet ARC AGI challenge. Which one are you talking about?
Alessio [00:49:25]: Yes, the AGI challenge. Well, first of all, I'm curious if you think that actually is AGI, if you solve it, and just overall thoughts on the more challenge-driven things rather than evaluation, benchmarks driven.
Clémentine [00:49:41]: I don't think if you solve it, it's AGI. I think that focusing at the moment on trying to reach AGI is also a very bad objective, to be fair. But I'm very excited about this specific dataset. I'm looking forward to see what happens because I took a stab at some questions and basically they are great. They are pure logic. One of the things that we are missing at the moment in terms of LLM evaluations is complex logic, I think. Models are very bad at this. If they manage to learn the patterns and generalize on something which is logic-based, then we will have reached a step in reasoning, which will be very interesting.
Alessio [00:50:26]: Just overall, more meta question. How do you figure out whether or not a benchmark is actually useful? Everybody wants to build benchmarks, kind of like test sets and things like that. Do you have any quick ways, kind of like you have a vibe check for model? Do you have a vibe check for benchmarks?
Clémentine [00:50:43]: Actually, I do, but... Okay, so first thing is, and like the low investment version is, you first look at the paper and you want to see who made the dataset. And by this, I mean, was it model generated? Was it human generated? Were the annotators paid properly? Are they actually native English speakers if your dataset is in English? Etc. You want to know what is the quality of the dataset from the metadata, basically. And then you want to know what were the assumptions behind the dataset. What do they think their dataset is a proxy for? And does that sound logical? And then you want to look at the questions. You want to actually go through the dataset. You want to look at the prompts. Are you able to solve them? Do you see obvious mistakes? Are the prompts cohesive in terms of format? Like, is the formatting consistent? And you want to ideally take a small look at the codes. And if you have more time to invest on this, you can basically just use it for yourself on a bunch of models that you know are good. You want to use it on a small good model, like maybe, I think, 5.3. Like, it's very debated, but it's not that bad for its size. You've got a bunch of around 2 billion parameter models, which are good enough for this. So it wouldn't be too expensive. And then you want to test it on a very big model. That everybody knows is good. Like, when to or command R plus, for example. And if it's a generative model, you look at the generations. Are they, like, well made? Are they truncated? Do they look realistic? And then it gives you more of an idea of the quality. Because the quality of a lot of benchmarks will rely on the quality of their metrics. And if you are using, for example, an exact match metric, you want to make sure that you can actually extract something from the answer. GSM 8K is very good at this, because the output format is very constrained. But some evaluations are very bad at this. Drop, for example, is using a combination of bag of words to estimate whether the correct answer was given. And this is not a good metric, for example.
Swyx [00:52:58]: There's an old school NLP thing to use bag of words. Yes. It's kind of like a blue score. Okay, just in case you have one. Is there something that you wish somebody built a benchmark for that you really wanted to include, but you couldn't find it?
Clémentine [00:53:16]: Yes. I think that there are a bunch of things that we would need. But one thing is model calibration. Nobody's evaluating model calibration at the moment. And I think it's a problem. Model calibration is...
Swyx [00:53:29]: What is model calibration?
Clémentine [00:53:32]: You have a very confused face. This is very fun. Basically, a model is said to be well calibrated if the log probability score of an answer correlates well with how correct the answer is. So you want a model which... You can basically see it as the self-confidence of the model, right? So you want a model which tells you, yes, this is true, to have high probabilities of this, if it is actually true. And this thing specifically is called calibration. And it's not that hard to measure. You could use any multi-choice evaluation set to test this. I think there are more interesting datasets to build to test that. But if we have well calibrated models, it will open the door to basically being able to have models with confidence intervals about their answers. And you would be able to say, the model is highly confident about what it's saying. Or the model is in doubt, and you could give small confidence scores. I think this would be very interesting.
Swyx [00:54:42]: Yeah, there's some papers at ICLR on uncertainty as well. The quick response I'll give to that is, I think it's well known that base models are better calibrated than instruct-tuned models, right? So just the instruct-tuning just screws it up, makes it overconfident, makes it too much like a human.
Clémentine [00:55:00]: Therefore, it's... Yeah, it's tricky.
Swyx [00:55:02]: But yeah, I agree with this thing. We should have a benchmark for it and it'll get better.
Clémentine [00:55:07]: Yeah, I hope so. And I guess a bunch of other things would be interesting to evaluate. I think that robustness to prompting, nobody does it because it's too expensive. But if I prompt a model with 10 variations of the exact same prompt in terms of content, I don't want to get 10 different answers, right? And it's kind of linked to calibration. It's something that should be taken for granted in LLMs, but it's actually not working that well. And if I had to take a third choice, because I'm very greedy, you asked me for one, but you're getting three. I would love to see more things about psychofancy and basically all the ways into which models can be problematic in their interactions and put people in basically thought bubbles. You don't want people to be on social network too when they talk to a chat model, right? You want the chat model to be assertively saying what is factually true or not. Some things are factually true, right? The earth is round, gravity exists. A lot of things should not be debated and models should be assertively telling users that they are wrong if they are saying that those do not exist. Awesome.
Alessio [00:56:27]: This was a great kind of run through the leaderboard and a lot of the questions we already took a lot of your time. Before we wrap, maybe just one last thing. Any predictions for like leaderboard v3? Like if you go one year from now, do you think most models will have kind of top this new v2 too? Or how long do you think it's going to last before you need a new one?
Clémentine [00:56:48]: I'm actually working on the next version.
Clémentine [00:56:53]: I'm actually working on the next version, which now I'm not going to talk too much about it. But I think that we still have a lot of range for reasoning and mass evaluations at the moment. I think that we still have a lot of the evaluation space to explore. Long context, we're just getting started. I assume that some things like instruction, for like EF eval, for example, I assume that models are going to become very good at it very soon. And sadly, probably GPQA, because I think that it's going to be contaminated at some point. But yeah, basically the next version of the leaderboard would be depending on how fast models changed. It would be a similar version with reasoning, mass, maybe code if we can add it, because now all models should be able to code a bit. And I would really like to add a psychofancy evaluation for the next version. Yeah, well, but it's in the far future. So that's the end of my predictions.
Alessio [00:57:58]: Awesome. Yeah, thanks so much for coming on. We're going to link all of your previous work in the show notes so that people can read through it. And people can follow you on Twitter or X to stay up to date. Sorry, Yvonne, don't unfollow us.
Swyx [00:58:10]: Follow her on Hugging Face. Hugging Face is a social network.
Clémentine [00:58:13]: Yeah, that's true.
Alessio [00:58:16]: Yeah, that's it really. Thank you so much.
Livestreams for the AI Engineer World’s Fair (Multimodality ft. the new GPT-4o demo, GPUs and Inference (ft. Cognition/Devin), CodeGen, Open Models tracks) are now live! Subscribe to @aidotEngineer to get notifications of the other workshops and tracks!
It’s easy to get de-sensitized to new models topping leaderboards every other week — however, the top of the LMsys leaderboard has typically been the exclusive domain of very large, very very well funded model labs like OpenAI, Anthropic, Google, and Meta. OpenAI had about 600 people at the time of GPT-4, and Google Gemini had 950 co-authors. This is why Reka Core made waves in May - not only debuting at #7 on the leaderboard, but doing so with all-new GPU infrastructure and 20 employees with <5 people on pre-training and a relatively puny $60m in funding.
Shortly after the release of GPT3, Sam Altman speculated on the qualities of “10,000x researchers”:
* “They spend a lot of time reflecting on some version of the Hamming question—"what are the most important problems in your field, and why aren’t you working on them?” In general, no one reflects on this question enough, but the best people do it the most, and have the best ‘problem taste’, which is some combination of learning to think independently, reason about the future, and identify attack vectors.” — sama
* Taste is something both John Schulman and Yi Tay emphasize greatly
* “They have a laser focus on the next step in front of them combined with long-term vision.” — sama
* “They are extremely persistent and willing to work hard… They have a bias towards action and trying things, and they’re clear-eyed and honest about what is working and what isn’t” — sama
“There's a certain level of sacrifice to be an AI researcher, especially if you're training at LLMs, because you cannot really be detached… your jobs could die on a Saturday at 4am, and there are people who will just leave it dead until Monday morning, or there will be people who will crawl out of bed at 4am to restart the job, or check the TensorBoard” – Yi Tay (at 28 mins)
“I think the productivity hack that I have is, I didn't have a boundary between my life and my work for a long time. So I think I just cared a lot about working most of the time. Actually, during my PhD, Google and everything [else], I'll be just working all the time. It's not like the most healthy thing, like ever, but I think that that was actually like one of the biggest, like, productivity, like and I spent, like, I like to spend a lot of time, like, writing code and I just enjoy running experiments, writing code” — Yi Tay (at 90 mins)
* See @YiTayML example for honest alpha on what is/is not working
and so on.
More recently, Yi’s frequent co-author, Jason Wei, wrote about the existence of Yolo researchers he witnessed at OpenAI:
Given the very aggressive timeline — Yi left Google in April 2023, was GPU constrained until December 2023, and then Reka Flash (21B) was released in Feb 2024, and Reka Core (??B) was released in April 2024 — Reka’s 3-5 person pretraining team had no other choice but to do Yolo runs. Per Yi:
“Scaling models systematically generally requires one to go from small to large in a principled way, i.e., run experiments in multiple phrases (1B->8B->64B->300B etc) and pick the winners and continuously scale them up. In a startup, we had way less compute to perform these massive sweeps to check hparams. In the end, we had to work with many Yolo runs (that fortunately turned out well).
In the end it took us only a very small number of smaller scale & shorter ablation runs to get to the strong 21B Reka Flash and 7B edge model (and also our upcoming largest core model). Finding a solid recipe with a very limited number of runs is challenging and requires changing many variables at once given the ridiculously enormous search space. In order to do this, one has to abandon the systematicity of Bigtech and rely a lot on “Yolo”, gut feeling and instinct.”
We were excited to be the first podcast to interview Yi, and recommend reading our extensive show notes to follow the same papers we reference throughout the conversation.
Special thanks to Terence Lee of TechInAsia for the final interview clip, who are launching their own AI newsletter called The Prompt!
Full Video Podcast
Show Notes
* Yi on LinkedIn, Twitter, Personal
* Full prep doc
* Reka funding/valuation
* Building frontier AI teams as GPU Poors
* Yi’s Research
* 2020
* Efficient Transformers: A Survey went viral!
* Long Range Arena: A Benchmark for Efficient Transformers in 2020
* 2021: Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study
* 2022:
* UL2: Unifying Language Learning Paradigms
* PaLM -> PaLM-2
* Emergent Abilities of Large Language Models vs the Mirage paper
* Recitation Augmented generation
* DSI++: Updating Transformer Memory with New Documents
* The Efficiency Misnomer: “a model with low FLOPs may not actually be fast, given that FLOPs does not take into account information such as degree of parallelism (e.g., depth, recurrence) or hardware-related details like the cost of a memory access”
* 2023: Flan-{PaLM/UL2/T5}
1.8k tasks for instruction tuning
* Encoder-decoder vs Decoder only
* Latent Space Discord discussion on enc-dec vs dec-only
* Related convo with Yi Tay vs Yann LeCun
* @teortaxes:
* If 2024 papers are to be trusted: You don't need (most) attention you don't need (most) kv cache You don't need (most) FFN layers You don't need a reward model You don't need… all the stuff that still makes frontier models work, ironically
* “there have been no real advance since 2019's T5 models”
* The future of Open source models - relevant to a16z vs Founders Fund debate. Open source cannot compete!
Timestamps
* [00:00:00] Intro
* [00:01:57] Yi Tay Intro
* [00:03:02] Path into LLMs
* [00:09:41] Google Brain: PaLM, UL2, DSI, Emergent Abilities
* [00:11:54] PaLM 2
* [00:15:27] Emergent Abilities
* [00:18:26] Quoc Le
* [00:24:16] Marketing Research: How to Start from Zero with No Reach
* [00:27:34] What's needed to be a successful AI Researcher?
* [00:30:31] Reka Origin
* [00:33:24] Starting Reka Infra
* [00:35:04] Why not to use TPUs outside Google
* [00:36:29] Chaotic vs Stable Infra
* [00:38:04] Risk Sharing of Bad Nodes
* [00:41:05] Checkpointing and Orchestration
* [00:43:39] Reka Flash/Core/Edge
* [00:46:59] Recruiting the team
* [00:47:22] Noam Architecture - Swiglu, GQA, RMSnorm, ROPE
* [00:52:26] Encoder-decoder vs Decoder-only
* [00:55:52] LLM Trends - Llama 3 and Phi 3 Glowup
* [00:57:46] LLM Trends - Benchmarks and Evals
* [01:03:25] LLM Trends - Early vs Late Fusion Multimodality
* [01:07:22] LLM Trends - Scaling Laws
* [01:09:41] LLM Trends - Long Context vs RAG
* [01:12:31] Long Context vs Finetuning
* [01:14:14] If emergence is real, when does Efficiency work?
* [01:17:41] MoEs and Upcycling
* [01:20:47] The Efficiency Misnomer - Efficiency != Speed
* [01:25:05] Open Source vs Closed Models
* [01:28:08] Personal Productivity
* [01:33:19] Singapore vs US Academic Scene
* [01:37:42] Building Silicon Valley outside Silicon Valley
* [01:40:29] TechInAsia Meetup
Transcript
[00:00:00] swyx: Thanks for watching. Bye bye.
[00:00:05] AI Charlie: Welcome back, friends. It's only been a week since the World's Fair, and it was incredible gathering the community to see the latest and greatest in AI engineering. You can catch up now on the four live stream track days on the AI Engineer YouTube, and our team is busy editing the remaining workshops and five other tracks, including the surprisingly popular AI Leadership track.
[00:00:28] Thank you all for your support, and stay tuned for news about the next event. The 2025 AI Engineer Summit. Last week, we did a very special deep dive with Josh and John of InView and Databricks Mosaic on training LLMs and setting up massive GPU clusters. And today, we're pleased to follow that up with a very special conversation with Yi Tai, formerly tech lead of Palm 2 at Google Brain, and now chief scientist of Reka.
[00:00:56] ai. Raker's largest model, Raker Core, was at launch. The fifth best model in the world. And the only GPT 4 class model not trained by a big lab like OpenAI, Google, Anthropic or Meta. In fact, while Google Gemini has 950 co authors, Raker only has 20 employees. With up to five people actually working on pre training.
[00:01:21] Swyx was excited to return to Singapore to delve into Yi Reka and building a new AI model lab outside of Silicon Valley. Stay tuned to the very end for a special bonus clip from Yi's recent appearance at the Tekinesia meetup for his spiciest take on why senior management is overrated and why this is the time to build up senior 10, 000x individual contributors like himself.
[00:01:46] Watch out and take care.
[00:01:48] swyx: Welcome to lay space. This is a long time coming, but I'm so excited to have you here.
[00:01:52] Yi Tay: Yeah, thanks for, thanks for inviting and excited to be here. chat about a lot of stuff.
[00:01:57] Yi Tay Intro
[00:01:57] swyx: Yeah. So you are interesting to research and introduce. You are now chief scientist of Rega, which is a super interesting model lab, but before that you were at Google Brain, you were architecture co-lead on POM two, you were inventor of UL two.
[00:02:10] You're a core contributor on Flan, you're a member of the Bard core team, and you also did some work on generative retrieval. That's a very, very illustrious three year career at Google Brain.
[00:02:19] Yi Tay: Yeah, thanks, thanks, thanks, yeah.
[00:02:20] swyx: And then since then, Reka, you joined in March 2023, announced a 58 million series in June 2023.
[00:02:26] I don't know if you know, the post money valuation, or the pre money valuation is public. So it's, crunch basis is, is, Oh, okay, okay. I
[00:02:33] Yi Tay: did not know that yet. 50
[00:02:34] swyx: something million. So you don't even have to leak it. It's on the internet. Okay. Rekha's stated goals were to work on universal intelligence, including general purpose multimodal and multilingual agents, self improving AI, and model efficiency.
[00:02:45] In February You released Rekha Flash. In April, you released Rekha Core and Edge. And then, most recently, you released VibeEval. Is that a good summary of the last six years? No, it's not. Four years? Four years, yeah. Oh my god. We're talking about AI I was wondering, since when did I,
[00:03:00] Yi Tay: like, step into a time machine or something?
[00:03:02] Path into LLMs
[00:03:02] swyx: Yeah, okay, so can we just talk about your transition into, you know, you did your PhD, and we can talk about your PhD, transition into brain and research and all that. You know, I saw you do some work on recommender systems, I saw you do some work on quaternions. What the f**k was that?
[00:03:17] Yi Tay: Okay, let, let, let's, let's forget about
[00:03:18] swyx: that.
[00:03:18] Just describe your path into modern L lms, right? Because you were, you were, you didn't start there.
[00:03:24] Yi Tay: Yeah. Okay. Sure. Sure. I, I, I think the world also didn't start start there, right? I mean, I think in so I joined Google in 2019, end of 2019. And the world looked like really different at the time, right?
[00:03:34] I think that was around the time the first GBT was released by. GPT 1 or something was released by OpenAI. So, research, like ML research and NLP research looked very different at that time. So I was mostly, I identified as like a language researcher. I don't like to use the word NLP, Jason will kill me if I use the word NLP.
[00:03:51] But like, I was like, okay, a language researcher. I, , but I was more like an architecture kind of researcher. And when I joined Google, I was also I continued on this as a model architecture researcher. I worked a lot on efficient transformers. That was your first viral paper. Yeah, yeah, and like, you know, I worked in the long range arena.
[00:04:09] I spent quite a lot of time looking at what we could do without attention. Like, there was a synthesizer paper back in 2020. I think that was like my early days in Google. There wasn't like a At that point of time transformer research was mainly like WMT, like machine translation and like perplexity and stuff like that.
[00:04:25] It's not really about You know, there wasn't like, I think it was in field short, field short learning and field short in context learning came only about like, you know, when GPT 3 came out and beyond, right? And then, so I think that at that time, the meta, I would say, the meta looked very different. And at that time, a lot of the work was focused on Like fine tuning things like T5 or BERT or something like that, right?
[00:04:45] So I think a lot of the research, not only myself, but like around me or like even the broader community were working on those kind of things. And so I think that was, which I feel that in hindsight today is actually pretty useful to like kind of think about because a lot of people came into like, AI into right after ChatGPT came out, right, so they saw AI as kind of, I think there's a lot of benefits of you know, understanding how, you know, transformers and like, I've broken this thing apart so many times, trying to, it's like, these things actually, you know, help to improve intuition and it's not totally disconnect I think a lot of things are still relevant today and, and it's just the scale has gotten much larger and also the paradigms shift a little bit from Single task, fine tuning to like generally do everything kind of universal foundation models.
[00:05:29] Foundation models, right. I think it's just a slight change in paradigm, but fundamentally, I don't think like the underlying principles of research hasn't really changed that much except for like compute. Yeah. So basically algorithms
[00:05:42] swyx: stay put and then compute and data scale.
[00:05:45] Yi Tay: So I have some thoughts about this.
[00:05:47] So I think back then a lot of the academic research, I think people have talked about this, like Sasha Rush has talked about this, or other people have talked about this, it's like, the conferences were always organized by like, Applications, right? They were always organized by like, Oh, like question answering.
[00:06:02] It was always by this, right? I think there was, there's like a bit of a transpose going on. Things become universal and then becoming like, okay, there's a data work stream, there's a model architecture work stream, and then people work on improving like a universal model and general purpose algorithms to improve this model rather than finding domain specific tricks.
[00:06:20] I think for, even in 2019, I think I've already been Like focusing on works that are like you know, you could improve on general architecture at that time. It was like, like maybe LSTMs in 2017 or something, and then you try on like 10 different tasks and the kind of thing, right? But like a lot of the research community have been focused more on like, how do I get that extra 2 percent on question answering or like, and then sentiment analysis.
[00:06:44] I think. There was this phrase of like, in 2017, 2018, where this style of work was still very fashionable in academia and conferences, right? And then, I think the big thing about the chat GPT moment of like, 2022, the thing that changed drastically is like, it completely like, it was like this sharp, Make all this work like kind of like
[00:07:02] swyx: obsolete.
[00:07:03] So November 2022, you're saying? Exactly, Charged GPT launch? Because I feel like if you're in the research community, this was coming.
[00:07:08] Yi Tay: Yeah, yeah. That's what I'm saying. I'm saying that like the big labs and stuff, like people have already been moving towards general, like even T5 was already like general purpose.
[00:07:15] Yeah. And that's the thing, right? But like, there was, it's like, there's a bit of a time okay, like places like Google and Meta, OpenAI, we will be working on things like three years ahead of everybody else. And academia will be like, Still working on like this path specific things, Got it, got it. And then like, I think the faulty function was the, the ChatGPT moment actually really like, It was coming, it was coming, it was just like the final, the last straw, and then it's finally like, Yeah,
[00:07:39] swyx: now it's serious.
[00:07:40] Yi Tay: Yeah, now it's really, the thing completely changed. I don't know how it turned from my, from my background to like, talking about the meta.
[00:07:47] swyx: I think that you navigate the meta very well, and part of my goal here is to also isolate how you think about the meta for other people to reflect on, because I think obviously you do it very well.
[00:07:57] Oh, thanks. I'm looking at your papers published somewhere around 2021. You had a hard cut to 22 Y two and Palm, and you did Y two Palm Emerge Abilities, DSI, REIT recitation, augmented Generation, all in the same year-ish. Mm-Hmm. So like there was, did you change teams? Did you, did you like have a research focus?
[00:08:17] Like when did you become,
[00:08:19] Yi Tay: oh, you're still saying that like language research became the
[00:08:21] swyx: model guy.
[00:08:21] Yi Tay: My research became emergent. It was like, it's very obvious. No, I don't think I'm like a person that like, I'm not like super, super great at like forcing a trend, like two years ahead. And then especially, especially like, Plan for that, right?
[00:08:34] Yeah. I think I smoothly and as like, kind of like as like I, I never actually really thought about this, this way. I just did like at every step, I just like optimized for like what I found to be most impactful and most promising. And then that gradually, and also it is, it is also a lot of influence by talking to people.
[00:08:52] Right? And then at the time I started working more with. I had some close collaborations with Jason and other people. I mean, Google is, you can work with anybody you want, basically. So you're kind of like, also like, partly it's like the environment shift. And I think the environment shifts like very quickly, but like, I was always like pulling for like the environment.
[00:09:10] I was not like, I think it's always good to like have an open mind and move along with the field rather than, okay, this is my research area. I'm going to get stuck in it two years. I think I just move along to find like things that interest me. And naturally I think like that turned out to be like, The things that were most impactful at that time.
[00:09:27] In retrospect, I kind of did well, but like, I never actually really saw it as intentional. Sure. I didn't do anything really intentional, except that's doing what I find interesting, actually.
[00:09:37] swyx: Cool. Well, we'll just talk about the main work at Google Brain, and then we'll move to Rekha.
[00:09:41] Google Brain: PaLM, UL2, DSI, Emergent Abilities
[00:09:41] swyx: So out of UL2, Palm, Emergent Abilities, which of these came first?
[00:09:46] Yi Tay: Yeah, I, wait, I did, I need, I can't really actually re remember. Okay. What will make you talk about year two then? Year two and DSI, the differential search index? I I was working on it like the December of 2021. So like at Google they were like projects that are like big efforts that are like a researcher will be like part of the effort and then this will be kind of top downish to some extent.
[00:10:04] Right. And then they are, they were also like. Bottom up research that one could do I can't speak for the Google now for sure, but like, at least at that time, right? So UL2 and DSI, Differentiable Search Index, were like, works that I kind of tinkered with in the December break where nobody was around.
[00:10:19] Palm also has this kind of differentiation because there's Palm 1 and there's Palm 2. Right. So Palm 2, I was actually like the co lead of one of the work streams, but like Palm 1, I was more of a contributor and Palm 2, I was like, so, so they were like, now I have to think back of like, okay, what's the timeline, which came first, right?
[00:10:35] In general, they were like three categories of works. One is like broader efforts that are efforts. And then there are some that like UL2 and DSI were like my own projects, like projects I use to compute that. That I had, and then I just played with it. You accidentally left the auto run in for a month.
[00:10:50] Yeah, yeah, yeah, that was in the paper. It was fun, I think. It was really fun. And then, there was also like a third category where those were like, the efforts that my good friends were driving and I contributed. So Flan was just one of them. I know like, maybe on, I would like to just maybe say this publicly, like a lot of people like, I talk a lot about Flan.
[00:11:08] You're Flan's show number one. But like, yeah, but like, the first author is actually Hsiung Wan, who is great, and then like, another guy, Le, I was her cook. core contributor but I mean just because I'm a little more visible so I kind of Accidentally took a little bit more credit for that, but I was a co contributor, but I was not like The lead authors are obvious.
[00:11:25] Yeah, so the third category was projects that my friends Emergence was also like Emergence Abilities No, actually, that paper was supposed to be only me and Jason on the paper. And I actually became friends with Jason From the paper and then that led to like this streak of like, I dunno, 10 papers or something together with Jason and now we are like super good friends, the Ultimate Romance.
[00:11:44] But that was like the immersion paper. But I, emergent paper was also like a belonged to be like a, a bottom up kind of like a thing. And fun times. Yeah, it was fun. ,
[00:11:54] PaLM 2
[00:11:54] swyx: maybe I'll pick on Palm two. Because I feel like, I'll pick on Palm 2 and Emergence, because I really want to make sure I tell those stories.
[00:12:01] Those are important stories. Palm 2, I think it's a career story. It effectively became a co lead on the second version of a very high profile, company wide effort. How did that happen? I think people would like to know, to know how to, you know, what's like the sort of career strategy there.
[00:12:16] Yi Tay: To be clear I was one of the co leads, but there were a lot of co leads, so, so I don't want to take too much credit for that.
[00:12:21] But my involvement with Palm 2 came from the after UL2 was working well, and then it was getting some visibility within Google. Was UL2 the largest model that Google had released
[00:12:32] swyx: at the time? Yeah, I think so. That was the largest. And you just, it was a personal project? It was a personal project.
[00:12:37] Yeah. Yeah. Isn't that unusual? How can it be like one person's decision to like suddenly release something that, you know, effectively changed the trajectory of, I think how, how, how, how people brain, how,
[00:12:47] Yi Tay: how we work was that, I mean, 20 B is not that much larger, but from 11 B to 11 B T five, actually at that time there was starting BT five, right?
[00:12:55] So I think UL two is code decoder 20 B model. I think when we got it approved, it was like. It was released as like, kind of like, like the big brother of T5, you know? Kind of like, okay, we updated T5 with like a new objective and train this new model into DBM we want to, and it uses the same pre training data set and everything, right?
[00:13:13] So like from PRC4. Yeah, from, yeah, that was the easiest because there was precedence, right? It was like, okay.
[00:13:18] swyx: But yeah, there was some architecture, like the mixture of denoisers. Yeah,
[00:13:21] Yi Tay: yeah, yeah. So, so back to Palm two, I think my involvement with Palm Two came from the work to, to, to, to add UL two, to Palm two.
[00:13:28] And then I, I, I mean, it was from the top down point of view. I, I mean, the leads were decided in a top down manner. It's not like, like there was not much like fighting or like, or any major things, right? It was like. It was a mixture of bottom up, top down ish, half half situation, and then from the top it was like, Okay, these are the people who are the most visible in contributing to this workstream, and then, okay, how about E and this other guy will be in charge of this modeling workstream, and something like that, right?
[00:13:58] So I think it just happened that way organically, and yeah, I think that was how I kind of was co leading the modeling workstream.
[00:14:07] swyx: I think in retrospect, you understand now that this is a very valuable experience. And I think now, today, it will be much more competitive to get the job that you got, whereas you didn't, you know, two years ago, you didn't have to try that hard to get it.
[00:14:20] Or like, you kind of lucked into it with you all too, and then like, it just compounded from that initial good decision.
[00:14:25] Yi Tay: I think it's very hard to counterfactually analyze these type of things. I think it's definitely true that there are more people working on generative AI now, and, you know, if you are in a big company, it's way harder to navigate.
[00:14:35] Like these type of things, right? I wouldn't say that there were like nobody or so wanting to work on this at that time. In fact, there were actually But you were the obvious choice. There were less people. There were definitely less people, but I think I would say that maybe it's slightly harder now, but like, it's also not like it was easy at the time.
[00:14:50] Yeah.
[00:14:51] swyx: Yeah. I imagine it's sensitive. But also in my mind this is now the most And this is the most valuable on the job training in the world. And so people want to know how to get it. This is what I'm trying to figure out.
[00:15:03] Yi Tay: Like, actually, individually we also cannot pick somebody else's experience and then try to replicate it on, because everybody's circumstances, their initialization point, their, That thing is kind of also like in different This is not only true for LLMs in general, right?
[00:15:16] Because a lot of times like, oh, okay, you did this in this position, and then because of this It's very hard to trace all this down to to find the causal path for this thing. So I think everything in life, there's some luck involved, I guess. Yeah,
[00:15:26] swyx: there is.
[00:15:27] Emergent Abilities
[00:15:27] swyx: Emergent Abilities. Very influential paper.
[00:15:30] Subsequently contested by the Mirage paper. Oh, yeah, yeah. So before we get to the Mirage, was there a story behind Emergent Abilities? You know, I'm sure it's Jason's Thesis or like what? Just tell, just tell more about like the behind the scenes, like was was, was there a discussion that led to
[00:15:43] Yi Tay: it that, you know, this one was like this, the idea, the inception of it was like mostly Jason.
[00:15:49] Okay. Right. I think I, I helped out to like. You know, shape up a little bit of the paper get some stakeholders involved and stuff. I was discussing quite a bit with Jason, but this, the idea itself was Jason itself. So, actually, when the Mirage thing and everything came out I didn't okay, I was just hot takes for the sake of hot takes.
[00:16:06] I didn't feel, I believe in emergence I have to just go on the record and just say I mean, I believe in emergence. And then I was not feeling very strongly because I think that, I can't speak for Jason, but I would just imagine that he would be Maybe personally offended because because I know Jason is a person that takes a lot of like feedback like very well He's a very like he's not offended by harsh feedback and he rebuts well like online as well, right?
[00:16:29] But like I would imagine he will be the one that is the most like affected by Criticisms of emergence. I was believing in it, but I have to say that the paper, I mean, that's why he's the first author and I'm second, but that was mostly Jason's thesis, and I have to really say that Jason has really good ideas, and I was more of a support role for that paper, yeah.
[00:16:49] swyx: Sure, yeah, you know, lots more to discuss there, but you believe in emergence, that's enough for me to work with.
[00:16:55] Yi Tay: I also think that the, the, the Mirage paper is mostly like I don't know who, actually I don't even remember who wrote it. Ryland
[00:17:01] swyx: Schaefer. Yeah, I, I covered him on, on my NeurIPS podcast.
[00:17:03] Yi Tay: Okay, okay.
[00:17:04] swyx: He's a very good speaker, and the paper was well done. It's just that people drew the wrong conclusions from the paper. Because they had a very good title. Do you believe in emergence?
[00:17:12] Yi Tay: Of course. Okay, high five.
[00:17:14] swyx: I mean, how can you read any paper, read any, the progress of LLMs and not believe in emergence?
[00:17:20] It's so stupid. Like, just because you re parameterize some benchmarks in evals and, you know, make it linear, doesn't mean emergence is completely gone. And even in the Mirage paper, they acknowledged that there were some metrics that were true, genuine emergence, according to them. I think it was something like 25 ish percent in the ballpark.
[00:17:38] That's not the exact number, but it's in the ballpark. So I was like, okay, fine, like some benchmarks you disagree with, but on the whole, there is emergence, it's just, now we're just talking about the magnitude.
[00:17:47] Yi Tay: Yeah, yeah, yeah, for sure, for sure. I don't think the authors of the paper had really very Like they, they didn't, I mean, nobody, we, we should just assume people don't have bad intentions.
[00:17:55] Right. But like, no, they, they definitely were just doing this. But like the, the, I think the Popul media, I was more like annoyed by the nearest best people. I mean, okay. Best people was, let's take the thing, take of a grain of salt, right? Yes. But like, there were people come to me like, oh, you should care about this because it's the nearest best disprove because it's the nearest best paper.
[00:18:11] I'm like, paper awards like mean anything. Actually, it doesn't mean anything. Right? Like. I think that was more of my where my angst was coming from. I don't, I don't think I really had, I don't even remember who were the authors of that paper, right?
[00:18:23] swyx: I'm sure they're doing well for themselves, and we don't have to dwell too much on that.
[00:18:26] Quoc Le as manager
[00:18:26] swyx: Okay, so a couple more things from Google, and then we can go to Rekha. Kwok Le was your manager.
[00:18:30] Yi Tay: Yeah, yeah. What is I had another manager called Don. Like, I had two managers during my time at Google.
[00:18:34] swyx: So I'm just basically going to ask for quick hits from what did you learn from Kwok? What did you learn from Jason?
[00:18:38] What did you learn from, you know, Juan? Who they are, who they represent to you,
[00:18:42] Yi Tay: like, how they advise you and all that. So Kwok as a manager, he was more like a friend, and we would like talk a lot about, I think Kwok is a very researchy person, he has a lot of like, he's more of like an intuition, I learned a lot about like from him about like, there was no like concrete, like it was more like over time, and it was very implicit, soft kind of feeling, but I think like a lot of research science, we were like brainstorming a lot about like, , I quite like that, you know, when we were But there was this U palm paper that, that didn't like get much, like as much attention that I feel it deserves, but like, I think that was one of the works that I, I kind of like discussed with court quite a bit and like and that time you're releasing the fund two stuff and everything.
[00:19:16] And then like, I think court has a lot of good sense about like what makes a work a good hit and like you know, publicly a good hit. And like a lot of research sense about like what what makes like. like research cool, you know, so I think he has good like intuition as a researcher and I learned quite a little bit about and I also I was going to say that I think Jason also probably learned like quite a bit from Quark and and this also influences like more of like it was not only just like me getting influenced but that there was like Jason getting influenced and then Jason influenced me so I think overall what I learned from Quark's like intuition, research taste, people like chat about AGI sometimes, singularity and stuff like this it was Like, he's nice to talk to as a friend, manager, kind of, he's like kind of a friend figure to me.
[00:20:01] He's very much a researcher more than like a corporate, you know, manager kind of thing.
[00:20:06] swyx: I totally expect that. It
[00:20:07] Yi Tay: was fun, it was fun.
[00:20:08] Jason Wei
[00:20:08] swyx: Jason Wei, what did you learn from him? What is your distillation?
[00:20:11] Yi Tay: Okay, Jason is very interesting. So, I learned in my career, I learned two or three things, major things from Jason, right?
[00:20:18] So, I think the first thing I learned from him is that so Jason was actually, okay, I'm going to talk about the more casual, more fun stuff first. Jason was the most spicy on Twitter first before me. There was an era where I was a goody two shoes, I only had my main account, I only tweet my only tweets Newspaper alert, you know?
[00:20:34] Right. And then Jason was starting to post like, like hot takes, right? Yeah. And I just thought to myself, oh damn. Like, you know, and there were, there were types that I was like, Jason, you should not post this. You're gonna get cancer. Right. And he, he, he was fine. He, he always break through the storm and everything until I, I looked at him and I'm like, maybe it's not that bad after all.
[00:20:50] Just be, be, I love it. Right. So there was like kind of like, which is very interesting 'cause Jason is much younger than me and I. And the other thing also, our accounts, right, we created them around the same time, right? And the interesting story behind it was that Jason's account and my account has our own, our original it was not like an anime character that nobody know who is it.
[00:21:09] We have our identity. It's pseudonymous. It's pseudonymous, right? And then I asked Jason why do you want to have a So like, why don't you just make like, and he told me this thing which was quite true was that like, Okay, you can post a thing that is spicy and it's hot, but if you cannot stand by the opinion, then you should not have the opinion in the first place, right?
[00:21:25] Wow. Right, so there was something that, oh, okay, I thought that was profound because so far this, I mean, there are times where, okay, I post something and it's spicy and then, okay, it gets a little bit bad, and then I, okay, I kind of agree that, okay, this is bad, then I will retract it. But if I could stand by the opinion, then I would just stand by it because that's the point of making it It should be said.
[00:21:42] Right, it should be said because. I can put my name behind it, right? So that was This is part of the first bucket about how it kind of influenced my online persona like, a little bit, and then, I mean, and then it turns out that now AGI Hippo is so much more spicy than the cola The cola is just hibernating somewhere, it's not even around, right?
[00:22:00] So, I mean, Jason is also more constrained because he works for he has Like an actual employer, right? And he has to be a little bit
[00:22:08] swyx: more The worst thing about Twitter is that, you know, anytime anyone from OpenAI tweets anything, they're like Did you see this researcher from OpenAI said something?
[00:22:15] And they read tea leaves that are not there, and it makes you very cautious to tweet anything. And so it kills the golden goose, is what I say.
[00:22:22] Yi Tay: There was one tweet, I mean, at the time when somebody was, people were speculating the GPT 2 chatbots, right? And then Jason just posted something on his main account something excited about new experiments being run just a random just, and then people screenshot that, and post like, Yeah, I hate that.
[00:22:35] So I think, I, now I think for, All the count is mostly like personal, like personal stuff , very personal I think he would stay away from non work things non work things.
[00:22:44] swyx: The golden goose has been killed because people on Twitter cannot control themselves from drawing random conclusions from, you know, all these hints and all that Yeah, yeah, yeah, yeah,
[00:22:52] Yi Tay: but Going to like the actual, this is like filler, filler, this is not canon, it's filler.
[00:22:57] I think the second thing I learned from Jason is more about like, from my you know, kind of like, from my own career, it's like, the importance of like marketing and PR. So Jason is actually like, I mean, I was just like, he was actually like really, you know, the emergence, like how many blog posts you wrote about the emergent abilities and how many talks he's given about, about emergent, like a lot, probably like the other day I was just at this webcom keynote and he was giving a keynote again about emergent abilities and it's been two years, right?
[00:23:25] So I, I think one big success of him is that like he, he does the work. Okay. Thanks a lot about like marketing the work itself. I did not like in my early parts of my career, early parts in Google, right? I was putting out a lot of work, but I didn't put in a lot of like effort in like thinking about the, like how the work is going to be received.
[00:23:42] I'll just be like, here's a paper, here's a paper, here's a paper, right? But Jason will be like, I'm going to write this paper and I'm going to like market the s**t out of it. So I, I, I think I learned a lot about like, so every single first author paper that Jason writes in the last, he has like 1000 citations in one year.
[00:23:56] Oh my god. Like no, I mean, not every, but like most of it that he leads. So his hit rate is very high. His hit rate, like impact density, like it's very high, right? So, It's pretty interesting but I kind of see him as like a peer and I learn a lot from his basically some, some people are just like talented in, in different ways.
[00:24:11] And, and I think that like, I, I looked at how he markets his own work and markets himself actually, right?
[00:24:16] Marketing Research: How to Start from Zero with No Reach
[00:24:16] Yi Tay: If someone is starting from zero,
[00:24:17] swyx: like no Twitter presence, what is the second best thing to do? You mean as a researcher? For marketing, yeah.
[00:24:23] Yi Tay: I, I think you would like the, the, the most obvious. If you're like a re like say hypothetically, you're like a researcher in like a place without visibility or without an end.
[00:24:32] You have no personal visibility. The first goal is always to try to find a mentor or coworker that is like within this circle, and then you start from there, right. Because, and then you get, you get, you know. people from like, who has a visibility and following to retweet. So you will like, work with them the big goal is not about like, I learned this, I mean, this is like, probably a career mistake in my early days, was that you know, instead of like, focusing on like, so called people okay, if you do good work, it's more of like, okay, how am I going to I see this visible researcher from DeepMind, right, or how can I collaborate with this person, and then kind of do something that feels cool, and like, I can win their respect, and that they would like.
[00:25:09] You know, they will be willing to co author for me because the exercise itself was so about how to, you're not trying to please reviewers or anything, you're just, if you can find one semi visible, you don't even have to be like a famous person, that's like a semi few thousands of followers, has a good reputation of research, and then you collaborate with this person, and then when you post the work, you are co authored with this person, and then you get the person to vouch for you, or just, over time, this would It could be from internships, it could be from, you know, just DMs.
[00:25:38] I think, you know, people are nicer than some people, they seem scary, but if you DM them, they're actually willing to collaborate, actually.
[00:25:44] swyx: I was scared of you, actually. And when I DMed you, you turned out a lot nicer than I feared. So thank you for being nice. That's really great advice for people.
[00:25:55] I just want to leave that out there for people. For others who follow, you know the work that the career advice that I give, the title topic of this is pick up what others put down and specifically pick up what your mentors put down. Like, mentors always have more work to do than they have personally time for.
[00:26:09] The high visibility mentors, and if you can show that you're a good collaborator with them, they will lift you up. Accordingly, that's a pretty good formula for career growth. Should I ask about Hyungwon? Or I don't know how close you are. Oh, we're
[00:26:21] Yi Tay: still good friends. Hyungwon is a great engineer and he's very systematic in the way he thinks.
[00:26:26] I think Hyungwon is without going into detail, I still spend a lot of time talking to Hyungwon, even like in the, even after we both are different places, about like very interesting algorithm, arithmetic ways to think about life. Very interesting like, perspectives on life rather than research.
[00:26:43] But Hyungwon is a great engineer. And the one thing that scares me about Hyungwon is he doesn't have multiple monitors. He just codes with one small screen. And he does everything with very hyper optimized. And then I
[00:26:54] swyx: want those U curve where one screen, one screen, and then many screens.
[00:26:57] Yeah, yeah, yeah.
[00:26:58] Yi Tay: So I think Hyungwon scares me because it's like, I think that was at NeurIPS 2022. Like, we were doing some work at the New Orleans. And then He'll be coding perfectly fine with this 13 inch MacBook with one terminal, and then he'll be like, he keeps telling us okay, it's more optimal to using keyboard is more optimal than moving your head, because if you can switch your screen fast enough, it's faster than your head, like moving to different screens and stuff.
[00:27:24] I did not actually distill that, because it's too painful to do that, but it's very interesting in a way that I'm he belongs to one of those hardcore people with one monitor and
[00:27:34] What's needed to be a successful AI Researcher?
[00:27:34] swyx: Maybe this is a relevant question to just close out the Google site. What do you think is a good programmer for AI research?
[00:27:42] Yi Tay: You mean set up or eating? No, no, not set up.
[00:27:46] swyx: Not even lifestyle. It's more about skills. Like what should people have? What do you interview for, maybe? What do you see that the high performers do differently than less high performers?
[00:27:54] Yi Tay: I mean, okay, like generally, there's like, I think like for AI researchers, like being a strong IC is like probably like the thing that I feel like is like important for AI researchers.
[00:28:03] Like not, not I think There's a certain level of sacrifice to be an AI engineer, AI researcher, especially if you're training at LNs, because you cannot really be detached from your jobs could die on a Saturday at 4am, right, and then there are people who will just leave it dead until Monday morning, and then, or but there will be people who will crawl out of bed at 4am to restart the job, or to Check the, you know, TensorBoard or something like that, right?
[00:28:31] I think a lot of being a successful AI researcher, I don't want to say passion is also the entire thing, but it's more of just the a kind of personality that, if something, there's a bug at 3am on Saturday night or something, right? And then you would like, be like, you couldn't go back to sleep unless you, you, I'm not, this is very unhealthy by the way.
[00:28:50] People should not do this for a long time. You know, I think this kind of things actually like, allows people to make progress faster. But it's unhealthy, so I'm also not even sure like, what's like the, checking out on like, Friday, Saturday, Sunday, and like, 9 to 5 if you want to like, make progress, or like, some people are just so good at detaching like, okay like 8pm, I'm not going to My job can die and then the chips can stay idle for like the whole night, but I want to watch Netflix, right?
[00:29:15] You cannot, I think there's a level, it's like a sport you cannot win an Olympic gold if you want to have super, ultra good work life balance, right?
[00:29:23] swyx: Yeah, passion, intensity, dedication. Yeah, intensity, right? So those are really good personal qualities. Just technical qualities wise, how much of the stack should people know?
[00:29:32] You know, if I Okay, so
[00:29:33] Yi Tay: that was the question.
[00:29:34] swyx: No, no, no, but that was important as well. Okay. It's just harder to interview for because you really just see it on the job.
[00:29:40] Yi Tay: I think stack is not not, not, stack is not that, like Like, should I know CUDA kernels? I don't know CUDA kernels.
[00:29:45] swyx: Exactly, right? Okay, good.
[00:29:47] So for all you listening out there, you don't have to feel like an imposter. No, but, but you need to be willing to learn if you have to, I think. Well, you haven't had to so far. Yeah, I haven't had to so far, right. So if I sling pie torch, okay, great. You know, what kind of do, do I do I know like distributed systems, like do I know, like what, what is the, what is the stack that you recommend for people that get, gets you like a well-rounded end-to-end researcher.
[00:30:08] Yi Tay: I don't, I, I don't think there's any specific thing. In fact, I will try to be as I don't really say like, okay, you need to learn Jax, you need to learn this. By the time you think there's a new frame out anyway, so, so it's more of like. Staying like constantly, like trying to, being able to continuously learn and update.
[00:30:24] I, I don't think that's a single, single stack or like a single single like workflow or I don't think that's a single one. Yeah.
[00:30:31] Reka Origin
[00:30:31] Yi Tay: Well, that, that leads us to Rebecca. Yeah. What's the founding story? So, I, I met some of my other co-founders while we were collaborating at that did my end. I was at, at brand and they were like a DeepMind.
[00:30:41] I'm not like a, a, a startup person. I, I I, I identify even today. As a scientist and a researcher more than like a startup person, right? My co founder, Danny, started this story. Right. And then this, this record was like, in the works from like, late 2022. I, I finally left in 2023. Then he kept asking me, he wants to do something.
[00:31:01] Do I want to go with him and do it? And, and it took, took a while for for me. Also, I was like, kind of the last co founder to kind of form the Was
[00:31:07] swyx: the plan always for you to leave at some point and join him? No, no. He was convincing you to do it. It was
[00:31:12] Yi Tay: like, it was like a six months, more or less, in fact I think more than six months period of like, I always had this at the back of my mind for since like, what, August actually, I didn't want to do it in the first place.
[00:31:25] But I think eventually in March, I felt that okay, it's time for me to experience something new. Like, my leap of faith was more of like, I want to experience something new. I've, okay, I've like, Wrapped up this palm to work at Google and then like more of like, okay, let me experience this new life and see Where we can go with this and I also I mean, we don't have a lot of like, okay The funny thing was that like many many years ago before I PhD I wanted to do a startup actually at that point and then over time I realized that like I was better off as a researcher And I just forgot about the startup thing and it's quite funny that today I end up doing a bigger startup, right?
[00:31:58] but even until now I I actually I did identify more as like a researcher and scientist. Well, I mean, it's not, when you
[00:32:05] swyx: left you already had a high profile coming out of Brain. You could have gone to any startup out there. They all had wanted you. Yeah, okay, okay, yeah. So why did you choose this one, basically?
[00:32:13] Like, is it just because of pre existing relationships? Because, you know, It wasn't obvious to me. A lot of it, the other coworkers went to OpenAI, others went to, the, if you're, if you're fair, you went to Misra, you know, that kind of stuff. Right? Like Rico, Rico was like not on the, on
[00:32:25] Yi Tay: the map.
[00:32:26] Yeah. I, I, I think it was, for me, it was the ion between staying at, at, at Google and like co-founding something. I, I, I didn't want to like, like it was more of the experience of like being a co-founder. And this is like what attracted me, right, and wanted to experience that. I wouldn't have left for Inflection, or something like that.
[00:32:42] Like, I mean, Inflation is gone, but
[00:32:43] swyx: like RAP? They're still alive. They're selling themselves as a model foundry or something. I don't know, there's a services company now.
[00:32:52] Yi Tay: Yeah, I know, but I also think that like Like, for example if you were to join another, it would be like a very big tech experience again, right?
[00:32:58] I don't know, I felt like, the experience I get is very complementary to what I have that's the experience I had at Google, right? But if I were to join something else, right, then I wouldn't have, I would have just stayed at Google, to be honest. Because to me, it was very clear just two decisions that, that I didn't really I was talking to a bunch of other startups, but I didn't really actually had the intention to I was happy at Google, actually, to be honest.
[00:33:19] I'm sure,
[00:33:19] swyx: I'm sure they have a lot of things to keep you happy. I was happy at Google, yeah, actually.
[00:33:24] Starting Reka Infra
[00:33:24] swyx: So, you describe yourself as GPU poor, but also you had 60 million dollars to play with. You got a whole bunch of GPUs. I think you disclosed somewhere, but I don't remember the exact number. And you had a good training run for Flash and Core and Edge.
[00:33:39] How would you tell that sort of story? Like, people can read the technical report. But also you know, what was that overall experience like? And you should also point
[00:33:47] Yi Tay: people to the blog post that you wrote. There were a lot of interesting things that happened along the way that So I think I left around like early April, the end of March, April and everything, right?
[00:33:58] But most of our compute actually came in December, actually. And there were delays. So H100, there were major delays, right? So we were sitting around, right? And to be clear,
[00:34:07] swyx: you don't own the compute, you are renting.
[00:34:09] Yi Tay: Yeah, yeah, yeah. So we were sitting around. Like, we've, you know, for a long period of time, we had 500 A100s, because we made a commitment and they were constantly being delayed, I think because of H100 supply, demand, whatever reasons.
[00:34:23] And it was also very hard to get a lot of compute in one place, right? And then we were locked in, and we had to wait for the compute to come, right? So, I think, It was very painful because even when the compute came, it was mostly broken most of the time. And it was broken to a very bad extent that, you know, before I left Google I was like, even in the early stage I was very optimistic about You Okay, this compute translates to this amount of flops, this is the model, right?
[00:34:48] But I never expected the reliability to be so poor that it just threw off all the calculations and then we had to, work ten times harder just to make the thing go smoothly. So, it was a bearable pain. I think the pain was bearable, but it was just way, way more than expected.
[00:35:04] Why not to use TPUs outside Google
[00:35:04] swyx: I think you addressed this in your post, but the temptation would have been just to run everything on TPUs. Which is the stack that you already know very well. That that works very
[00:35:10] Yi Tay: well. Oh, no, no. So, so, so TPUs outside Google and t inside Google are probably very different things, I think. Oh, how come?
[00:35:16] Okay. First thing is like infrastructure. Like, there was, there wasn't like a lot of good code bases like outside Google that was like still, right. And, and the code base that I was most familiar with was like T five X. It was a jack space. It would have been like, by, by the time we wanted to consider it, it was really like.
[00:35:31] Debrigaded for nine months, right? And then, TPUs I mean, we weren't sure about I mean, the availability of TPUs was not great, great.
[00:35:41] swyx: Oh, my perception is that it was a lot better. People have the learning curve.
[00:35:44] Yi Tay: Yeah, but at the point of time, we had our infra set up, we were training already, training models, and it would be so much cost to, TPUs.
[00:35:50] So I think TPUs, the experience of TPUs inside and outside Google, I have not actually run a single TPU job outside Google, by the way, but just looking through documentation from what I see outside, it's great. And from like, how much I think that people inside Google don't care about what people think outside Google, I kind of feel like, okay, we were a bit like, I don't think we considered, I mean, not like forever not considering this, but like, just like, At that point of time, it was like, The obvious choice is to stick to PyTorch.
[00:36:15] Just stick to GPUs and PyTorch and make like, I mean, it's not as if the chips we ordered were not there, they were there, they're just not. In the best shape. Reliable. Right? Yeah. So I think it was too much work to, to kind of migrate suddenly to TPUs. Yeah.
[00:36:29] Chaotic vs Stable Infra
[00:36:29] swyx: For those who haven't read the report, you had a very traumatic description about the chaotic and stable phases of various compute providers, and I was just wincing when I was reading all those things.
[00:36:40] Yi Tay: Yeah, no, that was like a 3 body problem reference, the chaotic and stable phases. I mean, I was watching 3 body problems at the time, and I thought it was fun to, there was a lot of like, I think we had a lot of fun adding a lot of references and memes into the tech report. I think like, you know, it goes to show like how fun the environment is within, within record, right.
[00:36:57] We had a lot of fun with this, but so I think chaotic and stable face mostly. It's like we, we actually found that, like usually when like provider provisions, new nodes or they would like Yeah. You don't wanna be the first to use it. Yeah. It is usually like, like bad like dog s**t. Like at the, like at the start.
[00:37:13] Right. And then. It gets better as you go through the process of returning nodes and, and, , draining them, giving it back to them, they will send it back for repairs, and everything and then over time, because it's more of it's more of a numbers game, right? If there's one bad node, It kills the entire job, right?
[00:37:30] So like, the fact of, the game became like, just eliminating bad nodes from the thing, right? And then, you know, I mean, just because of, maybe because of the supply issue or something, when the deadline comes to ship this, for example I just give rough numbers, let's say you order 1, 000 H100s, right? They will not be able to, usually they don't meet the demand of like 1, 000 H100s at the date.
[00:37:49] They will give you like 500 first, just not to piss you off, and then they'll give you like another 100, like every over 3 weeks, they were just like, okay I added like 4 nodes, added like 8 nodes, that kind of thing. And then over time, you reach like the capacity that you, or maybe you never actually reached the capacity that you ordered for.
[00:38:04] Risk Sharing of Bad Nodes
[00:38:04] Yi Tay: And then as they add these nodes, right, sometimes these nodes are bad. And then they just kill entire training runs. And the thing, Which I feel that, I mean for all those people trying to sell GPUs, people trying to sell GPUs now resell, sell, package, whatever, GPUs, right? And I think the most important thing that, that they are obviously they are SLAs, all this, in the contract and everything, and obviously, you know, you might be entitled to something, something, if something goes wrong, right?
[00:38:26] The thing that, for, Large model training runs, is that like one bad note kills the entire job? Right? So should the compute provider be liable to pay for all the note waste stage that No. No. It, it's because it's unlikely because otherwise it's unrealistic. Yeah. No one will take that on. No, no, no one take that on.
[00:38:42] Right. So I think that's also like a, a tricky thing. Who, who is taking the risk? It's the, the LM startup taking the risk. Or is the compute provider taking the risk? I think that, I mean, this is my sense, I'm not 100 percent sure, but I think like as there are more providers trying to sell GPUs inbounds so much about people trying to sell us GPUs.
[00:38:59] Right? The key differentiator is actually to find a way to To balance the risk of node failure with as long as the provider, I'm not going to say 100%, but if somebody can come and tell me that my nodes are so stable that I can share some cost with you if your job dies, this is green flag, green flag, right?
[00:39:16] The moment they start to I cannot Do any of the big clouds do that? As far as I know, no. They have the, you know, the size to guarantee that. But I think, Like for anybody who is watching or if you do like a compute startup or anything, the biggest green flag would be to share the cost of node failures with your customers, right?
[00:39:35] You mean the whole run? No, no, like if the node, it's very hard to go, because you need software to like, you need software to, so let's say you run it for 12 hours, right? And it dies after 12 hours, right? You get 12 hours of throughput, right? But then you get like some wastage because of like the, the you know, the downtime and everything, right?
[00:39:52] You know, I, I think it would be fair to find some middle ground to kind of split the cost of the failures, right? And this brings back to my point about like, work life balance. Because if the nodes fail, fail so badly, right? Like, it, it actually, basically, right, your engineers cannot sleep at all.
[00:40:06] You have babies sitting in rosters and everything, but you are living life with like constant anxiety, because even in the case, right, where the node failures are refunded, right, you still lose time. You lose three hours. You lose everything, right? So I don't know how to go around this, but I think if there are a lot of compute providers like fighting over I think a good A good thing to do is to figure out this pain point, otherwise, or at least, , figure out some hot swapping, but so far, most of the providers that we tried don't have this.
[00:40:34] They will also get confused when you try to ask them so my job is dead can you pay for the food can you refund for, or at least, they will get confused because this is a LLM specific thing that the large nodes, They don't care about, yeah. Yeah, they get confused about this, right.
[00:40:48] So,
[00:40:48] swyx: current status quo is the LLM started to pay for everything. Thank you. Maybe you could negotiate some,
[00:40:53] Yi Tay: like, refunds, but usually they will not be so generous to pay for say you run 500 you break for 4 hours, they, in their mind, they will be thinking, I should refund you for one node, but in your mind, you just think that they should refund you for the full job, right?
[00:41:05] Checkpointing and Orchestration
[00:41:05] swyx: Everyone who is from my background is going to be asking this. How is it so fragile? Like, how is it so brittle? Like, what's your frequency of checkpointing?
[00:41:13] Yi Tay: Our checkpointing is kind of like we, we see how stable the job is and then we decide, because checkpoint, it takes a we without a good file system checkpoint, it takes actually quite long.
[00:41:21] So it could be, it's like a few
[00:41:22] swyx: hundred gigs, right?
[00:41:23] Yi Tay: Yeah. I, I, I think so. I think so. I, I, I, I, I don't remember offhand, but , that doesn't take that long, but No, no. But sometimes if your, if your file system is slow, right? Your file IO is slow, your checkpoint thing could, for 20 B model could be like, what?
[00:41:35] 30 minutes or something like that. Okay. I don't know this by heart, by heart, by heart. Sure, sure, sure, but it's not hours. If you go larger, what if it's like a 200 bit
[00:41:42] swyx: model, right? Okay, so you should have some kind of ideal checkpointing to run ratio that is not catastrophic if you run into a node failure.
[00:41:50] Yi Tay: Yeah, no, so we see of it as like, like a MFU, like, because you can average out your your flop utilization, and then you can see how many percent hit, like, how much slowdown, right? So you probably go for something like, if it's like, you're taking off 1 percent of your speed, 2 percent of your speed, so basically, it's actually fine to just checkpoint more regularly, right?
[00:42:09] So I think checkpointing, like, you also never fully, you can get, like, from the clean slate, like, nothing, right? If, as you optimize, like, engineer, like, the system to automatically restart everything, you get some of the time back, but you'll never be, like, Like, perfect, perfect. Like, so you still lose, lose stuff like that.
[00:42:25] If you checkpoint too often, like, what, every 30 minutes, then your file system is going to blow up, right? If you're going to checkpoint every, like, like so, like, for us, we just see it as, like, how much Storage is cheap compared to compute. No, when your model is, like, very, very large, your storage can, can, can easily blow up.
[00:42:40] Going on to the models, I feel like
[00:42:41] swyx: I digress so much about all these fun side things. You like compute, right? You like, you like hardware and compute, right? I love hardware and compute. Oh, and also, I'm an orchestration guy. Yeah. So, one part of the question, one of the questions I'm skipping right now is, you know, there's, I came from Temporal, I'm familiar with Kubernetes, I've used Airflow, These are all the data eng, cloud, or cloud engineer type tools.
[00:43:02] It's surprising to me that you guys don't have your set of orchestration tools that you, that is solved, right? You wrote in your blog post you had like, the pain of multi cluster setups, and like, to this, to the rest of us, this is completely solved.
[00:43:14] Yi Tay: Okay. . I don't know if you know that. We use Kubernetes for, for a bunch of stuff, but like, I think like for experimentation and like stuff like this, it's still not fully, like we, we, we didn't have like the time to actually like, like, like build something that is, it should exist in open source.
[00:43:29] Someone should have done this.
[00:43:29] swyx: Okay. Okay. I'm not, it is what it is, but I'm surprised that's all. Okay. Say it seems like a valuable problem and someone much should do it. .
[00:43:37] Yi Tay: Okay. Okay. Okay. Yeah, yeah, yeah, yeah. Good
[00:43:38] swyx: to know. Good to know.
[00:43:39] Reka Flash/Core/Edge
[00:43:39] swyx: Okay, so Rico Flash Core Edge. You know, congrats on beating a whole bunch of state of the art models.
[00:43:44] Especially much bigger than, than, than each. People can see the papers for all the other stuff. Was this your expectation from the start that you would basically definitely be frontier? Like how do you, like, from the start of like, you haven't trained anything yet and you're about to kick off the run, like, are you able to like call your shots and say, we will beat GP 3.5?
[00:44:02] Yi Tay: Nobody can predict the future.
[00:44:03] swyx: generally?
[00:44:04] Yi Tay: No. How much confidence? Okay. We were confident. Like, we were confident. How? Why? Right. It's a good question. 'cause it'll be, it'd be
[00:44:10] swyx: a shame to do a whole bunch of work and then end up this in the middle of the pack, which a lot of people end up.
[00:44:14] Yi Tay: We were confident. I think that a, a lot of it was like Yolo. I mean, I'm, I'm, I'm mentioned in, in, in, in the thing. I think we would. Like, require a lot less iteration than this because of our prior experience in like training these models. Like, so I was confident in myself about like our models will turn out to be, to be, to be, to be good.
[00:44:32] And I, about exactly how, I actually don't really know. Like, pinpoint to a particular reason of like, I mean, we de risk stuff, so a lot of part of it is like de risking and like, okay, you run like 4B applications and you can see, okay, this is like my spice, if you run 4B and your loss is like going crazy, you know that this is going to be a s**t model, right?
[00:44:52] But I think it's like, we trained enough, like, okay, we don't have a lot of compute to do a lot of applications, but we did enough experiments to know that, ah, okay, our infrastructure and our, like, everything is set up to be good, right? Obviously, You know, the field moves, right? I won't say that everything was like, smooth, like the first time around, it's like smooth and everything, but I think we were confident in our ability to like, make the list, like we're not, like, really, we're more confident about, like, the ability to like, Move with as little steps as possible to the goal, more so than, like, my model is going to be this, like, level at this time, you know what I mean?
[00:45:30] It's more of like, , for example, let's say we run the first round of human evaluations, right? And then we see our number is this, right? And then we are confident that in five more tries, we will get to this. Kind of like get, get to like, like, like, like this. It's more of that kind of confidence rather than actually like, you know, it's also a little bit of like, you know, you see a new leaderboard hypothetically, like in academic.
[00:45:51] Like if as a researcher you see a release, a, a new leaderboard, right? You, you approach it like a puzzle. You don't know like. Whether you at the start of it, you might not have the answer to the puzzle, but if you're good at solving puzzles, like generally, right, you know that with one hour, I'll be able to solve it.
[00:46:07] You know, that kind of confidence, like, it's like, you know, it's the ability to, to hill climb or the ability to, to improve over arbitrary things, right? Rather than, I think we were confident more about that rather than like, Like, everything is different, right? The stack is different, the infrastructure is different, the data is also different from what, I mean, we have a lot of, which you
[00:46:25] swyx: haven't talked about, right?
[00:46:25] It's just, we have a lot of,
[00:46:27] Yi Tay: yeah, we have a lot of experience from prior, like, our jobs, but, like, it is not going to be that, like, we don't have actually, like, exactly the same thing because, , different companies have different stacks, different everything, right? So it's more about de risking, being confident in, like, solving the general problem of, like, improving over things which is why also I think that the team is valuable in the sense that we are not, like, valued by our model itself, but we are just valued by how we can see one problem and we can just solve it super quickly.
[00:46:55] And that's what we are confident about, actually, like the artifact itself.
[00:46:59] Recruiting the team
[00:46:59] swyx: Mentioning your team, you said at the largest your team was 3 5 people on the pre training side. Was that the team that you recruited? Was it all your ex colleagues? How do you find people that, you know, would have this kind of solid intuition?
[00:47:12] Yi Tay: So I think that some of the people in our team were like, I worked with them at Google, at ex colleagues and stuff, and some of them were like fresh hires, like they were like fresh PhDs or like and everything.
[00:47:22] Noam Architecture - Swiglu, GQA, RMSnorm, ROPE
[00:47:22] Yi Tay: Okay, so,
[00:47:23] swyx: I do want to comment on Noam (Shazeer) architecture. So if you want to, people have variants of all these.
[00:47:27] swigloo, gqa, rope, rmsnorm, and then obviously the big one is encoder, decoder versus decoder. Could you comment on each of those, like, were you just like, we're confident that no one got it right? Or did you actually do an evaluation of each of your architecture choices?
[00:47:40] Yi Tay: Oh, I mean like, okay, architecture wise is something that I feel like I'm easily able to, like, I've run so many architecture experiments that, like, I look at architecture and I'm like, okay, I don't want to be, like, overly, like, I think it's very hard to outperform the old genome.
[00:47:57] Why? It can't, I mean, on the surface of it,
[00:47:59] swyx: like, we have to have learned something in the last, like, No,
[00:48:01] Yi Tay: all the changes, all the changes that, like, Swiglu was this, like, okay, Swiglu is probably one of my favorite papers of all time, just because of the divine benevolence, like, the Noam (Shazeer) actually wrote, like we owe this success to divine benevolence, like, that was, like, it's always a meme thing, right?
[00:48:15] Okay, so, like, GQA, MQA was always, like, the multi career type, was always, like A big controversial thing because MQA usually you get a hit because it's MQA and everything so people kind of know that like it was a very hit or miss like it was like it could you could get a hit in a performance from MQA like MQA alone MQA was always like You know, the choice, right?
[00:48:36] It's always like, okay, should we use MQA, should we not use MQA, right? When GQ came in, right, it became like a no brainer to use GQA because you don't get the hit anymore, and then you just get the fast, like, inference benefits of GQA, right? So I think GQA I mean,
[00:48:49] swyx: 2 now. Yeah,
[00:48:50] Yi Tay: yeah, yeah. So, so, I think Lama 2 already.
[00:48:52] I'm not 2,
[00:48:53] The 70, 70 GQA, right? But, I mean, the reason why we call it Noam (Shazeer) Architecture because MQA came from DOM and GQA was like a follow up paper by some of my colleagues at Google, right? So I think GQA was, became a point where, okay, this is already accepted, like, it is good enough, like, it's a no brainer to use GQA.
[00:49:09] SuiGlu was an interesting thing because there was a very long period of time, so SuiGlu was a single author paper by Noam (Shazeer), and very few papers were, like, SuiGlu had very few citations, like, at the start. Only Google Papers was citing SuiGlu at one time, and a lot of them was like, like, I was like, at one point I was like, probably like, 30 percent of SuiGlu citations.
[00:49:27] Because every time, Like, SuiGroup became popular because of the updated T5, the T5 1. 1 that uses SuiGroup, right? And nobody actually really cared about SuiGroup for a long time, because I was checking why is this underrated paper not getting much citations, and then I think probably now it has like a few hundred citations by now.
[00:49:46] But I think SuiGroup is one of the things that I played around with a lot at Google. So SuiGroup really works. There was also a paper we wrote about Like, do transformer modifications, blah, blah, blah. Like, it was a paper with Noam, and Sharan, and Hyongwan, and stuff like that. And then, we ablated, like, so many transformer variants.
[00:50:06] Yes, yeah, I saw that. Some
[00:50:08] swyx: of them matter, but most
[00:50:09] Yi Tay: of them don't. Most of them don't. And then, the only thing that mattered in that two part paper was, The paper was, in the paper was Swiglu, I forgot which exact Swiglu variant was it, but Ansposity at that time, right? So, so that was strong enough, like, to finding, to
[00:50:23] swyx: For, for the listeners, this is the inductive bias scaling loss versus model architectures, how does inductive bias No,
[00:50:28] Yi Tay: no, no, not this one, there was another one, like to transformer modifications, something, something, something.
[00:50:33] Because portal auto was run, I think. It was run around,
[00:50:35] swyx: You gave the keywords. Yeah, yeah.
[00:50:37] Yi Tay: I think the rms norm rope thing Not controversial. Like, it's, it's, it's not like, like, like, Obviously, I think rope is probably, like, it has that extrapolation thing, which is nice. And then, like, like, it's also, like, default now.
[00:50:51] Nobody wants to add positional embeddings anymore, right? And I think, I mean, I like the T5 style relative attention for a bit, but like, I think, okay, Rope is I actually ran that emulation for Palm, like the T5 relative attention versus Rope. I think Rope is similar to other things, but it has this extrapolation thing, which is nice, and like
[00:51:09] swyx: Which is why your long context version can go to 256.
[00:51:13] Yi Tay: For most of the long context models, they use the Rope extrapolation thing, which is a nice property, right? So that was for Rope. I think there were also some things like the layer norm, like partitions and stuff like that, that were like, it mattered a little bit, maybe not too much and everything. But I think in general, there was not a lot of like, there are not a lot of things that people could do to the transformer.
[00:51:33] It's been like 4 5 years, right? It's amazing. The vanilla transformer, I think if you use it as it is today, will not be like that optimal, but like The transformer that we slowly evolve to now is like, Like the Noam (Shazeer) transformer is probably like very, very, very strong baseline that is very hard to like, I think you need a drastic shift to, to beat that, right?
[00:51:55] Or you could find like more like, like Swiglu is a small change, right? You could find like some small change that are like a big enough impact, widely that don't cost a lot of , because a lot of architecture changes, right? The moment they are Tedious to implement. Like, nobody, SQL is a simple thing, right?
[00:52:09] It's a pretty uneducated thing. It's a very simple thing to implement. Maybe that's why it's caught on, because it has, like, an additional boost. That's for the simplicity of it, right? So there's also a bit of implementation lottery, if you will, right? A little bit of if you propose, some very complicated thing for, like, 0.
[00:52:24] 1%. Yeah,
[00:52:25] swyx: nobody will use that, right?
[00:52:26] Encoder-decoder vs Decoder-only
[00:52:26] swyx: The biggest, biggest, I mean, I can't believe we're taking so long to come to this topic, but the biggest Noam (Shazeer) architecture decision is encoder decoder versus decoder only.
[00:52:34] Yi Tay: No, so encoder decoder is not like a Noam (Shazeer). The Noam (Shazeer) architecture is more like
[00:52:38] swyx: the Okay, maybe like more old school transformers.
[00:52:42] Maybe we want to just talk about the Decision on encoder decoder versus decoder only.
[00:52:46] Yi Tay: So I, okay, I won't be able to comment about like exactly our setup, but like, I think encoder decoder are kind of very misunderstood from thing, right? So there's encoder decoder, non causal decoder, which is a prefix LLM, and then there's a decoder only model, right?
[00:53:02] Technically, a causal decoder and a non causal decoder are very similar in the sense that it's just a bidirectional mask, right? And then a prefix LLM decoder has only The only difference is that Encoder Decoder splits the inputs and targets into different non shared transformer stacks. And then, like, there's encoder bottleneck in the end, right?
[00:53:22] So, technically, people, like, kind of always associate, like, Encoder I like BERT, or like something like, like, you know, people get confused about these things, right? But I think in the UL2 paper, we really, like, kind of explored this, and also, like, maybe some of the big science papers that also talk about this, right, is that prefix LLM and causal decoders are very similar, that's a must.
[00:53:43] At the end of the day, they're all autoregressive transformers. That's actually, like, the only big benefit of encoder decoders, it has this thing called, like, I mean, what I like to call, like, intrinsic sparsity. So basically, an encoder decoder with, like, n params is, like, basically, if it's, like, It has the cost of like an N over 2 decoder model.
[00:54:01] So it is a bit like a sparse model because you actually spend the same amount of flops. It's just that you have two sets of parameters, like, for encoder and decoder, right? So it's actually flop matched with a decoder model of, like, half the parameters. So like a, like UL220B is actually about A 10 B decoder only model.
[00:54:18] Right. So you get free sparsity from that. It's, it's something that, okay. The, the, the, the OG T five paper talks about this. You, you can look at it. There's this complex detail. I, I did, I didn't like, when doing the UR two paper, I kind of like was mind blown by like, like, wow, I could decode so much more not bounded by The causal mask anymore.
[00:54:35] A lot of the efficient transformers, like a lot of the sparse transformers, like, I mean, the old, early days, that's like, , Linformer and like, whatever, things like this, they cannot maintain the causal mask, and that's why you cannot train a proper language model with this, right?
[00:54:47] If you separate out your very long context into an encoder, this encoder has no loss. Right, you could just do like aggressive pooling, you could do some crazy sparse attention that has like, final transformer or something like that, right? And then you could make that smaller than the decoder, you could make that faster than the decoder, that are just some of the advantages of like, why, , splitting into encoder and decoder could be beneficial to, like, just using a decoder only model.
[00:55:15] At the end of the day, the decoder in Encode decoder is a language model. It's still a regular autoregressive language model. So that's actually, I mean, it's not that much different from, like, a retrieval augmented language model. This is news to me. I don't know if you've ever expressed this, but
[00:55:30] swyx: yeah, this actually makes sense.
[00:55:32] Okay, okay, yeah, yeah, yeah. I don't, unfortunately, I don't know enough to push back on this, but on the surface of it, it seems to make sense. Would you make the same choices if you were not so focused on multimodality? You know, that's one of the ways in which I was thinking, like, Oh, encoder decoder makes sense, then it's more natively multimodal.
[00:55:48] Yi Tay: I just have to say that it's relevant, it's also relevant, yeah, it's relevant, yeah.
[00:55:52] LLM Trends - Llama 3 and Phi 3 Glowup
[00:55:52] swyx: Then we can move on to broader trends in LLMs, just commentary on the ecosystem stuff, like, completely independent from Weka. Commented on a few things, like, Lama 1 to 3 glowed up a lot. I call this the Lama 1 to 3 glow up, like, it improved into, like, an actual top tier.
[00:56:06] Open source model. Yeah. PHY 1 had a lot of criticism, but it seems like PHY 3 is getting a lot of love. Do you just generally see, like, in your open model tier list, like, what's going up and down?
[00:56:18] Yi Tay: I think Lama 1 and Lama 2 are, like, quite mid, right? But Lama 3 actually got Good, right? I think Lama 3 is actually strong, right?
[00:56:26] I don't really follow Firewatch, it's just that Their whole
[00:56:29] swyx: thesis is the textbooks is all you need thing, right? Like that we can, well, we can use way less data than everyone else and still
[00:56:34] Yi Tay: But I think you cannot cheat the scaling laws, right? Because, like, you, I remember saying, like, vaguely saying that, like, Like, oh, they match, like, Mixtra 8x22, or like, something like that.
[00:56:44] On, like, some Okay, I don't think these academy benchmarks are, like, that meaningful anymore, right? So, but then, like, then when you go, they go on LMCs, And then they get, like, maybe it just, like, seems slightly Maybe it's like I don't know about 5. 3. 5. 3 was
[00:56:59] swyx: just released like yesterday.
[00:57:00] Yi Tay: Oh, I don't even, I didn't even, yeah, but I don't know.
[00:57:03] I think there's some, I don't follow 5. 3 that much, but I don't, like, a model that is synthetically, Actually, I don't even know this, I didn't even read the paper, but I think that a model that is based on the premise of distilling and stuff, something like that, is like, Not that interesting to me, but I think that like Lama tree actually shows that kind of like meta got a pretty good stack around training these models.
[00:57:25] Oh, and I've even started to feel like, oh, they actually, you know, kind of maybe caught up to Google now, right? That kind of feeling. That's also maybe a hot take on itself. But, but yeah, I mean, fire, I don't really kind of follow you that much. And I, I just, There's too much, too much things to follow. So I think it's like, I, I, I think like Lama Tree is probably like the most, the first most legit.
[00:57:46] LLM Trends - Benchmarks and Evals
[00:57:46] Yi Tay: When you say these kinds of things,
[00:57:47] swyx: like most legit, obviously there's some, there's vibes eval or whatever but I feel like a lot of people, the very common feeling is MML is kind of saturated. Yeah. So like, what do you look at now? Is it just LMSYS?
[00:57:59] Yi Tay: Okay, so I think that LMSYS has its problems also. So LMSYS is not exactly like I mean, it's probably better than all these regular benchmarks, right?
[00:58:08] But I think, like, a serious LRM that's created their own evals, and a good eval set is one that you don't release.
[00:58:14] na: A good
[00:58:15] Yi Tay: eval set is the one that you, like, okay, you release some of it, but, like, it's like, you don't, like, you know, let the, like, Let it be contaminated by the community. Yeah, I think iOS 6 is probably the most legit one.
[00:58:28] I mean, like, you know, the things like GSMK, human eval, the coding, they're all, like, Contaminated. Like, not, not, I would say, they're all, like, saturated, contaminated, no, like, you know, GSMK, whether you're 92, 91, like, no one cares, right? That kind of thing, right? But we still report three decimal places in all of our reports.
[00:58:46] Yeah, yeah, yeah, but it's kind of like, almost like this, like obligatory thing to do. You have this table of numbers of your thing at the bowl. It's interesting to see how the, the field evolves also over, over time for, for, for this type of, like, benchmarks. But I think evals are going to be important, and it's on the, actually, interestingly, it's on, probably, probably on the academics to, to set the correct.
[00:59:03] I mean, they, they have Like there been, academics have always been like, like, oh, we have no computer this, but like, okay, this is your chance to like steal the field in the right direction. Right. I think the, the
[00:59:11] swyx: challenge is getting attention so, you know, now MMLU, you know, is reaching its end of its life.
[00:59:16] Like what, what is next? Right? There's MMU or there's MMLU hard, which someone recently released. There's Pro MMU Pro, I think it's pro. Oh yeah, that's right, that's right. Pro. But like that only lasts you like a year. Right, and then, you have to find something else. So, I don't really know what is that.
[00:59:32] Well, so, one thing, you know, you had a comment, I think, in your breakup paper about there's two types of evals. This is a Vibe eval paper. One is LLM says judge, and then two is arena style. Right, that's sort of the two ways forward for just general evals that cannot be gamed.
[00:59:48] Yi Tay: Oh, no, there's also Human evals that you, like instead of LLM as a judge, there's also like human evals that you run.
[00:59:54] Like that's kind of similar to Arena, but kind of different to SummerStand also. Different in the sense that like By
[00:59:58] swyx: the way, do you use your own staff to do that? Or do you like hire an outsourcing firm?
[01:00:02] Yi Tay: No, we don't. We have like, we work with third party data companies to like, there are a bunch of these like around, right?
[01:00:07] But like, obviously we don't like eval them ourselves. Like,
[01:00:12] swyx: I don't know how much, how many evals you want to do, right? Like, I do think Andre Capalti mentioned that. Sometimes, like, the best researchers do their own evals.
[01:00:19] Yi Tay: Yeah, looking at the outputs and stuff is something that, like, researchers should do,
[01:00:25] swyx: yeah.
[01:00:25] Yi Tay: Well, there
[01:00:26] swyx: is one element of parametric evals, which I'm hoping that more people come up with, where, like, you kind of The benchmark is formula is generated from a seed, let's say. And you can withhold the seed, or like, you can vary the seed, like, you can report how your model did on the benchmark, given a certain set of seeds or whatever, and you can maybe average them.
[01:00:47] But in that way, it becomes harder, much harder to contaminate. I wonder if that is an example of this. Not specifically, this is just something I'm wondering for myself, but I did someone did recently put out GSM 1K which was Oh,
[01:00:59] Yi Tay: the scale thing. I think,
[01:01:01] swyx: is it scale. ai?
[01:01:02] Yi Tay: Yeah,
[01:01:02] swyx: yeah, yeah. Which this is some similar in that respect, like make it easy to make variations of a, of a one known benchmark, but like that is more likely to be withheld from from training data.
[01:01:11] Yi Tay: Yeah, yeah, yeah. But eventually those will work. Like, so it, it's always a, like, like even we put out vibe. We also are quite, are quite like upfront with like, if the more people use it, there's a lifetime. It's like a car right. After you drive, run, run a certain mouse, it, it is time to shelf it. Right? Yeah. So I, I don't think there's like a, actually like a.
[01:01:29] Like a good solution. In general, I'm also like a bit I think this is like important for the community to think about, right? But like, is it like a fundamental limitation that any benchmark that goes out? Like, also there's also one thing is that in the past people used to like withhold test set, right?
[01:01:42] Like squat or something. They used to withhold test set. But then, like, after a while, I think people also realize that like, when you withhold, like MMMU, no, like when you withhold, it's like so much extra work for like the community to like eval on this that they just don't do that, right? It's either your.
[01:01:57] Dataset becomes, your benchmark becomes unpopular. I think it's also incentive things, right? So if you, let's say you are, you want to run like a contest, right? And then your goal as an academic is to get as much citations as possible on this benchmark paper, right? Like, then you, or like this, this, you want to be as famous as possible.
[01:02:14] You will not want to withhold the test set, because if you withhold the test set, and then people have, like, there was once, like, I mean, like many years ago, There were even some benchmarks where you had to, like, package your model and send it to them to run. And, like, these benchmarks never ever, like, took off.
[01:02:28] Like, took off. Just because, like, so at the end of the day, right, it's, like, It's the root problem, like, incentives. Like, it's the, also, the benchmark, the benchmarking problem is also, like, an incentive problem, right? So, like, it's also, like, like, people want to show their model is the best. And then the game masters want to gain as much clout as possible.
[01:02:42] And I think, also, LMCs also get caught into some, I don't have a, I don't have a take on this, but, like, there's, like, people who also feel that, They are also optimizing for hype, right? Their own cloud, right? So there's all this, I think it's a lot of interesting, like I don't know what field this will be, but I don't know, like, I think there's a lot of papers to be written, right?
[01:03:00] I mean, about how these incentives like rewards and incentives, like, kind of it might not be soft, so, I don't know.
[01:03:06] I would
[01:03:06] swyx: say SweetBench is probably the one that's kind of broken out this year as like now a thing that everyone wants to compete on as if you're a coding agent. I don't know if you have a view on it, but it's just, like, it should be known to be hard.
[01:03:17] You should be able to make progress on it quickly. That makes you popular and cited a lot. Yeah, yeah, yeah, yeah, yeah.
[01:03:25] LLM Trends - Early vs Late Fusion Multimodality
[01:03:25] swyx: Multi modality versus omni modality. So this is a little bit of commentary on GPT 4. 0 and Chameleon. I don't know if you saw the Chameleon paper from Meta.
[01:03:33] Yi Tay: Briefly saw it yeah, I'm not, I didn't really take a look at
[01:03:36] swyx: it.
[01:03:36] Basically, the general idea is that most multimodal models, like Lava or Flamingo, which are late fusion, which is you freeze, freeze, and then you join together, versus early fusion where you do it properly, where, like, everything is, you know, All the modalities are present in, in the, in the early training stage, and it seems like things are trending from late fusion to early fusion.
[01:03:55] Is is the general thesis with GP four Oh being very obviously early fusion, you guys, I I would class it as early fusion. I, I, I don't know if you have commentary on whether this is obvious to you or this is the, this is the way, or they'll just be, they'll coexist.
[01:04:11] Yi Tay: I think whenever possible, like early fusion is better, I think there will still be a lot of work steps.
[01:04:16] Dual late fusion just because of like it's a GPU, poor No, no, no. GPU. Okay. Par partially, right. I, I see this as like an art, as an artifact of the line between language research researchers and vision researchers, and more of like, okay, like people who are training language models, they put out like LAMA whatever, and then somebody takes it and then.
[01:04:36] Do Lakefusion on top of it. It's more like a It's Conway's Law. They're shipping the org chart. Yeah, yeah, yeah, I think so. I don't know what law it was. Conway's Law. Okay, I didn't know about that. But it's kind of like an artifact of the organization, don't you think?
[01:04:50] swyx: No, it's just because people don't have money to train things from scratch.
[01:04:53] I don't know.
[01:04:54] Yi Tay: No, no, I mean, even in big companies, right? I mean, I don't know how things have evolved in many companies, but like You're talking about Flamingo? Like language and vision and Teams don't use to be the same team. Right? Yeah. So I think this is like a artifact of, of this, but as early fusion models get more traction, I think the, the, the, the teams will start to get more and more.
[01:05:14] It, it is, it is a bit like of how all the tasks that unify like from 29, 2 0 1 9 to like now is like all the tasks are unifying now is like all the modalities unifying. And then I think like eventually everything moved towards like early fusion. Yeah.
[01:05:28] swyx: Yeah. The other element of multimodality is I, I've been calling this screen modality.
[01:05:32] Screen vision versus general vision, in the sense that Adept is like very, very focused on Screens, tables, charts, most vision models focus on things in the real world and embodied, sort of, images. Do you have a view on the usefulness for this?
[01:05:50] Yi Tay: I don't think that's like a huge, like, I mean, I think at the end of the day, like maybe screen intelligence is like more useful in general, but like, what if you have like a natural image in the screen?
[01:06:00] Yeah, I mean, no, no, no, I think at the end of the day it should be mixed, right? If a model can do natural images well, it should be able to do screen. Wow, and everything. I think at the end of the day, like, the models would become like, I don't, I don't see that there will be like, like, screen agents and like, natural image.
[01:06:16] Humans, like, you can read what's on the screen, you can go out and appreciate the scenery, right? You're not, like, say, I only can look at screens. Right? So, I mean, I think eventually the models would, like, be this good on everything. I look at it from a point of, like, capabilities. And screen is, like, you know, there's even screen that's also, like, , like, mobile phone screen and there's also, like, you know, laptop screen, like, also, like, you know, Different type of interfaces and everything like reading emails, whatever, right?
[01:06:38] But like reading a page from a website or like, you know, buying something from like Amazon or something like all kinds of things, right? And then even in the picture of like a shopping website, there could be like a natural, like for example, like picking Airbnb, right? But like, there's then there's a natural image in there.
[01:06:52] Then it's like, you have to understand like how nice is the scenery, right? Or like, , like, where is it? Right? Like, so I think at the end of the day, it's probably like the same. If you want to build a general model. Yeah, yeah, yeah. But I think The natural images is like, way easier, like, as in, just way, like, the models currently, current models are actually already very pretty good at, at this natural, natural images.
[01:07:12] And I think, like, screen images are just something that people need to enhance the capability a little bit more, that's why there's, like, some focus on.
[01:07:19] swyx: I'll touch on Three more things, and then we'll just go to career stuff.
[01:07:22] LLM Trends - Scaling Laws
[01:07:22] swyx: Scaling laws. Palm 2 was Chinchilla, which is one to one scaling of model parameters and data.
[01:07:28] Now you are training a 7B model with 5 trillion tokens. What are you thinking about the trend in scaling laws for data?
[01:07:35] Yi Tay: Chinchilla scaling laws are just like optimal for like with this amount of compute, how much is the thing, right? But like actually the optimal Like, there's no, I mean, this is something that even before I left, we already knew that, like, Chinchilla scaling laws are not the end of it, right?
[01:07:48] Obviously, there's also a inference optimal scaling law, which is, obviously, you take a small model, and then you just blast it with as much compute and data as you can, Until? Until you saturate on everything that you care about, right? So I think, like, Lama tree is for what? 15 T tokens or something, right?
[01:08:03] So I think Which is ridiculous. It is ridiculous to be honest. But at a certain point of time, your value per flop is not great anymore because you just, you know, your models eventually get saturated. But then the problem of, like, the question of, like, where is this saturation is also, like, you always find, like, some metric that you still continue to improve a little bit, and then you're like, okay, maybe, like, like, Oh, 100k it to continue training, like, just a little bit more, right?
[01:08:27] But then it's like, where does it end, right? But I think at the end of the day, like, the thing about Chinchilla scaling laws is that it was a bit misunderstood as though, like, like, this model, you need this compute, and, and, and if you train the Chinchilla scaling laws, like, you kind of, like, Like, I don't know why so many people had this idea that you will not improve past the Chinchilla scaling law.
[01:08:46] And then, people make so much big deal about trading past Chinchilla scaling law, like, Oh, Lamaldu is the first model. Like, T5 base, right, was 1 trillion tokens. That was already so much beyond Chinchilla scaling law, right? Because that was T5 base,
[01:08:58] swyx: right? I think OPT and GPT maybe set that as an industry standard.
[01:09:03] It's GPT 3 specifically. No, sorry, wait, GPT 3 was not Chinchilla.
[01:09:07] Yi Tay: No, I think like OPT and Bloom, right, models like this, they train a large model and with a very small number of tokens, and the model turned out to be bad.
[01:09:15] swyx: Yeah, yeah, so I'm talking about Kaplan, the pre Chinchilla one, the Kaplan scaling loss.
[01:09:20] Yi Tay: Oh, okay, okay, I see, I see.
[01:09:21] swyx: That one was from OpenAI. Anyway, dev of Chinchilla covered. Agreed. But Trinidad is still a cool paper, I think Trinidad is still an
[01:09:27] Yi Tay: important paper. I love any
[01:09:28] swyx: scaling laws paper, to be honest. It's like, such a service to the community, in general. Hugging Face recently did one, Datablations, which is like a data scaling laws paper, looking at data constraints, which was kind of nice.
[01:09:41] LLM Trends - Long Context vs RAG
[01:09:41] swyx: Long context, people are touting million token context, two million token from Gemini, magic is everywhere. talking about 100 million tokens. How important is it, do you think? I think we need
[01:09:52] Yi Tay: to solve benchmarks first before solving long contacts. We have your benchmark. No, no, no, no, not like benchmarks for long contacts.
[01:09:57] OK, yeah. because the needle in haystack is basically like an MNIST, or like a unit test for these sort of things, right? But I think there's one But about, like, hitting the context line and the other part about, like, actually Utilizing. Utilizing, right. I think Gemini's long context is surely, like, amazing.
[01:10:13] Right, but I think, like, for the community to move forward in this, then it comes to a problem of, like, How do we evaluate this? I think I've seen some long context benchmarks, like, coding one, like, And stuff like that. Like, I think Making those are important, and for the community to heal crime, but I think long context is important, it's just that you don't have a very good way to measure them properly now, and yeah, I mean, I think long context is definitely the future, rather than RAC, but I mean, they could be used in conjunction.
[01:10:42] Definitely, okay. Yeah, yeah, yeah. That's a hot
[01:10:44] swyx: take. Which part of the Long context is the future rather than RAG. Like, you would, they will coexist, but you are very positive on long context. I will put myself on the other, so your mirror image, which is like, long context is good for prototyping, but any production system will just move to RAG.
[01:11:01] Yi Tay: There are a lot of application use cases where you want a model to take the time and then come up with the right answer, right? Sure. Because RAG is like
[01:11:07] swyx: But you will use those sparingly because they're expensive calls.
[01:11:09] Yi Tay: Yeah, you, it depends on like the nature of the, the, the application, I think. Because if in rac, right, like you, there's a lot of issues like, okay, how you, like, you, the, the retrieval itself is the issue.
[01:11:18] Or like, you know, you, you, you might get fragmented if it's like, what if it's like a very complex story, right? That you like a storybook or like a complex like thing, right? And then, and then like we, like rec is very like, you kind of chunks, chunks and chunks, right? Yeah. The chunking is like, and you definitely have lots of information, right?
[01:11:35] So there I, there are a lot of application use cases where you just want. The model is like you were like, okay, like a hundred bucks, like take your time, take one whole day, come back to me with like an answer, right? Rather than like, I pay like, like one cent and then like get back a wrong answer. So I think that's like, that is actually very easy to show that RAC is better than long context because there are a lot of tasks that don't need this long context.
[01:11:57] You like, like fact retrieval, you just like RAC and then you do this thing, right? So like, long context may get a unfairly bad rap sometimes because like it's very easy to show like, RAC is like, 100 times cheaper, and it's very easy to show this, right? But then it's also, like, not so easy to emphasize the times where you actually really need, like, the long context to really make, like, very, very, very, very, very good, like, decisions.
[01:12:21] So, yeah, I mean, I think both have pros and cons depending on the use cases. Using them together is also interesting. hyperparameter that you have to wiggle around, right? Yeah.
[01:12:31] Long Context vs Finetuning
[01:12:31] swyx: There's another wiggle on the hyperparameter, or there's another fog on the hyperparameter, which is how much you fine tune. New knowledge into the model. Are you positive on that?
[01:12:39] Do you have any views? So, for example, instead of doing RAG on a corpus and then inserting it into context, you would just fine tune your model on the corpus, so it learns the new knowledge. In whatever capacity,
[01:12:52] Yi Tay: right? This is cumbersome, I guess. This is cumbersome, and you don't want, like,
[01:12:56]
[01:12:56] Yi Tay: You don't want so many of, like, the point of in context learning is so that you don't actually have to do it.
[01:13:00] I think this one is depending on, like, the business use case, right? If fine tuning is actually, like, the, you are very clear, like, you want this knowledge, and then you just fine tune once, and then you don't ever have to pay, like, context, like, in the context window. If there's a cost again, then maybe that makes sense.
[01:13:14] But if the domain keeps changing, then you might not like it.
[01:13:16] swyx: Yeah, obviously it doesn't make sense if the domain keeps changing. But I think for the model to maybe update fundamental assumptions, or you know, re weight associations between words, for let's say a legal context versus financial or medical context, like it might Work.
[01:13:29] This, this is the arguments that some, some people are talking about. So, you know, I see this as a trio, like it's long context, it's rag and it's fine tuning. Like people always have this, like whether either of them will kill, rag, basically , because rag is kind of the simplest approach.
[01:13:43] Yi Tay: Yeah, yeah. Okay. I, I mean I, I could see like, like if you wanna like a model for medical domain, legal domain, then fine tuning really works.
[01:13:49] It's always the move, like the, you know domain specialized model, universal model and, and you know, the kind of this. Tension between both of them. I think it definitely, like makes sense. It also makes sense, like, to, fine tuning can also be, like, an alternative to, to RAC, yeah.
[01:14:02] swyx: Yeah, well, there's some, there's some companies that are set up entirely just to do that for people.
[01:14:07] So, it's, it's interesting that, I mean, I, I, I kind of view RACA as, like, not working in that space, but you could potentially offer that if you wanted, wanted to.
[01:14:14] If emergence is real, when does Efficiency work?
[01:14:14] swyx: Okay, I was going to ask about efficiency and scaling. I'll just mention this briefly, and then, and then we can talk about MOEs, because I discovered that you, you, you wrote.
[01:14:23] You're a co author of the Sparse Upcycling paper, which is excellent. Oh, no, I was just advising on that. Oh, okay. Yeah, yeah, yeah. But you can talk about Sparse Upcycling, it's a topic that's hot. But more generally, efficiency, in my mind, when I go to ICI Clear, or I go to NeurIPS, I see efficiency paper, 90 percent of the chance, I'm just going to ignore it.
[01:14:39] Because I don't know if it's going to work. And I think this is related to your Some of your
[01:14:43] scaling work and your inductive Oh, okay,
[01:14:46] Yi Tay: scaling log Which is
[01:14:47] swyx: like, okay, there was this T. R. Texas, I don't know who this person is Yeah, he keeps talking about me. It's f*****g amazing Oh, okay. Yeah, he does have some obsessions, but like, he's good.
[01:14:56] I don't know who he is, but he's good. So he says, if 2024 papers are to be trusted, you don't need most attention, you don't need high precision, you don't need most KV cache, you don't need most feedforward network layers, you don't need a reward model, blah blah. Like, it's like, a lot of efficiency papers are just like, hey, on this small example We cut this thing out.
[01:15:14] Works fine, or works great, works better, whatever. And then it doesn't scale. Right? Like, or So it's a very interesting observation where like, most efficiency work is just busy work, or like, it's work at a small scale that doesn't, that just ignores the fact that like, this thing doesn't scale, because you haven't scaled it.
[01:15:30] It's just fine for a grad student, but as for someone who's trying to figure out what to pay attention to, it's very difficult. to figure out what is a worthwhile direction in efficiency.
[01:15:37] Yi Tay: Yeah, that's, that's, that's a good point. I think there's a couple, I agree with you fundamentally that like, it's actually quite easy to tell, like when you see a paper, okay, this one doesn't work, this one works, this one doesn't work.
[01:15:47] I guess the hippo account will just tell you that, sometimes it's just a diary about this thing doesn't work, this thing works, everything. Right, sometimes it's not like, you know, you can always find a dataset where your efficiency method gets neutral results, right? You can always find one, I have comparable complexity.
[01:16:04] And you know what's the most, the cutest thing ever? Every time some people propose like this, they run like some zero shot score on like some LME Valhannes or something like that. And at 1B scale, all the numbers are random, basically. Like all your boolkill, they're all like, Random chance performance, right?
[01:16:21] And they will be like, okay, I get like 50 versus 54, I'm better. But like, dude, that's all random chance, right? Like, you know, sometimes I see people that run experiments that like, And then it's like
[01:16:32] swyx: That's a good tell.
[01:16:33] Yi Tay: I think it's very, like, the sad truth is that like, it's very hard to tell until you scale up.
[01:16:39] And sometimes the benchmarks that we have don't even probe entirely about what, , I mean, especially all the works about, you know, the transformer alternatives, right? You can always find, like, this alternative that at 7B scale, at 1, 3B scale, you kind of like, okay, I met transformer this and this, this, this, right?
[01:16:55] But then what's the implications when you go to like 200B? What's the implications when you go to 100B? No one knows that, right? So that's, that's one thing, right? And I think developing your own intuition of like what works and what Doesn't work is, is important. For example, if somebody's like, Okay, to be honest, all researchers, like, sometimes are also, like, guilty of this sometimes.
[01:17:14] Because you cannot test on, like, everything. You cannot test on everything, right? So sometimes, you also just want to show your method works on this. But it depends on the objective. If the objective is to write a paper to ICML, sure, you can find two datasets your stuff works, right? But will you get adopted?
[01:17:29] I am not sure.
[01:17:30] swyx: Yeah, researcher metagame is one thing, but as a consumer of research, I, like, I'm also trying to figure out, like, what is, how do I know what is a, what is a useful direction, you know, that, that's the interesting thing.
[01:17:41] MoEs and Upcycling
[01:17:41] swyx: So, for example, MOEs seem to have worked out. Yeah, yeah. I, I, I, I'll go so far as to say it's the first form of sparsity that worked, like, Okay.
[01:17:50] 'cause there's, there's so much varsity research, like we can, chop all these parameters and look, we still, still perform the same, but then it, it never actually works. But, but OE is really, oh, you mean like
[01:17:59] Yi Tay: the pruning line of work?
[01:18:00] swyx: Pruning? Pruning line of work. Okay. Sorry, I, I should have used that word.
[01:18:03] So like, you know, I don't know if you have any commentary on like ra, deep seek Snowflake Quinn all these proliferation of Moe e models that seem to all be spars op cycle because, you know, you, you were advisor on, on the spars op cycling paper.
[01:18:16] Yi Tay: So the spas abstract Bay was mostly vision focused with a little bit of T five.
[01:18:21] Okay. Experiments. So it was, early stage of like abstract. But it was good that Google was really think about this like longer and, and normal so had on it,
[01:18:29] swyx: right?
[01:18:29] Yi Tay: Yeah.
[01:18:29] swyx: I think always the way to go. Is it like a hundred experts, a thousand experts , for some reason the, the community settled on eight.
[01:18:35] Yi Tay: Oh, you probably get more gains from, from more, more than eight, I think. But like, I think in general it's like. MOE's are just a trade off with like, prime and flop, right? And then you're able to like, kind of, make, like, you kind of make that. That, that in like that, that scaling log increase from, from that additional.
[01:18:55] So you, you can keep a low flop but kind of have more parameters. It's just changing the flop parameter ratio. Mm-Hmm. Keeping in mind there's a lot of inefficiency
[01:19:01] swyx: between the experts.
[01:19:03] Yi Tay: Yeah. Yeah. Yeah. I think as a architecture itself, the flop brand ratio makes it like worth it. Right. But I think the, the thing that's not very well understood is that, like, how does like MOE, like, like for me as a research question, is that like when you.
[01:19:15] Like, how does it, like, relate to capabilities and stuff like that, like, does this inductive bias actually, , for example, when you do, like, massive instruction tuning, I think there was this paper, like, Flan MOE or something, like, they showed that, like, , instruction tuning, I'm not, like, fully sure, I don't recall fully, but, like, when you do massive instruction tuning, like, MOE models are, like, they behave differently from a, from dense models and stuff like that.
[01:19:36] Like, I think, Okay, like, fundamentally, I just think that MOEs are just, like, the way to go in terms of, like, flop parameters. They show that they bring the benefit from the scaling curve. If you do it right, they bring the benefit from the scaling curve, right? And then, that's the performance per flop argument, activated params, whatever.
[01:19:52] That's, like, kind of, like, that's a way to slightly cheat the scaling law a little bit, right? By having more parameters, right? I think the more interesting thing is about, like, what trade offs do you make in terms of capabilities? Because of this new architecture. Mm. I think that's actually like the, the question that I, I think I, I guess all the frontier labs, they already know this, but nobody is writing papers anymore about this.
[01:20:12] So like, you just have to live with, with what? Like, but I think OI think I'm, I'm, I'm, I'm bullish about Moes. Yeah.
[01:20:18] swyx: Yeah. I had to, I mainly exercise for myself on reading research directions and what their asto asymptotic value is. Mm-Hmm. and I put OS pretty low because I think you have a good base model and then you upcycle it and it bumps you a little bit.
[01:20:34] And I think that's it. But like, I'm always seeking to invalidate my hypothesis, right? Oh,
[01:20:39] Yi Tay: but like, from scratch, MOE is also promising, right?
[01:20:42] swyx: From scratch, MOE is promising I
[01:20:43] Yi Tay: think in the IU case, you'll do MOE from scratch,
[01:20:46] swyx: I think. Okay.
[01:20:47] The Efficiency Misnomer - Efficiency != Speed
[01:20:47] swyx: The last part that makes me uncomfortable about MOE debate is actually it's related to another paper that you wrote about the efficiency misnomer, in the sense that, like, now people are trying to make the debate all about the active parameters rather than total parameters.
[01:20:58] But it seems like, it sounds like that's something that you're comfortable with, like, flops at inference is, is a relevant metric. And it's, it's not that Well, thanks for, like, actually reading all the, like, reading the papers. You're trying, man. It's very hard to copy. You have a lot of papers.
[01:21:12] Yi Tay: I'm actually very impressed that you're bringing up these papers.
[01:21:15] Yeah, I'm using attention.
[01:21:16] swyx: Yeah, thanks, thanks. And also, I mean, I'm interested in efficiency that works. It's just very hard to find efficiency that works. And so, like, anything that helps me have high signal on efficiency is helpful.
[01:21:28] Yi Tay: So I think for the inefficiency misnomer, by the way, I love the paper, by the way, it's quite a fun time working on it.
[01:21:33] I think inefficiency misnomer was like, we found that a lot of people, like, they use params, like, especially, like, like, to the kind of, like, right, and then MOEs was not very hot, like, in the community at that time, right, but MOEs were, like, a thing long ago. So I think using active params, I'm comfortable with using active params to kind of approximate like cost of the model, but like in the efficiency misnomer paper, we actually made it quite clear that you should always look holistically about like, because you have serving, like additional serving costs, like fitting in the GPUs, like fitting on single node, and something like that.
[01:22:04] The
[01:22:04] swyx: interesting one was speed. Nobody really talks about speed, but your paper actually talks about speed.
[01:22:08] Yi Tay: I have something to say about speed, throughput, right? There are so many methods, right, that are proposed about efficiency, right? They are like, theoretically, like faster because of some complexity or like something like that.
[01:22:20] But because there's no way to work around the implementation, or like your implementation becomes so hard, it becomes like 10x slower. There's so many papers around. It's not hardware aware. It could be hardware, it could be software. Just the way that, like, you have a convenient way to, like, in its mathematical form, it's actually, like, okay, linear complexity, like, whatever, and it's actually theoretically faster.
[01:22:42] But, like, just because you have to, like, do a scan or something like that, and then it becomes, like, actually, like, ten times slower in practice, right? There are a lot of things, like, Not a lot, but like, there are some things that are like, some methods that are like, like this, where you don't take into account throughput, right, which is also the problem of like, sometimes, like, the incentives of like, like people working in efficiency, you can easily just like, sell a paper as like, more efficient, People will not suspect that, because the reason why we wrote the paper is that so many people were confused about, like, efficiency itself, right?
[01:23:12] Yes. And then they will be like, okay, like a lot of these unsuspecting reviewers, especially, like, even academics, or, they, they, they don't have, like, that, that real, real, real feeling. They were less like, okay, less parameters, more efficient, right? So you could have a method that's, like, less parameters, but, like, three times slower, because, you know, a lot of times when you add things to the model, It becomes slow.
[01:23:31] Every time you add complexity, especially if it's like something that's not hardware optimized, no kernels, or like something that is like bad for TPUs or whatever, your model just becomes like slow. Oh, that's a
[01:23:40] swyx: temporary issue.
[01:23:41] Yi Tay: People can fix it, but some things are not like so, some things may not be like so easily fixed, or like it just adds a lot of like, like SWE costs to to optimize it, right, and everything, right.
[01:23:51] But then it's always marketed as like, because I save params, so I save. Right, and then also like, the params will add a different place of the model. Like, for example, like, If let's say you, even in the case where you param match models, right? If I take out like, some brands from like, FFN, right? And I put it to like, embedding layer.
[01:24:11] Embedding layer is like a, it's just, it's a cheap operation for embedding layer, right? But my model becomes like, lopsided, right? I could say I brand match this. But it's not Flo match, it's not throughput match, right?
[01:24:21] na: Yeah.
[01:24:21] Yi Tay: Because the, it's unbalanced. It is unbalanced or the, the side, right? So there's also of this style of tricky things that like when mixed comm model comparisons like very, very, very, very, very difficult.
[01:24:31] And because you cannot even put like flop throughput and speed flop. Params and speed, like actual speed, right, in the same plot, right, and then there's always like one money shot in a, like, there's always like a Pareto kind of compute, like, whatever, plot, right, like for marketing in papers or something like that, it's always very easy to, like, I mean, not intentionally, but like, to subconsciously, like, show one story when it's actually, like, there's, like, all these other things to consider.
[01:24:58] Yeah, yeah, it's
[01:24:58] swyx: a selection bias, self bias, whatever. Very cool. Okay, well that was mostly of most of the technical side.
[01:25:05] Open Source vs Closed Models
[01:25:05] swyx: We have one commentary that will happen today on the future of open source models. Basically Founders Fund said, like, the future is closed source. You were agreeing with it. And a lot of the open source fanatics, you know, are up in arms over this.
[01:25:19] I don't know if you get a comment about just Oh,
[01:25:20] Yi Tay: okay. Okay.
[01:25:21] Open
[01:25:21] swyx: versus
[01:25:21] Yi Tay: close
[01:25:22] swyx: and close, whatever. So, so,
[01:25:23] Yi Tay: I mean, I, I don't really like when, I mean, like if you're, if you're referring to the tweet that I wrote, but like, I wrote something about, about it, but
[01:25:30] swyx: this is huge. Like, so many people are commenting about it 'cause they, they have personally, physically offended their open source cannot catch up.
[01:25:35] Yi Tay: Okay. No, no. Wait. Okay. So I, I, I want to say it's like I'm not, like I contributed to open source in the past, so I'm not like. against like open source per se. But the interesting thing that I want to talk about here is that like, there's a difference between like, I draw a line with like, open source, as in like, okay, Lala, Luma, Lama tree is like, it's like, metal has a that is like, okay, hypothetically, very similar to to like Gemini or something, but they just didn't decide to release the weights.
[01:26:01] Yeah, it's open weights. Right, it's open weights, everything, right. I think when most people try to say that like, open source is catching up and everything They kind of mean like, this grassroots, like
[01:26:11] swyx: Yeah, this distillation No,
[01:26:12] Yi Tay: this bottom up people that are like these indie developers that are like, coming together to like, like, fight, like it's romanticized and it's dramatized to some extent just to fight against like this, right?
[01:26:23] Definitely, yes. And To be very fair. I think that there isn't really much, like, like so far, if you just look at the, the fractions of people, the big labs are just pushing and pushing and pushing. The academics like Stanford and stuff, they came out with DPO, they came out with things like that. They, they make some like, but they, they're kind of in, in between the line of like open source community and, and then there's also like the developers that are like.
[01:26:45] Fine tuning on GPT 4 distilled models and everything, right? I don't, I think the open source, the underlying, like, thing about, like, collectively improving something, I'm not, like, criticizing it for the sake of criticizing it, but, like, I'm just saying that, like, in order to make progress, right, I think the incentives of Open source, like, what I observe is that, like, people like to do things like, they like to take somebody else's model, they rename it, they make a quick win from there, and then, like, you notice that, like, when people realize that, like, this turning on the GPT 4 tab, and running some DPO, it's not going to give them the reward signal that they want anymore, right?
[01:27:22] Then all these variants gone, right? You know, there was this era where, There's, wow, there's so many of these, like, I can't even, I lost track of this, like, all these model variants. But now they're all gone, because people realize that, that you cannot climb LMSYS, because you need something more than just something that is lightweight, right?
[01:27:37] So I think that was just my overall, like, Honestly, the Hugging Face leaderboard contributed to most of that. It's not LMSYS. No, no, I think LLC is probably they realized that they could not. Yeah, right. The open LLM leaderboard is probably like a big problem, to be honest.
[01:27:52] swyx: We're talking to Clementine in one of our future episodes.
[01:27:55] Okay,
[01:27:55] Yi Tay: okay, okay.
[01:27:56] swyx: They dedicate a lot of, I mean, there's so much attention to them, it's a tough problem. But they're providing a public service, for sure.
[01:28:03] Yi Tay: Yeah, I mean, good intentions are always good. I mean, good intentions are always good. I'm interested in, like,
[01:28:08] Personal Productivity
[01:28:08] swyx: Just like, just career wise what is your productivity practice?
[01:28:12] Or, and so I'll split it into three, three things. Keeping up, like reading papers and whatever, the outside world. And then two, like how you organize your own work. And then three, like work and life. Just use any, any, take that in any order that you wish.
[01:28:27] Yi Tay: I don't have much of a life, actually. But I am trying more to have more.
[01:28:31] I mean, you're a father now. I have a baby now, so like, I'm trying more to have more life and and everything like this. I think the productivity hack that I have is this, like, I didn't have like a boundary between my life and my work, like, for a long time. So I think I just cared a lot about working most of the time.
[01:28:47] Actually, for the last like, during my PhD, during my, at Google and everything, I'll be just like working all the time. It's not like the most healthy thing, like ever, but I think that that was actually like one of the biggest, like, productivity, like and I spent, like, I like to spend a lot of time, like, writing code and I just enjoy.
[01:29:03] Run experiments, writing code, and stuff like that, right? So you kind of, if you enjoy something, it's not work, right? So like, it's very strange. It's like, it's like, I would get distracted by, sometimes I have to watch some Netflix series, because like my wife asked me to, like, watch it, like, or somebody tells me that I've, I've, I'm, I'm back on time on some, some shows, right?
[01:29:19] But then I get distracted by, My experiment is running and I just end up like, like writing code instead of like, so things like this. It's not the most healthy thing, but I think that's one. I'm
[01:29:29] swyx: looking for like a practice where like, okay so Andre recently had a thing where like before, when he wakes up, he doesn't look at social media.
[01:29:35] He only goes to , street to work. Damn, I check Twitter the moment I wake up. I know, see, it's just something I do as well. But I'm like, damn, that's a smart rule. And like, I'm looking for rules like that. No, he doesn't check social media because his phone is exploding all the time. All the time, yeah.
[01:29:48] I don't have so
[01:29:48] Yi Tay: many likes and followers, so it's
[01:29:49] swyx: fine. Yeah, you get there. Rules like that, mantras that you've developed for yourself where you're like, okay, I must do this. So for example, recently for me, I've been trying to run my life on calendar for a long time, and I found that the only way that I work is I write things down on pen and paper, and I cross them off individually.
[01:30:06] And the physical action really, really helps me, you know, get things sorted. And that's work wise. Reading wise, I don't know if you know, but I've been running this AI newsletter. Like all those summarizes, all Twitter, Reddit, discord and all that. So that helps me keep up, because I have like a socially graded, and I personally vetted the entire pipeline from beginning to end, so like, this is my input algorithm, I know how to keep up with news because I now have a Information condenser.
[01:30:34] So like, I'm trying to figure out what is your algorithm or what is your rules for keeping up. I've
[01:30:38] Yi Tay: got something for keeping up. So I used to check archive like every morning when the gate opens, I just check archive. I will wake up 9. 30am Singapore time, the archive gate opens, right? And then I'll be very sad if there's no papers to read.
[01:30:52] But you usually just pick one paper or two papers that you find interesting. I don't read them, I just like skim like the thing, right? So I used to do that. I don't do that anymore. I mean, ever since I have been in the startup, I You have a real job now. I read less papers, right? But I used to cam at the door of archives quite frequently just to see
[01:31:09] swyx: That's not a good use of time.
[01:31:11] I'll come out and say it. It's not a good use
[01:31:13] Yi Tay: of time. It's a newness bias. Sorry, go ahead. It's just because I ran out of things to It's just that the new stuff comes out, right? Yeah. The new stuff comes out, so that's how I keep up to date. So in the space of three years, you read every No, no, I didn't read everything.
[01:31:27] It's just that, it's just that. But these days I realize I don't have to do that anymore. Just because if the paper is important enough, Twitter will show it to me. So I, I, there isn't really, like, And one thing I do is that I actually don't read papers like that, that much anymore. I just like skim them, like, almost, right.
[01:31:42] The so that's for keeping up, like, with papers, research, everything. And the other thing more of like, just like a productivity point of view is that I used to always keep, like, the, like, you know, the text. Like, I usually start writing. The thing while working on that thing itself. Like, so even, like, let's say, like, like, if you want to launch something, like, then the end goal is like a blog post or shipping something, everything, right?
[01:32:06] I like, I'm not, not, not really a launcher or like, like, just papers. I always like to look at it from, like, what's the, the story and the end. And then I just like figure out what I need to do to get to, to, to kind of, right. So I think as a researcher, like, this is something like, I would have, like, Like so many drafts of like, like when I'm start, I start the project.
[01:32:24] I don't know the experiment instead everything. Right. But I like to imagine like what the title would be. Yeah. Right. And then I always check, like, I always like, so I, I mean my friends at Google would know that I always have like, like a like the overly draft of like so many. And then I would just spend time looking at it, like looking the title, is it better to second?
[01:32:39] So I care about, I used to care about a lot of things, but this actually helped my product. 'cause every time I look at it, I'm like, okay, this is the final product. I'm like booking towards it. Right. 'cause I think a lot of researchers, they, they tend to like. They swoo around with their experiments and they never like ship the final story.
[01:32:52] It's like the shipping, like, like I mean, it started out with ship products, but like, as a researcher, your product
[01:32:58] swyx: management, yeah, you're shipping
[01:32:59] Yi Tay: the thing. So I like to, I like to hang around a lot in my, in my drafts and, I get motivated from that. And that's like one productivity thing that I did as a researcher.
[01:33:08] Yeah. So I think that that's other than that, I don't really have any things that I do that. Probably different from others. Yeah, probably you don't know it.
[01:33:15] swyx: This is unconscious competence versus
[01:33:19] Singapore vs US Academic Scene
[01:33:19] swyx: what's it like just NTU PhD, you know, just the story of like, how is it coming out from NTU, which is Which is like a good school, but like not, you know, not typical target school for like a big lab.
[01:33:31] Yi Tay: I did my PhD unknowingly. Like I didn't have very, like when I was, I was a very regular undergrad. I had decent grades, but not the best grades. I was not like super smart in school or something like that. I, I was I wanted to do a PhD just because I was like curious and, and I, I mean, like, and then I wanted to stay in Singapore at that time, so I just like naturally just did a PhD there.
[01:33:52] I didn't even know Vet, my advisor. I didn't even think too much. I just like fell into the PhD program. And then that was when I realized that, oh, actually I can do research. Like, I'm like pretty decent at research. Like, I just fell into a PhD like, like unknowingly. And I definitely like, NTU leaves a lot to be desired.
[01:34:08] Actually, to be honest, I think that I mean, Singapore leaves a lot to be desired in general. Like the research community here is like, like probably not great. So how, how did you like
[01:34:16] swyx: break out? , if I was you, I would have, I would have no idea how to break onto the international scene, and
[01:34:21] Yi Tay: I think, I think it was, okay, to be honest, like, in retrospect, it's a bit of, like, a bit of a miracle, or, like, I mean, it's not easy to, I think, I could not, if I had, like, a product, like, someone to mentor, like, I could not, like, Tell somebody how to replicate the same thing that I did.
[01:34:36] It's much easier now, maybe, compared to in the past, but I've been mostly self supervised during my PhD. Like, my advisor was basically like, like Grammarly. Like a free plan of Grammarly. He won't watch this, so it's fine, but like, there's a lot of things that, that, that, it was like this strange arc of my life where I was figuring out research by myself and, and everything.
[01:34:56] And, and okay, maybe going back to the, the change of opinion is that like the biggest culture shock I had, like, when I was moving from a Singapore PhD to Google, I think my research, like, If you went straight to Mountain View. Yeah, I went to Mountain View. I started at Mountain View. Like my research taste and everything, like, like I was, it was so different.
[01:35:13] Like the research culture is so different in, in US and in Asia. I had to grow so much, like doing my time at Google to like actually evolve. And then whenever I come back, right, I still have friends in like faculty in here and everything. I don't think that I'm a snob or they think that I'm like, Being like a very nasty person.
[01:35:31] Because I think to be honest, the research here is like in Singapore is just basically like, they just care about publishing papers and stuff like that. And then it's not impact driven. I think at US it's mostly focused on impact driven and the thing needs to make real impact, right?
[01:35:46] swyx: To be fair, you're also working at an industrial lab versus an academic circle, right?
[01:35:51] Like, you're comparing apples and oranges here a little bit.
[01:35:54] Yi Tay: I mean, at the end of the day, I think research is still Like fundamentally like, we, as an industry, RIS, you still write papers, your goal is to advance science and everything. To be honest, it's, it's all the, you know, the incentives rewards system is, like, different, and, and maybe, like, slightly different than everything, but, like, at the end of the day, I still feel that researchers are researchers, scientists are scientists, no matter, like, really, like, where you are.
[01:36:16] I, I will get so much dissonance when I come back and I talk to people. Like, I would feel like, oh, why do you think like this? But then I used to think like this. So, like, the environment shapes, like, like, a way a researcher thinks. The taste is very important. Sometimes I try to communicate this to people, and then maybe I come across as a snob.
[01:36:35] To, to, to, like, the local community here, right? But, like, It's, it's just that there's like, maybe there's so much dense information that I want to bring back, but like, there's no like receptive, fast way to like, like transfer, like all the, the like, like transfer all the, the things that I've learned. Yeah.
[01:36:50] Also a big culture shock. 'cause I was in brain in the Singapore office for a while and I reporting to You were the only
[01:36:55] swyx: brain
[01:36:55] Yi Tay: person Yeah. Yeah. Brain in Singapore. And then I had, like, I took on an intern from actually. And the, the research like vibes and the thing was so much of a conflict for me.
[01:37:07] That it was almost like my body was rejecting it, you know? Mm-Hmm. . But this, this person, so like, grew, grew and became, I'm happy with how this person grew with, from, from my mentorship. So he's now in a way better situation. But I would say that like a lot of people in the, in, in universities here are like, not like a bit like, like they, ignorance is blis, right?
[01:37:26] Maybe sometimes . So, well, no.
[01:37:28] swyx: It's exposure. I didn't know any better myself until I went to the U. S. for college and then, yeah, my world was expanded and it's a little bit of a Pandora's box because once you've tasted that, you're never happy. Yeah, yeah, yeah. You know?
[01:37:42] Building Silicon Valley outside Silicon Valley
[01:37:42] swyx: So, okay, last question would be, just a sort of Singapore question.
[01:37:46] So, I'd like to know, Be visible, visibly non American, covering the AI scene, because it's very US centric. Every non American I talk to always wants to be like, How can we build Silicon Valley in my city, you know? My country, my city, whatever, that is not Silicon Valley. I feel like you have Basically, just kind of like me, you kind of operate in the US circles, but you just don't live there.
[01:38:08] Do you have any advice for like, if Singapore, okay, so I'm wearing a red shirt today. This is the official Singapore government sort of community group that is trying to guide Singapore AI policy. If we want a hundred more ITAs to come out, what should governments be doing? What should communities, ecosystems should be doing?
[01:38:25] Yi Tay: So I actually think that like, Sometimes, like, not doing too much is maybe less is more, maybe? I don't think there's actually much the government can do to influence. Like this kind of thing is like a natural, like an organic natural thing, right? The worst thing to do is probably like to create, like, like create a lot of artificial things that like Exchange programs?
[01:38:47] Okay. I mean, Singapore used to have a lot of exchange programs. Like they send people to, to, I mean, just talking about AI specifically, right? I think that, for example, like sometimes like trying to do too much or like moving in the right, wrong direction is just better than not moving at all. Especially if you, if you accelerate in the wrong direction, you actually get into a worse situation.
[01:39:02] Sure. So I think it's very dangerous to move in a bad direction. I think respect your talent more. Maybe the government should just respect their talent more. And I don't know whether this is too much of a No, no, no, no. But maybe not moving in a wrong direction is, to me, is a Already a very good thing.
[01:39:22] swyx: Funding, for startups, incubation, holding academic conferences, I think iClear next year is going to be in Singapore, so people come here and get exposed to it.
[01:39:30] But like, I don't know, it's just very interesting. Like, everyone wants to build up AI expertise within their own country, and like, there's a massive brain drain to the US. I'm part of that. I live there. I feel guilty. I don't see any other way around it. It's such a huge problem. I also do think that there is, like, cultural hegemony, let's call it, like, US values basically being asserted on the whole world, right?
[01:39:53] Because we decide our LHF on these models and now you shall use all our models. And it's just troubling for, like, national sovereignty should be AI sovereignty and I don't know how to achieve it for people. It's very scary.
[01:40:06] Yi Tay: Okay, that's a lot to unpack.
[01:40:08] swyx: Yeah, this is not technical, but I was just saying, you know, curious.
[01:40:11] We can make this the ending conversation, which is, I think you're an inspiration to a lot of other people who want to follow your career path, and, you know, I'm really glad that we got the chance to, like, walk through your career a bit. Yeah, I'm sure this is just the start, so.
[01:40:23] Hopefully there's more to come and I want to inspire more of you. Yeah. Yeah. Sounds, sounds good. So I'm just glad that you shared it with us today.
[01:40:29] Tech in Asia Meetup
[01:40:29] AI Charlie: As a special coda to this conversation, we were invited to join the Technasia meetup featuring Yi by managing editor Terence Li. Terence asked a similar question on how other countries can create conditions for top AI labs to spring up outside of Silicon Valley.
[01:40:46] Yi Tay: So, like, where do you see Singapore playing a role in AI? So, like, how, how, how, how would you Oh, okay, right. I got a practical one. Okay. I got a practical one that is actually actionable. I feel like one thing that people don't get, like, like, the advice, that practical advice, like, that is that, like, the era of, like, people who talk versus people who do, like, the people who talk is, like, gone, right?
[01:41:08] So like it's no, it's no longer about like, ah, I have a team, I have like 10 interns from, from Southeast Asia or like the region and then they're going to do this, do this, do this, do this for me, right? So I think one thing that senior people in any government, right, may not get, right, is that the world has shifted into this paradigm where senior ICs, ICs as individual contributors, right, are actually making the most impact in AI, right?
[01:41:37] So. In GDM and in OpenAI, I mean, in Frontier Labs, they're all very driven by individual contributors and not actually this is not even related, this is like, like, I'm talking about, like, This is advice I give, but it's actually general, like, it's a very general thing, so multi purpose, basically. It's not AI specific?
[01:41:54] No, it's also, it's AI, it's very AI specific, because The, the, the level, the difficulty of making impact and making breakthrough has started to become Like, it's no longer about, like, it's not like software engineering where, where, where it's, it's like, you know, I think AI is a little bit, like, harder, like and then, like it's mostly about, like, getting very senior people who are hands on and have a lot of, like, experience rather than, like, management style people that, like, try to, like, think they know what to do.
[01:42:26] They're doing but they actually don't. So I think, I, I mean, I, I'm not going to, like, say, like, names, obviously, right? But, like, I, I mean, I, I meet a lot of, like, people like this like, in general. I mean, not only in Singapore, but, like, right? But AI has shifted quite a lot into this IC driven paradigm where the people making impact are the people who are, like, on the ground fighting the war, right?
[01:42:51] So it's no longer about, like, I have 10 interns, 20 interns, 100 interns, you do this, you do this, you do this, I just take meetings, right? No, right? The senior person writes code. Everybody writes code. Nobody should not write code, right? And then everybody, so I think this is, okay, this is a bit extreme, but, but, but, this is a bit on the extreme side, but I think from people, like, I just the advice is just, like, maybe, like, just take 20 percent of what I say.
[01:43:18] And incorporating, right, right, so instead of, like, you know, like, if you, if you, if you, for example, hypothetical, hypothetical situation, right, say you want, you want to organize, like, an AI conference in Singapore, right, and then you want to make it, like, like, like a, you want to show Singapore as, like, the AI hub in the world, right, maybe you don't invite, like, policy people and, like, you don't invite, like, policy people to come and talk about, ah, AI safety, AI safety, AI safety, right, You invite people who, like, actually know their stuff, right?
[01:43:46] And then, if you organize a conference and then, like, hundred people, like, go there and then they feel very productive and everything, but, like, the problem is that, like, Singapore doesn't have, like, like, people who really can do it, you know? Right? So, I mean, I've, through the grapevines, I mean, I hear about, you know, people, like, fighting for territory here and there.
[01:44:09] I mean, this is what I hear, right? I don't want to hear this, but I hear this somehow, right? And then sometimes I just ask them, like, who's actually going to do it? Right, who's going to do it, right? The model is not going to train itself, right, unless we have AGI, right? So, yeah, I mean understand that, like, times have changed.
[01:44:27] It's no longer about, like, it's no longer about, like, you know, like, Oh, I'm very senior, very senior, very senior. Okay, okay, okay, can you code, right? That's the question, right? I think that's, that's, like, the
[01:44:39] Well said. Spicy or not spicy? Spicy already. Okay, okay. We are like, Cocoa is in Baya, raise the cocoa age in Baya already to the maximum. Yeah, almost there. Okay questions, anyone?
[01:44:50] AI Charlie: Indeed, questions are very welcome. Head over to the latent space substack to leave a question, or tweet at @YiTayML directly with your feedback.
It’s return guest season here at Latent Space! We last talked to Kanjun in October and Jonathan in May (and December post Databricks acquisition):
Imbue and Databricks are back for a rare treat: a double-header interview talking about DBRX from Databricks and Imbue 70B, a new internal LLM that “outperforms GPT-4o” zero-shot on a range of reasoning and coding-related benchmarks and datasets, while using 7x less data than Llama 3 70B.
While Imbue, being an agents company rather than a model provider, are not releasing their models today, they are releasing almost everything else:
* Cleaned-up and extended versions of 11 of the most popular NLP reasoning benchmarks
* An entirely new code-focused reasoning benchmark
* A fine-tuned 70B model, built with Meta Llama 3, to identify ambiguity
* A new dataset of 450,000 human judgments about ambiguity
* Infrastructure scripts for bringing a cluster from bare metal to robust, high performance training
* Our cost-aware hyperparameter optimizer, CARBS, which automatically and systematically fine-tunes all hyperparameters to derive optimum performance for models of any size
As well as EXTREMELY detailed posts on the infrastructure needs, hyperparameter search, and clean versions of the sorry state of industry standard benchmarks. This means for the FIRST TIME (perhaps since Meta’s OPT-175B in 2022?) you have this level of educational detail into the hardware and ML nitty gritty of training extremely large LLMs, and if you are in fact training LLMs of this scale you now have evals, optimizers, scripts, and human data/benchmarks you can use to move the industry forward together with Imbue.
We are busy running the sold-out AI Engineer World’s Fair today, and so are unable to do our usual quality writeup, however, please enjoy our show notes and the excellent conversation! Thanks also to Kanjun, Ashley, Tom and the rest of team Imbue for setting up this interview behind the scenes.
Video pod
Timestamps
* [00:00:00] Introduction and catch up with guests
* [00:01:55] Databricks' text to image model release
* [00:03:46] Details about the DBRX model
* [00:05:26] Imbue's infrastructure, evaluation, and hyperparameter optimizer releases
* [00:09:18] Challenges of training foundation models and getting infrastructure to work
* [00:12:03] Details of Imbue's cluster setup
* [00:18:53] Process of bringing machines online and common failures
* [00:22:52] Health checks and monitoring for the cluster
* [00:25:06] Typical timelines and team composition for setting up a cluster
* [00:27:24] Monitoring GPU utilization and performance
* [00:29:39] Open source tools and libraries used
* [00:32:33] Reproducibility and portability of cluster setup
* [00:35:57] Infrastructure changes needed for different model architectures
* [00:40:49] Imbue's focus on text-only models for coding and reasoning
* [00:42:26] CARBS hyperparameter tuner and cost-aware optimization
* [00:51:01] Emergence and CARBS
* [00:53:18] Evaluation datasets and reproducing them with high quality
* [00:58:40] Challenges of evaluating on more realistic tasks
* [01:06:01] Abstract reasoning benchmarks like ARC
* [01:10:13] Long context evaluation and needle-in-a-haystack tasks
* [01:13:50] Function calling and tool use evaluation
* [01:19:19] Imbue's future plans for coding and reasoning applications
* [01:20:14] Databricks' future plans for useful applications and upcoming blog posts
Transcript
SWYX [00:00:00]: Welcome to the Latent Space Podcast, another super special edition. Today, we have sort of like a two-header. John Frankel from Mosaic Databricks, or Databricks Mosaic, and Josh Albrecht from MBU. Welcome.
JOSH [00:00:12]: Hey, glad to be here.
SWYX [00:00:14]: Thank you for having us. Hey, so both of you are kind of past guests. Jonathan, you were actually one of the most popular episodes from last year talking about MPT7B. Remember the days when we trained large models and there was 7B?
JONATHAN [00:00:30]: Yeah, back when reproducing LLAMA1-7B was considered a huge accomplishment for the field. Those are the good old days. I miss that.
SWYX [00:00:38]: As the things have accelerated a lot. Actually, let's do a quick catch up and Josh, you can chime on in as well. So Databricks got acquired. I talked to you at New York.
JONATHAN [00:00:45]: Mosaic got acquired, although sometimes it feels like Mosaic acquired Databricks because, you know, we're having a lot of fun being here. But, you know, yeah.
SWYX [00:00:52]: Yeah. I mean, you are chief scientist now of Databricks.
JONATHAN [00:00:55]: Chief AI scientist. Careful with the title. As much as I would love to understand how Spark works, I'm going to have to defer that to much smarter people than me.
SWYX [00:01:03]: Got it. And I don't know about like what you would highlight so far as a post-acquisition, but the most recent news is that you guys released DBRX. Is that the thing that most people should be aware of?
JONATHAN [00:01:13]: Actually, that's no longer the most recent news. Honestly, the most recent news, we announced this, but it was at our Data and AI Summit last week. So it was announced among like 100,000 other things, is that we finally released our text to image model, which has been a year in the making through a collaboration directly with Shutterstock. There was a lot of work put into finding a dataset that we were comfortable with working on and trying to build a model that honestly, I felt like I could trust and that others might be able to trust to put out in the world. So that model was released last week. It's unfortunately just available via API due to the fact that the data is quite sensitive and quite valuable. It's Shutterstock's entire business in a lot of ways, but I'm still really excited that there's now a model that is trained on a dataset where the provenance of every single image is known, and it's a damn good model. So I'm really proud of the team on that.
SWYX [00:01:55]: Yeah, amazing. Josh, do you have any thoughts on image model questions?
JOSH [00:01:59]: That is not my area of expertise, but I was excited to see the release of it last week as well, and very happy that you guys did a nice job on the data side of everything there. So that was cool to see.
SWYX [00:02:09]: I think what's unusual is like, I think Shutterstock's doing multiple deals in multiple labs. So what is the Shutterstock model? Like, I guess, is this the house model for Shutterstock? Is this Databricks' version of the Shutterstock model? Like, what is this?
JONATHAN [00:02:22]: The way that I would think about it is that Shutterstock is doing an amazing business in AI across the board. Their dataset is kind of widely known to be the best stock photos dataset in the world, the most comprehensive, the biggest. When you think about like, what dataset am I going to train a multimodal model on? You call Shutterstock. And I, at least I've heard in the news, like OpenAI, Google, Meta, Apple have all called Shutterstock and made those deals. So a lot of models have had Shutterstock data incorporated into them. But this is the only model I know of so far where it was, you know, exclusively and specifically trained just on the vanilla Shutterstock data. There was nothing else mixed in. We didn't go and scrape the web and find other data or combined datasets or anything like that. And so this is, in some sense, the house blend. But the other piece is that it's just a dataset where the provenance of every image is known in public. Where did the data come from? It is the Shutterstock collection. That's it. You know, nothing less, nothing more. And certainly being at Databricks, if I've learned one thing, I've learned about enterprise customers and what they want out of AI. And one of the things they ask for most is just, what can you tell me about the data the model was trained on? And here, especially for text to image models, where images are just tricky subject matter, there's been a lot of kind of legal conversation about images, especially. It's nice to just have something where I can point to it and say, you know, if you want to know where the images came from, these are what they are and this is how they got there.
SWYX [00:03:36]: I will talk a little bit about Databricks because it's relevant to the rest of today's episode. So Databricks, sorry, I keep misspeaking. It's DBRX.
JONATHAN [00:03:46]: DBRX, actually, there's been a pronunciation update. It is now D-B-Rex. So we have decided to add a dinosaur mascot because what model doesn't like a mascot? So literally, I wish I could pull it up. There is a little plush dinosaur that we had made. It's like the world's cutest dinosaur, but it is the official mascot of D-B-Rex. And there's a little dinosaur logo that, you know, you'll probably see around a little bit more because DBRX is a mouthful, but D-B-Rex, like, you know, it's just kind of...
SWYX [00:04:13]: Rolls off the tongue. I love mascots. Like every company should have a mascot. And I think Hugging Face got it right. You need an emoji mascot because that's the minimal viable image.
JONATHAN [00:04:21]: I probably shouldn't talk at all about, you know, Velociraptor, but, you know, that's a, maybe that's something we can talk about later in the summer. I'll just leave it at that.
SWYX [00:04:28]: Okay. That's a hint to names. I feel like your names leak a lot of alpha. So just to quickly cover the headline details, DBRX, as Make Sure Experts model, that's fairly big, 132 billion total parameters, so 36 billion active on any input, pre-trained on 12 trillion tokens of text and code, and did really well on evals to the point where you had to dye your hair blue. That's my high level conclusion.
JONATHAN [00:04:53]: Never make a bet with your team two weeks out from model launch, even when, you know, human eval is looking quite bad. Because if you set some bar, even if it's arbitrary and you think there's no way in hell they're going to hit it, apparently money doesn't motivate people anymore. Humiliating their boss motivates people. So Josh, you should really take a hint from this. You know, you cannot pay someone enough money to make up for you dyeing your hair blue.
JOSH [00:05:15]: I'll keep that in mind for our next model.
SWYX [00:05:17]: It works. So speaking of Imbue's next model, perhaps Josh, you want to actually just say hi to the general sort of latent space audience and talk about what we're releasing today. Yeah.
JOSH [00:05:26]: I'm Josh, CTO of Imbue, and we're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models. So I think right now, training foundation models from scratch is like a very difficult, time-consuming, expensive, kind of risky endeavor, especially for smaller companies. And the things that we're releasing hopefully make that at least a little bit easier. So the things that we're releasing fall into kind of three different buckets. One is infrastructure and scripts for dealing with the kind of hardware and hardware failures and understanding how well is the actually lowest level of thing actually working so that you can actually do your training at all and at a reasonable speed without having to constantly restart, etc. So infrastructure and training scripts. A second set of things is around the evaluation. So after you've trained it, like how well is this actually working and how do you know how well it's working? We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like pool queue or ANLI, where we've gone through and kind of cleaned up the data as much as possible by looking at all the ones that models get wrong or that are flagged for ambiguity and also our own kind of private reproductions of those where we've done like a kind of clean room black box, like, okay, this is what the data set is supposed to be. Here are some examples. Let's make our own version of this to make sure that there is no data contamination, etc. To make sure that we're actually, you know, not testing on train. And then I think a final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used in the process of cleaning these evaluations and we also hope will be helpful for other people training kind of similar models. And then the third thing is CARBS, our hyperparameter, our cost-aware hyperparameter optimizer, which was especially helpful for being able to experiment at much smaller scales and then scale those experiments up to the much larger scale kind of on the first try without having to retry it. You don't want to be training, you know, 10, 20 different 70B models. You really want to get these larger models
SWYX [00:07:30]: right on the first try.
JOSH [00:07:30]: And so the ability to kind of tune things very precisely and learn scaling laws, not just for, you know, the like data and flops, but also for learning rate and all the other hyperparameters and see like how should you scale these things up was extremely valuable to us as we were training the larger models. Yeah, that's a lot of stuff.
SWYX [00:07:49]: Yeah, exactly. So there's a bunch of stuff
JOSH [00:07:50]: we'll have to go through all of it.
JONATHAN [00:07:52]: Yeah, I just want to throw in how excited I am about this. This is the stuff that nobody ever talks about. That is the difference between success and failure in this stuff. Like, can you get your cluster to run? Can you get software on your cluster? Can you figure out what broke? Because fault tolerance is still not really built into any of the fundamental primitives of training models. And so if something breaks, you have to go figure out what broke, your job stops, you have to restart your job. It is a nightmare just to get to the point where anything can train on the cluster. A basic MPI hello world that has the GPUs talk to each other is hard enough, let alone actually training a model, let alone getting good performance out of the GPUs, let alone actually getting a model that converges to anything interesting. There's so many levels of things you have to accomplish. This is the kind of stuff that matters. I think to a point that Josh made earlier, before we got on here, there are plenty of weights out there. Nobody's released this.
JOSH [00:08:46]: Yeah, that was part of the motivation actually is that there are lots of other things that are complimentary, but I have not seen nearly as much discussion about some of these other things that we think are pretty important. I mean, in some sense,
SWYX [00:08:56]: I'm very excited to have Jonathan on because this is a little bit, you're a bread and butter with Mosaic. And I think you've released some part with Composer. And I think it's just really interesting to see like a different take, basically a full stack take that's kind of open source today.
JONATHAN [00:09:18]: Yeah, it's really kind of, it's been an ordeal to figure this out. And every time something changes, whether it's a new GPU or even a new driver update, you get new creative errors and new things go wrong. And, you know, we've dealt with the weirdest things from, you know, our InfiniBand cables getting stolen from the data center twice, like in boxes before they arrived at the data center. Like, you know, Porch Pirate basically had stolen our InfiniBand cables back when those were hard to come by. To like, you know, weird recalls of switches to like the strangest stuff has happened. I have my favorite GPU failures I've seen, like ones where the GPU doesn't fail, it has a correctable memory issue and the memory correction causes the GPU to become a straggler and hold up the whole job. Like weird stuff happens and figuring out how to not just identify all of that, but then eventually productize it, is in some sense, the entire story of Mosaic and now Databricks in terms of our ML offering. Really, the thing we offer is we have gone through this suffering and figured out how to even productize that. It has been a pain in the butt.
SWYX [00:10:20]: Yeah, it's a lot of work.
JOSH [00:10:20]: I think my favorite failure was GPU is just giving wrong math. Like if they give errors, great, because you can see the errors, but if they just give you the wrong math back, not so fun.
SWYX [00:10:30]: When did they give you wrong math?
JOSH [00:10:32]: Like literally you could just, you know, add two things. For example, the numbers come back. They're not the numbers that they're supposed to be.
JONATHAN [00:10:40]: I think it's important to say at this stage, just because like it, I think it goes without saying for Josh and I, but it's worth saying here, this isn't to say that like anything is wrong with us. It's not like NVIDIA did a bad job or, you know, Mellanox did a bad job or the like the server builder, the data center operator, the cloud provider, like the million other parties that are involved in building this. We are running these insane chips that are huge and complicated and built on tiny transistors at insane frequencies with insane heat in data centers that for the most part, were not built remotely for this kind of power or heat and have been retrofitted for this. Like failures happen on a good day with normal CPUs. And this is not a good day and not a normal CPU for the most part. It's fun to joke about all the weird things we see. This is not to say anybody's done anything wrong. This is just kind of part and parcel of working on a massive cluster running at multiple megawatts of power at a time.
SWYX [00:11:32]: It's crazy. Yeah.
JONATHAN [00:11:33]: So optical cables, like all sorts, like everything.
SWYX [00:11:37]: I'll take the opportunity to start going to the sort of infra piece. There's just like a description of the infra just to give people a sense of what we talk about when we talk about massive clusters. So I'm just going to read off the blog post here. This post is about one cluster that has 4,092 H100 GPUs spread across 511 computers. They use unified fabric manager nodes, which manage the infinite band network. And you talk a little bit about your networking. Is there anything unusual about this setup that you'll call out to people?
JOSH [00:12:03]: Yeah, actually this particular cluster is a little bit non-standard. The normal, like vanilla setup for these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 1024 GPUs instead of 4,000. Here we have a larger cluster. As you start to get into the larger clusters, the networking becomes a little bit more custom. It's a little bit more, it's a little bit trickier. It's a little bit more difficult to get these things to all be able to talk to each other at the same speed. And so this has, in this particular case, this is a three tier network architecture instead of two tiers, kind of the normal one. So most of the clusters are a little bit smaller. As you get to even larger scales, then this becomes even much more complicated,
SWYX [00:12:43]: much more expensive.
JOSH [00:12:43]: So we chose this particular scale, kind of knowing our own workloads and kind of what we wanted to do. This was kind of the right size for us. But yeah, I think it's not exactly vanilla already. It's already getting into kind of the custom territory.
SWYX [00:12:54]: So my understanding is that there, and is there any part of this that comes with the Voltage Park deal that you guys had? Is that part of the hardware that you got from the deal with them?
JOSH [00:13:04]: Yeah, so we worked really closely with Voltage Park to set up all their clusters and infrastructure and everything and kind of decide even like what to order, how should the networking work? Like we were very involved in kind of the construction and bring up of this. And that's what this post is about, is about that process of like bringing up all these, there's like different clusters in different places of different scales. So in this particular post, we're talking about this one 4096 GPU, but there are other clusters that they have as well. And we were very closely involved with figuring out the exact architecture and kind of the trade-offs that go along with picking, you know, those exact components. You really don't want to like place the wrong order because it takes months to get it and it's very expensive. So yeah, we were happy to help out with that.
JONATHAN [00:13:43]: And then your bit of good cables get stolen.
SWYX [00:13:44]: Yeah, yeah, exactly.
JOSH [00:13:47]: We wanted to make sure that we ended up with compute that would work for us and that would also work for their other customers. And so we kind of helped design something so that we would get exactly what we were looking for. We knew that these kinds of details would be super important and that getting down to the level of the hardware and like having these good scripts and everything was going to be a core part of like actually getting this to work. I'm very glad that we did that. I don't think that most companies kind of take that full stack approach, but for us, it certainly paid off.
SWYX [00:14:12]: Yeah, it's basically sort of built to spec. It's interesting that relationship because you usually, for the rest of us who don't operate at your scale, we take whatever we can get from cloud providers, but you are basically co-designing from the single machine up. And you described that a little bit. Do you want to take us through the process that you described here?
JOSH [00:14:27]: Yeah, so for the actual, like the blog post and kind of bringing these machines online.
SWYX [00:14:32]: Yeah.
JOSH [00:14:32]: So yeah, I think the process, as we have it broken down in the blog post, there's kind of a few different layers. First is like getting the individual machines to work at all and then getting the machines to actually be able to talk to each other. So getting the InfiniBand networking to work and then getting to a point where, you know, not just the machines are working and they can talk to each other, but everything is actually working correctly. There's a big gap between like it's working at all to it's working perfectly correctly. And then after you have all this stuff working perfectly correctly, nice and healthy, then now you get into kind of the software data, like training issues. And then after that, you're still not done. Like now, even once you're training at full speed, things are going to fail over time. Things are going to change. There's going to be new, you know, firmware updates. Like how do you kind of deal with this change and flux over time without going crazy
SWYX [00:15:16]: and pulling your hair out,
JOSH [00:15:16]: trying to like reproduce things or understand why there were regressions. And so there's a lot of work to kind of automate the infrastructure tooling as well. And kind of the first step, like bringing these things online in the first place, you know, you have hundreds of machines at this point. So you don't necessarily want to be like walking around with like a CD-ROM or a USB drive, like plugging it in with your keyboard, like hitting next, next, next on the OS install. That's not how this works. You do that for one machine. And then you use, we use this thing called Metal as a Service to bring up all the other machines. So it's a kind of server that can kind of install the operating system on these other machines. So most like when you're talking about these machines, like each machine is, you know, on the order of hundreds of thousands of dollars. So they usually come with a kind of out-of-band management interface as well. So they don't, they have their InfiniBand networking. They have their normal 100 gigabit per second Ethernet networking. These are like dual, redundant, et cetera. And then you also have this extra out-of-band management network. So you can log in and you can see like the boot screen or you can see the blue screen of death. You can like get in there and actually see what was wrong, which is pretty fun. And it makes it like possible to automate a lot of this work. So the beginning of that, and the blog post goes into much more detail about like exactly how we set these up and kind of the other errors that we ran into. When you're bringing these online, you'll definitely have failures. Even if they all worked in the factory, they get shipped, some parts come loose, something fails, something goes wrong. So when you're bringing them online, there'll be some that don't quite work for all sorts of reasons. As you start to be working with machines at this scale, like if something happens one in a thousand times, you're like pretty likely to see it. And so you can get pretty rare, weird things, especially since we had fairly early builds and fairly early versions of this hardware. Like these are some of the like first machines that were ever produced, some of the first GPUs. So you've got some extra special things there. We definitely worked with Dell, for example, on making fixes in the firmware level to be like, okay, like this thing is wrong. Like we need to update this at the firmware to like actually fix this particular thing. So we worked pretty closely with Dell and Nvidia. Yeah, that's what I'm saying. Like this stuff gets complicated. And the thing is like, you know, taking a step back, the whole reason we're doing this, right, is that we knew that this was going to be complicated. There would be these kinds of failures. And if we're just using, you know, AWS or some other cloud provider, these errors are still gonna be there and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong. And so we would much rather be able to like call up Dell and say, hey, this isn't working. And they're like, yep, okay, cool. Let's debug it together. Oh, I see. Yeah, cool. We'll ship a firmware update and actually fix this for you. That was a much better experience than like, great, just magically fails. I guess we restart and hope that that machine goes away. Like that's not a very good place to be. So yeah, that's kind of the first place is getting to a place where like GPU training is working on your single node machines. You can observe stuff. We have tons of tooling around like, you know, Prometheus and all sorts of other tools for understanding what's going on in these machines because you don't want to be like logging into each one and looking at the temperature or something you really need to have tooling to collect all these metrics, et cetera. Unfortunately, all of the scripts that we have for this are like for this entire cluster and for all this infrastructure are a little bit like special purpose for our particular thing. So it's not that every script that we have, it's not that you can just like take this and plug this in. Even if we did open source all the tooling that we have, you'd still have to do like a lot of work to open source it. What we are releasing is as many of the things that we can that are going to be useful for other people. You're still going to have to have some way of kind of managing these things, making your own like logging aggregators, et cetera, et cetera. So that's kind of bringing them up to the like, you know, the single nodes that are working. From there, it goes into, I'm happy to keep going if you want. Well, I just want to leave the opportunity for John
SWYX [00:18:53]: to comment if there's anything that's different from how he runs things.
JONATHAN [00:18:57]: Oh, I mean, all I'll say is I'll endorse this and say this s**t is hard. Like this is really, really hard. And, you know, I have a special props to, you know, the folks in Vue because they were building this from the ground up. You know, at Databricks and at Mosaic, we typically work with cloud providers because some of this stuff is just, there's too much to handle. It's complicated. There's a lot to deal with. And this doesn't even get into things like physical security, you know, securing power if you're the data center operator. Like this gets infinitely complicated and you have to abstract somewhere. Like, you know, and then you get to the folks who are literally building their own custom chips and like, good God.
SWYX [00:19:36]: Like, oh my God, that's, you know,
JONATHAN [00:19:38]: if you're one of those folks, you're having, you know, pour one out for the infra people at some of the AI chip startups who are having a really, really interesting time right now. But this stuff is really hard. And I don't think we talk about it much because there's so many other things that are hard. But the other hard things, I think everybody's becoming pretty familiar with at this point. This is something that I don't think there's ever really been a comprehensive discussion of, at least not that I've seen.
SWYX [00:20:00]: Yeah, so my impression is that you guys, Mosaic, have your own software for sort of spinning up and down machines, just like Imbue had to build. But Imbue probably, it sounds like Imbue, you guys went fuller stack. I don't know how to describe it. Like Mosaic is not working with Dell on like their firmware.
JONATHAN [00:20:21]: No, no, we're typically working with like, you know, pick your cloud provider on their Dell firmware or what have you. Like, it's kind of, I think one of the things, I don't know, Josh, you can correct me on this. It's kind of impossible if you're doing training to not go all the way through the entire stack, regardless of what happens. Like somehow I'm still chatting with cloud providers about power contracts, even though the whole point of dealing with the cloud provider is not to have to think about power contracts. Somehow I'm still asking them about which InfiniBand provider they used this time to see if this is part of the bad batch of cables I encountered on that cloud provider or what have you. Or like, we're still talking about a firmware update from pick your provider. You can't not do this. It's convenient that they have data center staff who are worrying about what to send back to which provider when, and they have people who can go and wait for the InfiniBand cables so they don't get stolen outside. But, you know, it's kind of, it's impossible not to really go full stack if you're thinking about the infrastructure at all. I don't know, Josh, correct me. No, I think that's right.
JOSH [00:21:17]: That's what we expected from the beginning as well, is that we would inevitably have to get into the details here. And I'm glad that we kind of just planned for it. I think it made it a lot easier from our perspective to have direct control over this. Instead of having to go to the cloud provider that goes to the data center, that goes to the supplier, we could just go direct to NVIDIA or Dell
SWYX [00:21:37]: or the data center,
JOSH [00:21:37]: whoever was responsible and be like, hey, this thing needs to change. And they're like, oh, okay. Yeah, that is our responsibility. Great, we can fix that. So it was just a lot easier for us to fix these bugs than if we had to go through an extra layer of email.
SWYX [00:21:48]: Something we discussed in the pre-show was that you had a rule of thumb for your cluster of reliability. You say here in the post, by and large, you expect around 3% of your machines to break every week. So you're basically going to turn through all your machines in a year.
JOSH [00:22:04]: As it says in the post. So that would be true if it was a uniform failure like that. But as it says in the post, it's usually these kind of problematic nodes. And to be clear, that is the number that we've heard from other people is like they're having about 3%. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig into a large, maybe larger number than we should have of these failures and get to the root cause of it and be like, oh, okay, like that's exactly what's going wrong.
SWYX [00:22:33]: How do we fix this?
JOSH [00:22:33]: How do we prevent this from happening? How do we make automated checks for this so that if it does happen, it just goes back to whoever owns that particular part of the process and they can fix it immediately.
SWYX [00:22:43]: And that's part of what you're also open sourcing, which is the health checks, right? You got the NIC health checks, GPU health check, this space health check, Docker D message. I don't know what that is.
JOSH [00:22:52]: That one is just a lot of stuff.
SWYX [00:22:54]: Yeah.
JOSH [00:22:55]: That one is one where we realized that actually like when these machines boot, sometimes they wouldn't actually boot cleanly all the way. Or when they rebooted, they had problems that they didn't have when they were working before, which was kind of frustrating. Like usually if you restart your computer,
SWYX [00:23:08]: it gets better.
JOSH [00:23:08]: Here you restart. It did not get better.
SWYX [00:23:10]: It got worse.
JOSH [00:23:10]: That was very frustrating. So this health check looks at every particular line we've ever seen from the boot, like in D message, like every single log line that your computer emits
SWYX [00:23:21]: and says like,
JOSH [00:23:21]: have we ever seen this before?
SWYX [00:23:23]: Is this expected?
JOSH [00:23:23]: Is this in the right order? Or is there something out of place? If there's anything out of place, let me say, okay, great. Like now it goes into this, like longer, more triage list of like, all right, great. Like, is this acceptable?
SWYX [00:23:33]: Should we flag this?
JOSH [00:23:33]: Like, should someone take a look at this? So we're looking down at a very, very granular detail level, what's happening on these computers to make sure that nothing is out of place. And that's critical because without that, if you're running your training, as Jonathan said, and this thing is slow, like what are you supposed to do? Right?
SWYX [00:23:49]: Like you really,
JOSH [00:23:49]: you really want to be very certain that like all 4,000 of these GPUs are working like they're supposed to.
SWYX [00:23:54]: We know that.
JOSH [00:23:54]: And so if it's slow, it's because like we messed up the config or something else and not because of this earlier thing that's like really hard to detect in software later.
JONATHAN [00:24:01]: Yeah. I think the, I'm just curious to ask,
SWYX [00:24:03]: like, you know,
JONATHAN [00:24:03]: suppose you were to set up another, let's say another H100 cluster and it were at a different data center. And instead of the vendor being Dell, it was super micro or what have you. How much of this would be repeatable? And how much of this would you have to redo? I, you know, I genuinely don't know.
SWYX [00:24:18]: A decent amount.
JOSH [00:24:19]: I think it would go a lot faster the second time. I think there's lots of learnings that we had. And also the blog post,
SWYX [00:24:24]: you know, yes,
JOSH [00:24:24]: we are releasing the health checks, releasing some scripts, but a lot of the valuable stuff is also in the blog post itself, in the details and kind of the, you know, the learnings that we've had and the sort of errors that we run into. We tried to as much as possible surface those to other people
SWYX [00:24:36]: could learn from those
JOSH [00:24:36]: and avoid the same mistakes or failures as well. But I think it would go a lot faster.
SWYX [00:24:41]: Although, yes,
JOSH [00:24:41]: there would certainly be some things that'd be a little bit different. I mean, there'd probably be different CPUs
SWYX [00:24:46]: or whatever,
JOSH [00:24:46]: but I think a lot of that stuff is less,
SWYX [00:24:49]: it's less,
JOSH [00:24:49]: that's the like, that's less variable. I think most of it would apply the second time around. Although I'm sure next time
SWYX [00:24:56]: we're building one,
JOSH [00:24:56]: it'll probably be, you know, at a scale that's 10x as big with a different chip or something like this.
SWYX [00:25:00]: And then who knows?
JOSH [00:25:01]: Yeah, with Kinect X8,
JONATHAN [00:25:02]: that will have its own fun behavior and all that good stuff. Yeah.
SWYX [00:25:06]: Perhaps there's something that people don't discuss about, and you don't even talk about this in the blog, but I always wonder is what is the timeline that's like kind of reasonable for this amount of work, at least the initial stages? And also what does the team composition look like for setting up a cluster, right? Like what are the mix of skills that you typically would require to get all this going?
JOSH [00:25:27]: I'm, I can't really speak to typical. One thing I am very proud of is how much we accomplished with such a ridiculously small team. Like our infrastructure team is like, you know, fluctuates from week to week, depending on like how many things are on fire and how much we need to build. But it's like between like three and six people, like it's small. It's not like some huge team of like tons and tons of engineers. But those people are very, very good at what they do. And so that has allowed us to get a lot of mileage out of out of these things. I think it's not that we're building everything, right? It's not that three to six people build this whole thing. I definitely want to like, you know, say thanks very much to Dell and H5 and NVIDIA and the other people that have done a lot of the work, like to bring up this cluster, you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables. So that's 24,000 things that need to be plugged in. Like that's just a lot of stuff to plug in, right? And you don't want to mess it up. Like each one needs to be done correctly. Like it's a little bit loose. Like it doesn't really work.
SWYX [00:26:23]: If you break it,
JOSH [00:26:23]: you need to replace it. Like there's a lot of work
SWYX [00:26:26]: that goes into this.
JOSH [00:26:27]: Yeah.
SWYX [00:26:28]: And then, you know,
JOSH [00:26:28]: that's just like that's it. That's if you were to do everything right the first time.
SWYX [00:26:32]: And if you didn't
JOSH [00:26:32]: have to fix anything. But inevitably, you know, you will have to replace something, which means like taking all the wires out, pulling the thing out, taking all the GPUs out, going and fixing some cable, putting it all back correctly, putting it back in, doing this every time. So there were a lot of people at Dell, NVIDIA and at H5 that all helped a ton with this stuff. I don't know the exact size of the Dell team. It also fluctuated over time.
SWYX [00:26:55]: Yeah, excellent. And then, you know, you so you have all the hardware set up and now you're firing it up for a single node. There's a long description that you guys have about just like monitoring the MFU, right? And what each situation might look might be indicative of. One of the most interesting things to me that I saw from here is like, you know, if training immediately starts off at 60 to 80% MFU, something's wrong.
SWYX [00:27:24]: But like, you know, like what what are like, you know, some anecdotes or, you know, notable scenarios here that you might you might call out as maybe counterintuitive or super interesting.
JOSH [00:27:36]: There's just so many of them. I mean, one of them, which I think is probably pretty common, like common knowledge by this point. But like we did have a sort of like
SWYX [00:27:46]: which one was this exactly?
JOSH [00:27:47]: I think for the MFU, like gradually getting worse over time. I think that one, when we saw that the first time we were like, what the heck is going on? Like, why does it get just like a little bit worse? This is so strange. Like, what is it getting lazy or tired or something? Like, is it heat? Like what's going on? And in this particular case, it was memory fragmentation. Because you have hundreds of machines, they're doing garbage collection slightly different times. And then they get slightly further apart and slightly more and more jittered until eventually they're all happening kind of at random times. And just like really messing up each one of your steps. So you just turn off garbage collection and call it a day, basically,
SWYX [00:28:20]: to be honest.
JOSH [00:28:20]: There's other things you can do if you want to be a little bit more sophisticated about it. But you can also just manually
JONATHAN [00:28:25]: have it all garbage collect on some interval. Like that's what we've done. We just have a garbage collection callback that just runs. But I've seen the exact same thing.
JOSH [00:28:33]: Yeah, yeah, exactly. So I thought that one was kind of funny. And we did trace that one down and look and we did find the actual call. Like, again, this goes to like having good tools. So we had really good tools where we could look at a bunch of like actual traces in C and be like, OK, cool. This is the thing that's taking a lot of time. Or like, you know, this is the thing that doesn't quite line up here. Like, oh, I guess it's garbage collection. OK, cool.
SWYX [00:28:52]: Interesting.
JOSH [00:28:52]: Yeah, let's just try taking it off.
SWYX [00:28:54]: OK, great.
JOSH [00:28:54]: That's what it was. Now we can fix it. So for each of them, like basically bugs are not hard if you have good tools. But if you don't have good tools, bugs can be very, very hard. So similarly for like heat, another thing that we saw was like, oh, you know, the CPU is getting throttled. OK, well, it's easy to see if you're monitoring the CPU throttling or monitoring the heat. If you're not monitoring that, it's really hard to know why it's just suddenly one of them is going slower. I noticed also in the piece
SWYX [00:29:17]: that you mentioned FSDP with 0.3. Actually, we met, I went to iClear and Guanhua from the DSP team was there presenting 0++. I was wondering if you want to make any call outs to, you know, particular open source or open library or open whatever implementation teams that were super helpful in your process. I think we ended up actually
JOSH [00:29:39]: pulling from a whole bunch of different ones to pull things in into our own particular pipeline. So we use things from NVIDIA's, you know, Megatron stuff. We use stuff from probably DeepSpeed. I think we pulled in a bunch of different pieces from a bunch of different places. So it was really nice to see all these working open source like examples. I think I really appreciate all the effort that has gone into actually tuning these things because you can tune them, but it's a lot of work to like tune this stuff and do all this stuff from scratch. It's really nice to have like a working example. I think those are probably the two biggest ones, DeepSpeed and Megatron alone, but there are probably other ones as well.
SWYX [00:30:13]: Is there a particular thing in the ecosystem where you would call out as like, you know, there should be something here that is open source, but like it's not really, it's like everyone kind of builds it on their own. I want to say something with the file system because everyone talks about the file system eventually.
JOSH [00:30:28]: The file system actually was,
SWYX [00:30:30]: I mean, we did something
JOSH [00:30:31]: kind of dumb there. Like we have our own sort of local mirror so that we can, you know, like a crappy version of S3
SWYX [00:30:38]: that's local,
JOSH [00:30:38]: but it's just a pretty simple script, right?
SWYX [00:30:41]: Like I think we run like
JOSH [00:30:41]: a little web server that just like serves files and then, you know, it can upload them
SWYX [00:30:45]: and download them.
JOSH [00:30:45]: Okay, great. And part of the reason we did that is that our internet connection
SWYX [00:30:50]: in the beginning
JOSH [00:30:50]: was not the like full speed
SWYX [00:30:52]: one that we would
JOSH [00:30:52]: eventually have. And so we are a little bit more kind of bottlenecked in terms of internet bandwidth. And so we had this. I think we looked at a bunch of services out there like Minio and some other ones, but a lot of these like come with a lot of extra overhead and maintenance. And since we already have so much infrastructure
SWYX [00:31:09]: to deal with,
JOSH [00:31:09]: we kind of didn't want to, you know, bring in a whole other like cloud provider, virtualize something, something.
SWYX [00:31:14]: We just wanted something simple.
JOSH [00:31:14]: So we went with that, which has been quite helpful. Like our tools
SWYX [00:31:19]: are usually quite simple.
JOSH [00:31:19]: It's like Bash and Python and SSH and Docker. Like we'd like to keep things simple so that's easier to debug, like less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes,
SWYX [00:31:30]: for example,
JOSH [00:31:30]: and we just directly launch these things. And it's just been much easier to debug this way. One tool actually that does come into mind that I will call out is Kraken from Uber. That was great. We love that tool. We were a little bit skeptical. What is it?
SWYX [00:31:44]: I'm sorry. Yeah.
JOSH [00:31:45]: So Kraken is this, yeah, it's a distributed like Docker registry, basically, that uses BitTorrent to like transfer things between the machines in a sort of nice optimal way. Like in the very beginning, the naive way is like you have this one Docker registry, which was outside of the cluster. So every time we change an image, you know, there's many gigabytes that each of the 500 machines needs to download.
SWYX [00:32:07]: So that just takes
JOSH [00:32:07]: a really long time. So what this thing does is like just one of them downloads it and then like they all sort of broadcast all the pieces to each other. And it was just like a really nice, fast way of getting these images down. And it was very robust.
SWYX [00:32:19]: Like there's a lot
JOSH [00:32:19]: going on under the hood, but I think it's a pretty cool tool that we haven't really had any bugs with it at all. Amazing.
SWYX [00:32:26]: Yeah. I mean, that's all my questions, I guess, for the info piece. I don't know if, John, you had something that you were sort of burning to ask or.
JONATHAN [00:32:33]: No, all I can say is just same
SWYX [00:32:36]: in a lot of places, like, you know, and they're done that
JONATHAN [00:32:38]: seeing this plus one. I think the one big difference, you know, perhaps in philosophies is we've tried to basically standardize on as much commodity stuff as possible, just because, you know, I think the reason I asked about trying to do this
SWYX [00:32:50]: on multiple different
JONATHAN [00:32:50]: pieces of infrastructure is like, I think we're running on like six or seven different clouds right now. And everybody has done something slightly different. And my gosh, the little differences add up as you know, you've seen. And so, you know,
SWYX [00:33:04]: our philosophy has been like, whatever the hell
JONATHAN [00:33:05]: we can standardize, please let's standardize it. Like vanilla off the shelf FSDB.
SWYX [00:33:10]: And like, you know,
JONATHAN [00:33:10]: we wrote our own data loader, but we've tried to make that as much of a standard as we can across our infrastructure and in Databricks, because things just start getting really complicated
SWYX [00:33:18]: or like we use
JONATHAN [00:33:18]: Kubernetes extensively because it at least gives us a uniform set of APIs. Like that's our hardware abstraction layer to a certain extent for everything else. So it's just, you know, a difference in philosophy there. But otherwise, like, yeah, this stuff is really, really hard. And I feel like we take for granted how much of this, you know, is done for us when you go and you just query chat GPT, for example. Like, oh my God, everything going on underneath that, you know, it's kind of a miracle that the machines boot up, let alone that you can like query a giant language model that's probably doing inference across multiple machines and was trained across thousands of machines. Like, you know, minor miracle.
SWYX [00:33:54]: Yeah, it is an awesome amount of power that we invoke with a single API call that we take for granted these days. It's absurd. Yeah, I mean, like Kubernetes, like that point about Kubernetes, I will say as a former AWS employee, like it seems like it would be ideal for imbue to at some point make it more abstracted or agnostic because you're going to want to, you know, replicate your setup. We do have our own
JOSH [00:34:19]: sort of replacement. It's just a much simpler version of Kubernetes. Kubernetes is really designed for running services, not for running experiments. Like that's not its like main architecture. And so for us, like we have everything that's like, cool, you're going to run an experiment. So you want it to run to completion, right?
SWYX [00:34:34]: OK, great.
JOSH [00:34:34]: Like the primitives are sort of built around a slightly different style. And that makes it a lot easier, like just a lot simpler to fit that the nature of like these machines are going to disappear. They will need to be rebooted for infrastructure upgrades. They will like something will happen to the GPUs. Failure is like baked into this as like a core part of our infrastructure. So it's not that we don't have an abstraction. It's that it's a sort of simpler, more tailored abstraction for the particular work that we're doing.
JONATHAN [00:34:58]: Yeah, I think it all depends on what your goals are. And like, I think the challenge in a lot of the deep learning stuff right now is that people are trying to like, people often build things that are more complicated than necessary to get the job done. And the complication is the enemy of everything. You know, don't use a fancier parallelism strategy than you have to. Don't use a fancier set of libraries than you have to.
SWYX [00:35:18]: Don't do anything
JONATHAN [00:35:18]: that you don't have to do because it's hard enough as it is. Like, don't overcomplicate
SWYX [00:35:23]: your own life.
JONATHAN [00:35:23]: Don't try to bring in more tools or more fancy architecture tweaks if you absolutely don't have to.
SWYX [00:35:29]: Like getting to the minimum
JONATHAN [00:35:30]: necessary to get the job done. And it's really tempting to want to try to use everything. So like, I totally understand that one.
SWYX [00:35:37]: I think the last piece I'll maybe call out is that I'm just going to weave this in just because I see the opportunity to do it. Are there any infrastructure shifts that need to be, that need to rise because of changing architecture? So I think, for example,
SWYX [00:35:57]: you're announcing a dense model, a 70B dense model, whereas John just worked on DBRX and the image-to-text model, which presumably has different bottlenecks.
JONATHAN [00:36:10]: That's correct for us. You know, we train both dense and mixture of expert models. The one we happened to, you know, kind of get permission to open source was a mixture of expert model. And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP 03 style, where there's just a lot of parameters getting shuffled back and forth. And your ratio of kind of compute to amount of data that you have to shuffle back and forth becomes a lot worse because you're now, you know, you're only using a fraction of the parameters for every token instead of all the parameters. And so we had to really push the envelope on getting all the stuff to the right places on time. And so actually the networking part of DBRX was the single hardest thing, I think, of the entire process. Just get MOE training, working at scale across a big cluster. We still managed to, I think, do it all with commodity parts, which was very exciting. You know, we were using FSTP and we eventually used HSTP so that we could have HSTP as a version of FSTP where you have multiple smaller replicas and you're doing data parallel within those replicas. And that helped a lot with network latency issues that we were running into just because we were transmitting so much data, you know, for every single part of the process. I think it actually, like, it was instructive for how Google designs their hardware and software together personally. Their training, as far as I understand, using kind of a 03 style of training and have been for a while. They also train mixture of expert models. TPUs have a very different network bandwidth to compute ratio. They have a lot more bandwidth just objectively. And TPUs per chip tend to be a little bit less compute intensive and have a little bit less memory. You know, it's just a different design choice. So the ratio of flops to bandwidth is very different. And that means that it's much easier for Google to be able to pull off
SWYX [00:37:54]: some of this stuff.
JONATHAN [00:37:54]: They also have interesting, you know, Torus style network architecture or Torus style, like, literal network architecture
SWYX [00:38:00]: is not like the model,
JONATHAN [00:38:00]: but the network.
SWYX [00:38:02]: Is this the sort of block attention? I forgot what you call it. So this is just more or the,
JONATHAN [00:38:07]: yeah, this is more, not the ring attention, but these are the ring all reduces. Like you have three different dimensions of rings because they kind of put you in these three dimensional Toruses from what I understand. And so like, you know, Google's infrastructure in some sense is kind of, I wouldn't say built for this, but maybe the way that Google trains models is built for a slightly different bit of infrastructure they have. And it's kind of neat to think about that. You know, as one thing that I think NVIDIA announced for, you know, for, for both the GH200 and the GB200 is this hybrid networking where you'll have blocks of NVLink network chips. I think for the GB200, I think it's like groups of 72 GPUs will all have NVLink to each other. So higher bandwidth, then you'll have normal networking of some kind, InfiniBand or Rocky or what have you between these blocks. And that's kind of a, you know, it's a change due to the fact that, you know, it's hard to build really high bandwidth networks over very large groups, but it is now a blocked networking. And you have to think about how you architect your model and your parallelism differently. You also have to think about fault tolerance differently because it now matters where you lose a GPU, whereas it didn't before. So, you know, it's, it's, it's just all really interesting and really fun speaking personally, but it's going to mean new nightmares when we all move to that generation and have to think about, you know, new versions of these problems.
JOSH [00:39:20]: As you go up to larger scales, it gets quite different. Like right now, you know, if you're experiencing, let's say, for example, you experience a GPU failure every day, that's fine.
SWYX [00:39:31]: Just restart.
JOSH [00:39:31]: If you make your thing 24 times as big, now it's once an hour. Now it stops being quite as easy to just restart, right? So now you have to kind of break, like bake in this sort of redundancy that you didn't have before. So I think as you go up in scale, you end up running into like a lot of really interesting problems that also inform the, the actual like design. Yeah, I mean, as an orchestration guy,
SWYX [00:39:52]: this is why I always emphasize like very cheap storage or very fast storage. So you can checkpoint more, but I don't think that's probably not the best solution to for fast, you know, training.
JONATHAN [00:40:05]: Which works fine when you're doing language and then you move to vision or video. And then, you know, you have multi petabyte datasets
SWYX [00:40:12]: and getting, you know,
JONATHAN [00:40:13]: cheap, fast multi petabyte storage starts to bite. Like I've certainly encountered issues where the literal data center where my GPUs were did not have enough, you know, object store to fit the datasets that people wanted to bring into that data center from whichever users were, were trying to bring them in. And then you get to a whole
SWYX [00:40:31]: different world of hurt
JONATHAN [00:40:31]: where you have to keep your data in a different region because the region is just out of storage. So things get fun really fast.
SWYX [00:40:39]: Speaking of vision, Josh, actually, you know, Embu is an agents company, but you're only, you're announcing a text-only model. What, where does, where does the vision side come in?
JOSH [00:40:49]: I think we've actually done a lot of work in the past and people can see kind of our blog posts about sort of self-supervised learning and some other kind of vision-related stuff in the past as well. So we're very familiar with, with that stuff. But I think our main focus right now is on kind of, as we say, coding and reasoning. And there, there's certainly a visual component to some problems. But, you know, it's not necessarily required for all problems. And actually we found that for most of the kind of like code writing and, and reasoning problems that we care about, the visual part isn't really a huge important part of it. Sometimes if you really need to, you can maybe describe
SWYX [00:41:24]: the thing.
JOSH [00:41:24]: There are other like, you know, multimodal models that you can use off the shelf to sort of plug in for those particular pieces
SWYX [00:41:30]: that you need, right?
JOSH [00:41:30]: Like if something is driving a browser or whatever, like you can sometimes get away with not having to have that baked into the original model. So our folk were, you know, in a sense, we kind of do a lot across the stack. We're working on our own infrastructure and pre-training and RL and fine tuning and products and everything. But in another sense, we're very narrowly focused on the application side. So all of the stuff across the stack is kind of going toward a very particular purpose. And so that particular purpose right now doesn't really need vision. So we think that people are going to make all sorts of really cool image models
SWYX [00:42:00]: like Jonathan, right?
JOSH [00:42:00]: And all sorts of interesting multimodal models into the future. We'll let them go do that. That's great. We'll take advantage of that, partner with those people in the future. And right now we're really focused on kind of the core reasoning and coding capabilities and aspects of the model.
SWYX [00:42:14]: I wanted to go into carbs since that's kind of the next layer of the stack. We talked about carbs in the first episode with Kanjin because you've actually had a blog post about it like a couple of years ago. Maybe let's introduce it.
JONATHAN [00:42:26]: Has that been a couple of years now?
JOSH [00:42:28]: No, it must have been at least one year. Hopefully it's not multiple years.
SWYX [00:42:32]: Sorry, I'm counting AI time. Yeah, yeah. Yeah, I was going to say
JONATHAN [00:42:35]: you're making me feel really old right now.
SWYX [00:42:39]: I count everything before the generally intelligent rename as like, you know, prehistory. Yeah. And now sort of modernity, right? So I actually thought carbs was more about hyperparameter optimization in a sense of like sort of parameters, hyperparameter search. Whereas, you know, when you introduced it, especially in this blog post, it's more about scaling laws and predictability of like, are we sort of in the right ballpark before we scale things up? Maybe sort of recount the history of carbs.
JOSH [00:43:10]: Yeah, so it really is a little bit of both. So carbs is, it's maybe a backronym, but it's for cost aware Pareto region Bayesian search. So this is about technically how it works, but carbs is like, you know, we like pastries and stuff.
SWYX [00:43:26]: So great, why not? But the point is that
JOSH [00:43:29]: it's a cost aware hyperparameter tuner. So most hyperparameter tuners, you kind of say, OK, here's this objective function. I want you to make this number as big as possible or as small as possible, whichever direction you want to go. So yeah, just go make this number, you know, as small as possible. OK, so it'll try a bunch of different
SWYX [00:43:46]: hyperparameters,
JOSH [00:43:46]: a bunch of different configurations
SWYX [00:43:48]: to figure out, like,
JOSH [00:43:48]: how do I tweak your network and architecture, et cetera, to get the kind of best performance I possibly can. That's usually saying, like, you know, almost all of these hyperparameter configurations are, let's say they're all going to use the same number of GPUs or the same number of nodes.
SWYX [00:44:01]: So it's going to run
JOSH [00:44:01]: for the same amount of time.
SWYX [00:44:03]: So you can do that.
JOSH [00:44:03]: You can get a number out and that's great. But what carbs does is it says,
SWYX [00:44:07]: OK, actually,
JOSH [00:44:07]: what if we relax that constraint? What if we say each of these different points, we're going to model how expensive it will be to sample this configuration. So if what if we train with just one one hundredth of the data? Like, how well can we do?
SWYX [00:44:19]: What if we train
JOSH [00:44:19]: with one tenth of the data? What if we train with all the data? That way you can understand, like, as we get more and more data, as we spend more and more compute,
SWYX [00:44:26]: as we make a bigger
JOSH [00:44:26]: and bigger network, how does performance change with these things that change? Like how expensive it is to even explore this data point. So by doing that, we can see the scaling laws for not just, you know,
SWYX [00:44:36]: the scaling laws
JOSH [00:44:36]: from like the, you know, Chantilla paper, the scaling laws for all parameters. We can see how does how does the number of layers change with this? How does the, you know, the learning rate change? How do the like, you know, various types of regularization change? So you can see these nice scaling laws. And as you're going across costs, like how should this be changing as you're scaling up your model? So that, coupled with the kind of metric that we chose, which is a very precise way of measuring performance, allowed us to really like hone in on parameters that worked really well
SWYX [00:45:05]: and understand, like,
JOSH [00:45:05]: how do we want to scale those up, especially as we're changing
SWYX [00:45:08]: things about the network?
JOSH [00:45:08]: Like one of the things that we did is we used a custom tokenizer. As we change this tokenizer, changes a bunch of other things about the model. So how should we scale up this entirely new tokenizer? Like no one has ever made a model this large with this tokenizer before. And so how do we want to
SWYX [00:45:22]: change all these things?
JOSH [00:45:22]: Harps kind of shows you, like, look, as you change these parameters, like these other ones are kind of dependent on this.
SWYX [00:45:28]: Like this is the, these are
JOSH [00:45:28]: the relationships between them. So you can better understand, like, OK, if I'm going to scale this up 10x or 100x, like, where do I want to be? I can only go so far. And so, you know, we did run, like, I think maybe it was like a 14b one or something
SWYX [00:45:40]: like that to check.
JOSH [00:45:41]: But and so we had a bunch of like 1b or 14b and then at 70b. I don't think we had a, I think we just did like one at 14b. So you can, we get to check that like, oh, is this on the curve? Like, is this where we expect? It was like right there. So then great, go on to the next one. Yeah, I mean, that makes a lot of sense.
SWYX [00:45:56]: I wonder if, so one of the key questions, and correct me if I'm wrong, but like usually people do search or do their evals just based on loss. But you actually evaluate based on, you know, the sort of end state evals that people might expect, like HellaSwag and Lombata, whatever. What is the norm here? Is there a norm?
JOSH [00:46:20]: Yeah, I don't know if there's a hundred percent.
SWYX [00:46:21]: I don't know. I only see loss on most people's reports.
JOSH [00:46:25]: I think it's easy to, like, loss is very nice because it's very precise. It will tell you, like, very fine grained differences between like really small changes in your hyperparameters or network architecture. Whereas, especially at the smaller scales, if you're looking at like accuracy, it's very noisy. Like it might be zero or a hundred or like, you know, fluctuating by like 10 or 20 percentage points, which makes it really hard to tell, like, did that change actually mean anything? So our loss is sort of a combination of these two. Instead of saying, like, let's just look at perplexity, we say, let's look at perplexity on the tasks that we care about for multiple choice questions effectively.
SWYX [00:47:00]: So we're saying like, yes,
JOSH [00:47:00]: this is formulated as a multiple choice question, and we're going to look at the, like, you know, the loss of perplexity for this particular answer token. And that ends up being something that's like both targeted to what you actually care about and also very precise. The nice thing about this though is that it's independent of the data that you train on. One thing that's annoying about perplexity or about loss is that as you change your data set, this is really obnoxious because now it fundamentally changes your loss, right? And so you can't tell, like, how do I tweak my data set? But because we have this held out evaluation data set where we're looking at perplexity, we can actually change the data mix. And so CARBs actually control what is the mix of data that we want to see, like how much code, you know, how much internet text, et cetera, in order to figure out what is the best optimal mix of data and we could do that because we have this other metric. So that was one of the things that was really, really helpful.
SWYX [00:47:46]: I think there is a trend overall about changing data mix as training goes on. I don't know how, you know, we're deciding not to talk about data sets in this podcast, but what have you observed about the changing data mix question?
JOSH [00:48:06]: We did some experiments
SWYX [00:48:08]: and we've actually talked
JOSH [00:48:08]: to a bunch of researchers who are doing work here as well
SWYX [00:48:11]: and looking at kind of
JOSH [00:48:12]: their experiments on this. And we were originally pretty hopeful because it sounds like something that should work and make sense, right? Like, oh, cool. Like maybe you would have your model, like learn the basic features
SWYX [00:48:22]: and then over time,
JOSH [00:48:22]: it could get really good at these complicated math problems or coding or something, right? But it just turns out that like, it's just not the way it works. Like we've done so many experiments and you can get like a tiny, tiny little boost from this, but it just is not like, it's just not the important thing, at least in the experiments that we've seen. So yeah, we've kind of, we're letting other people
SWYX [00:48:40]: explore that more
JOSH [00:48:40]: if they want, but that just doesn't seem like the most promising direction for us.
JONATHAN [00:48:44]: We've had some surprisingly good luck with this. We just released a paper on it. The details matter a lot and it really matters what you're trying to do with the model.
SWYX [00:48:53]: Yeah.
JONATHAN [00:48:53]: But it's been quite effective for us depending on the setting. And certainly when we're thinking about domain-specific models, this helps a ton. You know, to a certain extent, you can always think of this as like early fine tuning. But yeah, I like, there've been little glimmers of this in the literature for years. Like especially, I think the Gemini 1.5 paper mentions this. And I don't remember whether the Llama 3 paper mentions this,
SWYX [00:49:15]: but it's kind of,
JONATHAN [00:49:16]: it's one of those, like people have different ways to get to these endpoints.
SWYX [00:49:20]: I think, you know,
JONATHAN [00:49:20]: there are the architectural tricks that each lab has to mitigate loss spikes or what have you. And everybody's got, you know, their own bag of tricks and it leads to kind of sometimes this contradictory information. It's not contradictory. People are just kind of exploring
SWYX [00:49:33]: different parts of the space
JONATHAN [00:49:33]: in some sense. And there are lots of ways to get a great model. But certainly for us within our config, and it seems like, I guess for the folks at Google, within kind of the part of the world they live in, changing the dataset has helped, but the details matter a lot. And it's really hard to get those details right for the reasons Josh,
SWYX [00:49:48]: you know, just mentioned.
JONATHAN [00:49:48]: Like there's a lot of search involved and you essentially have to make hard choices about
SWYX [00:49:52]: what parts of the space
JONATHAN [00:49:52]: you're going to search and which ones you're going to leave be. And so, you know, some people have done an amazing job. Like I think the, who is it? The Deep Seek folks have done an awesome job looking at like batch size warmup. And that's been really, really fruitful for them. You know, other people are looking really hard at things like data mix, but it just gets tricky to look at everything.
JOSH [00:50:09]: Yeah, I think we've found that like we could get some things that looked like gains from datasets. But one of the things that I like about carbs is that when we applied carbs to like properly tune things, then a lot of those kind of evaporated. Whereas like, like if we just tune these other parameters, actually we can get almost the same gains without having to do this more complicated thing. So at least in the experiment and in the settings that we've, like in the particular metrics
SWYX [00:50:34]: that we care about,
JOSH [00:50:34]: we haven't seen these kind of like pan out or scale up in quite the same way. But not to rule it out. And I think you're right, Jonathan,
SWYX [00:50:41]: that there probably are
JOSH [00:50:41]: a lot of like details that go into like exactly what is the metric, exactly what is the dataset, exactly which, like what schedule are we using for this. And I certainly wouldn't rule it out working.
SWYX [00:50:52]: Quick question about emergence. Doesn't emergence throw a spanner into a theory of carbs? Ah, so there is a paper
JOSH [00:51:01]: of which I really liked and I think informed
SWYX [00:51:05]: a little bit of how
JOSH [00:51:05]: we thought about this, which is are emergent properties of language models a mirage? And I think if you look at that paper, it actually makes a relatively compelling case that in fact, you know, this emergent behavior that you're seeing is not really emergent behavior, but is really a function of the evaluation metrics that we're using. So if you look at accuracy as a metric, what's happening is that accuracy is actually going up continually over training, but it's in log scale. So it starts out at 0.001%, 0.1, 0.1, 10.
SWYX [00:51:35]: Only when you're going
JOSH [00:51:35]: between 10 and 90 do you see this happen, right? When you go from one in, you know,
SWYX [00:51:40]: a thousand getting right
JOSH [00:51:40]: to one in a thousand getting wrong, like there's many orders of magnitude happening here.
SWYX [00:51:44]: So when you're looking
JOSH [00:51:44]: at this in perplexity, then you just see this nice straight line. And so that's actually what carbs is exploiting. Like since we're, since our metric is in this kind of like perplexity log space, like you can see like, oh, it's just like getting better as you make it bigger in this nice, very predictable way. So that, and that is exactly what we saw. Like these things were really, really bad at, you know, predicting the multiple choice answer, just always guess A. OK, it's so terrible at it, but it was like learning to be less confident about that.
SWYX [00:52:09]: Yeah. One trick I saw from one of the papers recently was just like, just randomize the order of the multiple choice questions. And if you, if, if, if they, if they over, if that hits the performance a lot, then they're just basically memorizing the test set, which makes a lot of sense.
JONATHAN [00:52:28]: Yeah, this is, I, I mean, you know, I, I completely agree with what Josh said.
SWYX [00:52:32]: I think the, you know,
JONATHAN [00:52:32]: my bigger lesson is that anything can look however you want it to look. If you put it on a log scale to a certain extent and log, we love our log scales and deep learning for various reasons. Everything looks very clean on a log scale until everything looks very flat on a log scale. Um, I don't know. I like log scales always mix me up. That's, that's all I can say.
SWYX [00:52:51]: Great. I think the, the last thing I was, I was going to mention on, uh, carbs. Oh, well, I mean, let's, let's just kind of go right into evals because I think that's going to be, uh, the, the sort of crowd favorite. Um, so carbs, we already mentioned, um, you know, leans heavily on, uh, the sort of end evals that we would typically eval LLMs on, except that you had to make your own. Um, there are a lot of documented problems with many of the common evals out there and you fixed all of them. It sounds like, I don't know
JOSH [00:53:18]: about fixed all of them, but, uh, I think in the same way that we like to dig into the infrastructure and hardware and understand, like what actually is going
SWYX [00:53:27]: wrong?
JOSH [00:53:27]: Like what is the actual error on this machine with this GPU?
SWYX [00:53:31]: And why did that happen?
JOSH [00:53:31]: And how do we fix it? We take the same approach to the evaluations. So when we looked at the evaluations and actually looked at the data sets, you know, what we did is
SWYX [00:53:39]: like, okay, if we're going
JOSH [00:53:39]: to be, you know, evaluating natural language, understanding and reasoning, like, let's look at all the data sets that are out there. Let's actually look at a bunch of the examples and say, like, is this a good data set that we should use for evaluation? That's kind of how we selected the evaluation data set that we had. Uh, and then when we looked at the actual examples in there, we noticed like a lot of these are very messy. Like some of them messy
SWYX [00:54:00]: to the point of like
JOSH [00:54:00]: incoherence and some of the ones that we didn't choose. Uh, but even the ones that we chose, like people tried pretty hard on
SWYX [00:54:06]: these data sets.
JOSH [00:54:06]: They did try and clean them, but there's just a lot of data points in there and it's just easy to
SWYX [00:54:10]: make mistakes.
JOSH [00:54:10]: Right. And so, you know, it's not that they have a
SWYX [00:54:13]: hundred people looking
JOSH [00:54:13]: at every question, like that's just way too
SWYX [00:54:15]: expensive.
JOSH [00:54:15]: So you end up with questions that just don't make sense.
SWYX [00:54:18]: Somebody didn't really
JOSH [00:54:18]: see this. Somebody just clicked the wrong box for the answer. Uh, or the question makes sense in your head. When you write it, we've often seen this, it's not even like malice or
SWYX [00:54:26]: incompetence.
JOSH [00:54:26]: It's really just like, you know, you write this,
SWYX [00:54:28]: you're ready.
JOSH [00:54:28]: You're like, this makes
SWYX [00:54:29]: sense to me.
JOSH [00:54:29]: You show it to another person like that makes
SWYX [00:54:31]: sense.
JOSH [00:54:31]: You show it to a third
SWYX [00:54:32]: person.
JOSH [00:54:32]: They're like, this makes no sense at all.
SWYX [00:54:34]: That's because you're
JOSH [00:54:34]: kind of, you know, using a different meaning of
SWYX [00:54:36]: the word.
JOSH [00:54:36]: And then when they say that, you're like, Oh,
SWYX [00:54:38]: wow, you're right.
JOSH [00:54:38]: That is actually really confusing. It's easy for things to
SWYX [00:54:41]: kind of make sense in
JOSH [00:54:41]: our own head. So what we did for the evaluations is really dug into the details of each of these data sets and tried to ask, like, what makes a good
SWYX [00:54:50]: question?
JOSH [00:54:50]: What makes a good answer?
SWYX [00:54:52]: Like, what does it mean
JOSH [00:54:52]: for it to be ambiguous? We had a whole, like,
SWYX [00:54:55]: we looked at lots of
JOSH [00:54:55]: data, broke this down, asked lots of people
SWYX [00:54:58]: about all these
JOSH [00:54:58]: different questions to build a model of this and help us kind of clean these data sets. That was sort of one big piece of it. A second big piece was making sure that our data that we're training on is not data that we're testing on. So there we kind of took a step back and said, like, OK, well, let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves. And just make sure that this data is definitely not in the, you know, the training set. So we did that. And then we're able to, like, now be confident about, like, our performance of our model and also performance of other open source and other closed source models. Yeah, there's a lot there.
SWYX [00:55:33]: You had 11? I don't know how many data sets. I think so. One, two? Yeah. Any one you want to call out in particular to dive deeper on? Some of these are very famous, like HelloSwag, MitoGrand. Some are less famous, like Race. I don't know if... Race is a great data set.
JOSH [00:55:50]: See that one?
SWYX [00:55:51]: Yeah. Yeah. Just, you know, anything that's interesting you want on specific data sets? I think there are
JOSH [00:55:57]: a few asterisks in there. You know, definitely read the whole paper
SWYX [00:56:02]: as you're looking at
JOSH [00:56:02]: some of these, like the GSM8K one is a little bit weird. I think one that was
SWYX [00:56:06]: kind of funny,
JOSH [00:56:06]: it was, like, low performance on ethics from some of the more recent models. I think that was a
SWYX [00:56:11]: little bit funny
JOSH [00:56:11]: because the models, you know,
SWYX [00:56:13]: I think there was
JOSH [00:56:13]: a reaction to, like, oh, no, like, you know,
SWYX [00:56:16]: the models are saying
JOSH [00:56:16]: bad things.
SWYX [00:56:17]: And so they went way,
JOSH [00:56:17]: way in the other direction. And now, like, on the ethics data set,
SWYX [00:56:20]: it's always like,
JOSH [00:56:20]: this is totally unethical, even though it's really fine. So they've just been tuned to, you know, make sure they don't make any PR disasters.
SWYX [00:56:28]: I thought that was
JOSH [00:56:28]: a little bit funny. Not to say that it's necessarily like a flaw of the model, but just kind of like, you know, political or tuning opinion. I think the main takeaway, I was just going to say
SWYX [00:56:38]: the main takeaway
JOSH [00:56:38]: for many of the, like, actual performance is, like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated. Like, I think it's
SWYX [00:56:48]: important to look at,
JOSH [00:56:48]: like, you know,
SWYX [00:56:50]: like when you're
JOSH [00:56:50]: talking about performance on ANLI or race or pool queue or something, what you're really talking about is, like, performance on questions that make no sense. Like, it's just like, did it guess the answer in this, like, really weird scenario? Like, those are the ones that are left.
SWYX [00:57:03]: Like, when you look
JOSH [00:57:03]: at the performance on the ones that actually make sense to everyone, all the models agree.
SWYX [00:57:07]: We agree, like,
JOSH [00:57:07]: everyone's on the same page, which I think is kind of a really interesting result.
SWYX [00:57:11]: The question then becomes, you know, what are the new, like, set of evals that would be like the next frontier that often embeds with it your idea of what reasoning is, because it's obviously you're super interested in reasoning. And yeah, I mean, like, where does this, where does the state of evals go from here?
JOSH [00:57:30]: This work and this blog post is talking mostly about the public evaluations
SWYX [00:57:34]: and the things
JOSH [00:57:34]: that we can release. We do have our own internal evaluations. For example, one of them that we are releasing is the code understanding evaluation, which is about predicting,
SWYX [00:57:44]: you know,
JOSH [00:57:44]: what will this variable be or asking questions about code, et cetera. And that is one of the early benchmarks that we made that we can release. We can partly release it because we can generate an almost infinite amount of this data because these are programmatically generated. And so, you know, we're not really worried about there being like corruption in the kind of the training or test sets. So that makes it a little
SWYX [00:58:03]: bit easier for us.
JOSH [00:58:04]: But I think it's, you know, we have built other data sets as well that we can't release. Some of them, you know,
SWYX [00:58:09]: for example,
JOSH [00:58:09]: because they maybe use other open source code and so we can't redistribute it necessarily. Other ones, because, you know, that's, I think evaluations and data are like a core, important part of, you know, the business. And I think we take evaluations very seriously and are spending a lot of effort in terms of like, what exactly do we make as part of the evaluation set? How do you evaluate these things? We've done a lot of other stuff, you know, since these evaluations. But I think a lot around like code understanding for us, since that's our main focus. And it's a nice place to explore reasoning as well.
SWYX [00:58:40]: It sounds like you talk a little bit about like code understanding as like sort of variable level, like sort of very micro context. Is there a sense of like larger code context as well? I don't know what I mean by that, by the way. It's mostly just like if I told the senior engineer to go look at a code base, they would understand at a broad level, the architecture, but also the design decisions and be able to tell me that. I don't know if that's useful or not, but I mean, that's useful to me as a, as someone who might be working with them. Yeah.
JOSH [00:59:06]: This particular dataset is like the more low level code understanding,
SWYX [00:59:10]: like just literally
JOSH [00:59:10]: what happens in this code. And this is mostly because, you know,
SWYX [00:59:13]: this is part of the
JOSH [00:59:13]: carbs tuning metric, etc.
SWYX [00:59:15]: Like we care about
JOSH [00:59:15]: the low scale version
SWYX [00:59:17]: of this as well.
JOSH [00:59:17]: We want smaller scale models to be able to do something on this. And so that's kind of the focus for this.
SWYX [00:59:22]: And hopefully this is more
JOSH [00:59:22]: useful for other people. But yes,
SWYX [00:59:25]: those other questions
JOSH [00:59:25]: are also quite interesting. They get a lot harder to evaluate, like, is this a good architecture or not? Like you and I could probably debate for a while on, you know, different architectures. And so it becomes a lot trickier to do these evaluations as they become more realistic. So I think that's one of the things that we've been playing around with a lot, especially around like code generation.
SWYX [00:59:44]: So if you're saying,
JOSH [00:59:44]: you know, implement this function, okay, it can be kind of objective, but, you know, even MBPP, we've made our own internal version of this data set, right?
SWYX [00:59:52]: Where we've taken like
JOSH [00:59:52]: every single example
SWYX [00:59:54]: and looked at it and been like,
JOSH [00:59:54]: does this actually make sense? Like, what is the type signature? Like, can we remove all ambiguity, et cetera?
SWYX [01:00:00]: So you basically like reviewed every single question on, I mean, that's impossible for like HelloSwag, right? Yeah, yeah.
JOSH [01:00:05]: We didn't do that for HelloSwag, but this is for MBPP, which is only like a few hundred. So we just sat down and did it. Yeah.
JONATHAN [01:00:12]: I'm so excited to get to look at this data set. Like this is such a resource for the community. I absolutely can't wait. We should probably do the,
JOSH [01:00:19]: I don't know. I don't know if we were planning on doing the healed MBPP one,
SWYX [01:00:23]: but hopefully we can do
JOSH [01:00:23]: that one in the future. Did you look at SweetBench?
SWYX [01:00:26]: It's the sort of hot new data set of the summer.
JOSH [01:00:28]: Yeah, I've taken a quick look
SWYX [01:00:29]: at SweetBench.
JOSH [01:00:29]: It's really interesting. I like that it's a much more difficult kind of coding, code related task for bug fixing. I think it gets into some of these problems where it is a lot harder to evaluate these things once they get more realistic. Like we were looking at the AgentBench paper, I think just last week for our paper club and one of the things
SWYX [01:00:49]: that we noticed
JOSH [01:00:49]: is that actually like both of the examples in the appendix that are given as like traces where it got it right. This is actually not the right solution. And it's OK. You know, it's fine. Like it did make it past the test. That's what the metric is.
SWYX [01:01:02]: That's what the benchmark
JOSH [01:01:02]: is about, right? But like it just said,
SWYX [01:01:05]: you know, like,
JOSH [01:01:05]: you know, dot encode ASCII. Like, well, that's not the right way to do this. Like it just dropped all the other edge cases that you actually would have cared about in production for this thing.
SWYX [01:01:14]: And there is like
JOSH [01:01:14]: a better way of doing it.
SWYX [01:01:16]: And you know,
JOSH [01:01:16]: that's what the real golden patch was. But, you know, that's OK. But then how do you test all of that?
SWYX [01:01:21]: Like as you start to do
JOSH [01:01:21]: more realistic things, the test coverage, like getting test coverage over all possible ways of solving these bugs is really hard. Evaluation is the single
JONATHAN [01:01:28]: hardest part of the whole thing. Like I spend a shocking amount of time just telling our customers
SWYX [01:01:34]: we need to find a way
JONATHAN [01:01:34]: to measure what you actually want out of the model before you should ever touch a GPU. And, you know, trying to convince my team and me to follow our own advice a lot of the time on that. And I think everybody like on the one hand,
SWYX [01:01:46]: it's easy to laugh
JONATHAN [01:01:46]: at the state of the evaluations that we have. None of them are good. Like if you go read these eval benchmarks, you'll always come away
SWYX [01:01:52]: disappointed.
JONATHAN [01:01:53]: And yet they've given us useful hills to climb. And we do seem to be making progress and measuring
SWYX [01:01:58]: progress in the field.
JONATHAN [01:01:58]: And I think anecdotally, models are getting better year to year. So I feel like people tend to go and get into one situation or the other, like evals don't matter. I'm just going to look at loss
SWYX [01:02:07]: or like, you know,
JONATHAN [01:02:08]: the evals matter a lot and they're all broken. So what do I do? And I think like a lot of things in deep learning, we have to make peace with just complete imperfection. Like the most successful scientists I see are the ones who are OK operating in a world
SWYX [01:02:20]: where everything's
JONATHAN [01:02:20]: going to be broken.
SWYX [01:02:22]: And yet we can still
JONATHAN [01:02:22]: cobble things together and make something
SWYX [01:02:24]: interesting happen.
JONATHAN [01:02:24]: I mean, we were just discussing that with literal infrastructure. And now we're all the way
SWYX [01:02:28]: up to like,
JONATHAN [01:02:28]: how do we measure whether a model performed a complex coding task correctly? And everything is broken.
SWYX [01:02:34]: And yet we're still able
JONATHAN [01:02:34]: to make huge amounts of forward progress.
SWYX [01:02:36]: I think that's right, Jonathan.
JOSH [01:02:38]: And that the challenge
SWYX [01:02:40]: isn't necessarily
JOSH [01:02:40]: making perfect evaluations. I think our blog post here is about going really into the weeds on these to figure out like, what does that look like? And I think one thing is like, you know,
SWYX [01:02:49]: as you said,
JOSH [01:02:49]: we have been able to make a lot of progress without making these perfect.
SWYX [01:02:52]: That's great.
JOSH [01:02:52]: You don't have to have perfect evaluations. And, you know, the more interesting work is the stuff that we can't necessarily publish about, which is the imperfect evaluations that we have for actual coding tasks, for example.
SWYX [01:03:04]: Like, what does this
JOSH [01:03:04]: really mean as a person? And there, as you said, it's much messier.
SWYX [01:03:08]: So it's a lot harder
JOSH [01:03:08]: to put it out and say like, hey, everybody use this because there's so many
SWYX [01:03:12]: rough edges.
JOSH [01:03:12]: It's so hard to like even say, oh, is this even the right task? Is this even the right way to do it? And there's a lot of judgment.
SWYX [01:03:19]: There's a lot of intuition
JOSH [01:03:19]: that it comes down to. But yeah, I think that's where it's critical to do
SWYX [01:03:23]: if you actually want to
JOSH [01:03:23]: make these systems work.
JONATHAN [01:03:24]: Yeah, you have to make peace with with living in that in between.
SWYX [01:03:28]: Yeah.
JONATHAN [01:03:28]: And I think that in some sense,
SWYX [01:03:30]: when I hire researchers,
JONATHAN [01:03:30]: that's the number one quality I look for. Like, can they be at peace living in a house that is neither clean nor messy,
SWYX [01:03:36]: but it's just kind of
JONATHAN [01:03:36]: somewhere in between? And are they OK with that? Are they OK with a few dishes being out on the table and a few clothes
SWYX [01:03:42]: being on the floor?
JONATHAN [01:03:43]: Or will that drive them insane? Or will they just end up with all the clothes on the floor and like all the dishes out all the time? Like, it's kind of I'm looking for that perfect balance because, you know, we have to operate in this imperfect world. Like, yeah, go ahead and give me the perfect evaluation for programmers
SWYX [01:03:58]: or for an LLM
JONATHAN [01:03:58]: that is a program assistant tool. Like there is no perfect evaluation. But clearly we've made progress. And so the most important part
SWYX [01:04:06]: is just are we
JONATHAN [01:04:06]: climbing the right hills? And so this is why I'm so excited to see the ambiguity aspect of this. We often think we have more room to climb on these benchmarks. It turns out we don't. Or it turns out that actually we're climbing, getting good at the benchmark and not actually getting good at the task we care about underlying the benchmark anymore.
SWYX [01:04:21]: Maybe the model,
JONATHAN [01:04:21]: like this is the famous example where if you get 100% at MNIST, your model must be broken in some way because there are four examples mislabeled, you know, it's it's that all over again. Welcome to this.
SWYX [01:04:33]: Yeah, it's the accidental canary canary in this. I think one thing that's
JOSH [01:04:37]: actually really interesting about this also is that, yes, like the ambiguous examples are sort of, you know, not that great from the perspective of these particular tasks that we're evaluating.
SWYX [01:04:46]: But actually, one thing
JOSH [01:04:46]: that we're very interested in is ambiguity itself. Like, can we detect whether a task from a user is ambiguous or whether you've, you know, completed a task successfully? Like these are actually hard, messy problems, but are really important from like the user experience of using these models. I would much rather have a coding agent that will give me back a thing. And, you know, it's it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me 100% of the time like when it's not sure. Like that's so much more useful if it can communicate like, I'm not really sure about this or maybe there's some errors here. Then just like, here's some code. I have no idea if it works. And so these kind of like, you know, detecting ambiguity and detecting correctness
SWYX [01:05:25]: or uncertainty,
JOSH [01:05:25]: I think are really interesting problems
SWYX [01:05:27]: that we're really like
JOSH [01:05:27]: digging into quite deeply.
SWYX [01:05:29]: I want to touch on maybe a couple of hot topics in evals, maybe tangentially related, but we're on the evals train right now. So I'm just going to get on that. So ArcAGI, Francois Chollet's hot new thing, it's sort of my take on it is basically it's trying to measure reasoning through an abstract IQ test. Effectively, I noticed that you don't use it. There's a lot of community debate, pro and con about it. What are your thoughts on just more abstract reasoning and maybe ArcAGI specifically?
JOSH [01:06:01]: I think we purposely stayed away from the very, like there's BigBench, for example, that has a lot of, I think, to me, feels sort of similar types of tasks that are like very unrealistic. Like, oh, you know, we have books of different colors and then you're going to shuffle them and like which book is furthest to the left or something like, OK, cool, I guess it's neat. It's neat, I think, for us to explore in terms of like an agent reasoning in a larger loop. And we do care about these types of evaluations there. The types of evaluations we're talking about in the blog post here are for getting at, like, does this model in a base model sense, is this working at all? There's no chain of thought in these evaluations. These are just like, go straight to the answer. Does this make sense?
SWYX [01:06:42]: Like, is this a thing that
JOSH [01:06:42]: you can answer very quickly? That's what we were selecting for with these evaluations. This is not to say that these are the only evaluations we have. I think the Arc ones are like a little bit too, probably, visual for us to really be able to integrate with.
SWYX [01:06:56]: But I think some of the
JOSH [01:06:56]: BigBench ones are... You can tokenize it.
SWYX [01:06:59]: Yeah, but, you know,
JOSH [01:07:00]: I think it's not really... I think you can spend a lot of time getting really good at these kinds of benchmarks without making, like, kind of more general purpose progress. And so I think we're a little bit leery of going too far in that direction. Similarly, like, coding competitions. Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones.
SWYX [01:07:20]: So I think you can go
JOSH [01:07:20]: very far down that route
SWYX [01:07:22]: and make something that's, like,
JOSH [01:07:22]: really good at those problems, but not actually that useful as, like, a programmer day to day.
SWYX [01:07:26]: Yeah.
JONATHAN [01:07:27]: Take a different tactic, which is, like, at the end of the day at Databricks, I have 12,000 customers, or I think that's the latest number, all of whom are trying to do something with, you know, LLMs or AI or machine learning. And those things don't look like these tasks. I don't think I have a single customer that's asking to, you know, have AI solve abstract reasoning problems. Things are pretty, like, they can be ambiguous,
SWYX [01:07:53]: they can be challenging,
JONATHAN [01:07:53]: they can be really interesting,
SWYX [01:07:55]: but none of them look quite like this.
JONATHAN [01:07:56]: And so, you know, I think to Josh's point, like, it's really about asking, why are we doing this? Even if you're trying to build AGI, and that's not personally my purpose, and I, you know, Josh has much more interesting things to say about that than I do. I don't even know if this is the kind of intelligence I would get excited about or care about personally, or if I would consider, you know, to Josh's point, this to be the indicia of intelligence.
SWYX [01:08:17]: It's neat.
JONATHAN [01:08:17]: But, you know, for me, it's, like, more down to earth things, like having a model that can have a conversation with you about data
SWYX [01:08:24]: that on the backend
JONATHAN [01:08:24]: is running SQL queries on your literal data. That's a much more interesting task to me. That's something that really matters day to day for my customers and, you know, different perspectives, but, you know, I think Josh and I would probably say the same thing,
SWYX [01:08:36]: even though I would,
JONATHAN [01:08:36]: I'm guessing, I don't want to put words in your mouth. You would say that you're pursuing more general intelligence in your own way. And I would say that I'm very happy with narrow intelligence. Like, I'm very happy with my little SQL bot and building 12,000 of those because they're moving the needle for a lot of folks every day.
JOSH [01:08:51]: Yeah, I think we're, you know, we're not as far away in our position as it might seem. I think we're also excited about, like,
SWYX [01:08:58]: how do you actually
JOSH [01:08:58]: make these things useful? And that does end up being pretty narrow. I think these other tasks can be interesting as, like, ways to explore these more abstract reasoning questions or like, OK, how could an agent actually work through this? But it's important to keep in mind that it's like a toy, not a real problem. It's like it's a scientific tool to tell us something about the models.
SWYX [01:09:16]: It's not something we should
JOSH [01:09:16]: be optimizing for necessarily.
SWYX [01:09:18]: The one thing I'll point out is, you know, as a kid, I was graded into a gifted program based on my ability to solve these exact type of problems. And then I entered college based on my ability to solve SATs, which, again, have nothing to do with my college experience, but whatever. So, you know, we have a history in the humanity of doing correlated IQ tests to general capability. OK, so the two more, two more viral evals, and then, you know, I just want to be mindful of your time. Needle in a haystack, long context utilization. Oh, for the love of God. Something, well, OK, like, let's just assume that, you know, on our podcast, we've discussed the, you know, baseline problems with needle in a haystack, but just generally long context, right? It's a useful thing for agents. I assume. And it's something that, you know, it's out there. Like, we don't know, don't really know what the best way to utilize memory is. But like, I assume it's important, right? What I'll say is like, you know,
JONATHAN [01:10:13]: I spend a lot of time thinking about RAG these days. And RAG, you know, in one sense, you know, the way that I think about RAG is it's the world's simplest agent. It is an agent that basically, you know, there's at least more than one thing happening in the process of building models, at least a system. If you give the model the ability to decide when it wants to retrieve data from a context or retrieve data from a database, then we're talking about an agent. So RAG kind of, I think, like toes that boundary really nicely. There are a lot of reasons why you do genuinely need a long context. Like, I don't think long contexts are problematic in and of themselves. I know there's some controversy even about that. I love the idea of doing like thousand shot tasks as an alternative to fine tuning. I love the idea of pulling in lots of data into the context. I love the idea of once you get in a multimodal land, you're just going to end up
SWYX [01:10:54]: with giant context.
JONATHAN [01:10:54]: It's kind of unavoidable. The flip side is I don't know of anyone who like is hiding a secret passphrase in a book and needs the model to find it. Needle in a haystack is, it's interesting. The challenge with long context to my mind, and Josh,
SWYX [01:11:08]: I'm curious what you think,
JONATHAN [01:11:08]: is simply that annotating long context evals is really hard and really expensive, you know, intrinsically, because you need someone to read 10,000 tokens or 100,000 tokens, or like you need someone to read a 1,000 page book or the equivalent thereof in order to measure those long context benchmarks. I don't know if a human could solve these tasks, let alone that a human could do this in any amount of time where you're willing to pay the money to get the data annotated. And so any long context eval
SWYX [01:11:33]: has to, in some sense,
JONATHAN [01:11:33]: be correct by construction. And you have to, you know, the, you have to know the answer before you've created the example. And needle in a haystack is kind of the simplest way
SWYX [01:11:41]: of doing that.
JONATHAN [01:11:41]: I think the problems of needle in a haystack are well known, you know, it doesn't measure anything real. You're not even testing the model's ability to holistically use the context just to identify one part of the context. So you can do some wacky things to your model, like quantize the hell out of the KV cache and still get needle in a haystack to work quite well because it's not trying to holistically take advantage
SWYX [01:11:59]: of things.
JONATHAN [01:12:00]: You know, I have some thoughts on things that I like more that are also still correct by construction. Like, I really like the idea of doing thousand shot tasks where you can look at the scaling as you go from 10 shot to 100 shot to thousand shot to fine tuning on that data instead. And I like that as a way to, you know, have something that's correct by construction, or at least where you have
SWYX [01:12:19]: a nice baseline
JONATHAN [01:12:19]: that you can compare to automatically. So I'm typically looking for like contexts that are situations where long context is one way to solve the task, but not the only way
SWYX [01:12:28]: to solve the task.
JONATHAN [01:12:28]: And we have some other strong baseline floating around personally. But yeah, needle in a haystack, not my favorite thing in the world, to say the least.
JOSH [01:12:35]: Yeah, I mean, I agree with most of what Jonathan
SWYX [01:12:38]: said, I think.
JOSH [01:12:38]: I think one other thing that I will call out
SWYX [01:12:40]: is that, you know,
JOSH [01:12:40]: from like a coding application perspective, it's useful to have long context because the lazy thing of just like throw the whole repo in the context is like,
SWYX [01:12:48]: OK, cool.
JOSH [01:12:48]: Like, you know, you can just get started with that. But then in, you know, in real scenarios, you don't necessarily want to put the whole thing in there. You can have code bases
SWYX [01:12:56]: that are bigger.
JOSH [01:12:56]: You probably want to filter down to the stuff that's relevant anyway to not be confusing. Like you probably even if you did have a lot of context,
SWYX [01:13:02]: you might want to sort it
JOSH [01:13:02]: in some way to say this is more important than this other stuff. So and, you know, you don't want to wait for you don't want to be wasting all this time and compute
SWYX [01:13:09]: on inference and like
JOSH [01:13:09]: doesn't really matter. So, yeah, I don't know that it's the most important thing.
SWYX [01:13:15]: I think people will find creative use cases. And like Jon said, I think the multimodality examples will naturally lend themselves to long context. Cool. And then one last one on just general sort of agent related capabilities that we didn't really talk about in the eval section is function calling and tool use. There's a recent trend, I think, basically led again by OpenAI on parallel function calling. There's always there's been a limit on how many tools you can call from four to now, I think, 128. And I think theoretically, Claude and Jem and I support a lot more.
JOSH [01:13:49]: So just generally,
SWYX [01:13:50]: how do you think about evaling tool use? Is that super important for you guys? We're thinking about it
JOSH [01:13:55]: in a slightly different way, which is, yes, you can have this like hard coded list of tools. But if only you could have like this really large open set of like tools, maybe they would be like functions that you could call if only there was like a language or like a programming thing, like being able to write code. I think for us, it's like, well, look, if we can write code, like now you have all these tools accessible at the end of the day,
SWYX [01:14:16]: like function calling
JOSH [01:14:16]: is just a function invocation, like literally in code. I think our approach to this is like
SWYX [01:14:21]: instead of worrying about
JOSH [01:14:21]: like weird hard coded agents using tools, like let's just make them
SWYX [01:14:25]: able to actually
JOSH [01:14:25]: write code robustly and make that code work and be able to debug that code, know if that code is safe to run, like get really good at the like code writing and execution part of things, because that will open up the action space like far more than, you know, 128 tools, like just everything is at your fingertips, especially I think over the next few years, like we already have so many really good APIs. As we get better and better at writing code, we'll be able to make APIs to things that don't even have APIs today. That's kind of how we think about it is less as like a special purpose thing
SWYX [01:14:52]: and more as like
JOSH [01:14:52]: this is one of the reasons to focus on code.
SWYX [01:14:55]: On my end,
JONATHAN [01:14:55]: the way that I think about this is, you know, I think a lot about how models interact with data.
SWYX [01:15:00]: And so for me,
JONATHAN [01:15:00]: tool use is really a question of how do you take models
SWYX [01:15:04]: that are really built
JONATHAN [01:15:04]: for unstructured data
SWYX [01:15:06]: and have them interact
JONATHAN [01:15:06]: with structured data? So, you know, and I get the question a lot from my customers,
SWYX [01:15:10]: like what do I do
JONATHAN [01:15:10]: with tabular data? Or what do I do with like, you know, JSON? Or what do I do? I mean, you name it, like even what do I do
SWYX [01:15:17]: with a PDF?
JONATHAN [01:15:17]: Because PDF parsing is still an unsolved problem, even in 2024. And the answer, or even just the basic question
SWYX [01:15:24]: of like, should I bother
JONATHAN [01:15:24]: to structure my data anymore? Shouldn't I just toss the table? Shouldn't I flatten it
SWYX [01:15:28]: and just throw it
JONATHAN [01:15:28]: into the LLM context and like let the model
SWYX [01:15:30]: figure it out?
JONATHAN [01:15:30]: Answer is no. We've built all these fun APIs and fun languages
SWYX [01:15:36]: and paradigms
JONATHAN [01:15:36]: for dealing with structured data over the years. Just use them.
SWYX [01:15:40]: Have your model use them.
JONATHAN [01:15:40]: Train a model that can interact
SWYX [01:15:42]: with these things
JONATHAN [01:15:42]: in a meaningful way. Like text to SQL
SWYX [01:15:45]: is still,
JONATHAN [01:15:45]: or like having a model be able to make SQL calls in the backend is actually like one of the single
SWYX [01:15:51]: most useful things
JONATHAN [01:15:51]: for my customers. It sounds really boring. Models are really good at it. And it moves the needle day to day.
SWYX [01:15:57]: So tool use for me
JONATHAN [01:15:58]: really is that like, how do you just interact with structured data sources and take advantage of the fact that you have some
SWYX [01:16:05]: prior knowledge
JONATHAN [01:16:05]: about the structure of your data that an LLM would completely flatten away. In many ways, this is kind of one of the, one of my biggest frustrations with the fact that LLMs work well with code. We have decades and decades and decades
SWYX [01:16:17]: of understanding
JONATHAN [01:16:17]: about the structure and interpretation of programs. Like I think that's literally the name of a book on programming, if I remember right. And, you know, we have all this theory. We know everything there is to know about programming languages if they're well-formed languages and have the right properties. And yet when we have an LLM
SWYX [01:16:31]: work with them,
JONATHAN [01:16:31]: we literally just turn it into a token stream.
SWYX [01:16:33]: Despite the fact that we know
JONATHAN [01:16:34]: how to parse it. We know, you know, how to do all sorts of, you know,
SWYX [01:16:38]: reference, you know,
JONATHAN [01:16:38]: disambiguation and things like that. We're still just flattening it into a model and making the model relearn all of these things from scratch. And it frustrates
SWYX [01:16:45]: the hell out of me.
JONATHAN [01:16:45]: I don't have a better answer when it comes to code, but I really appreciate that with a lot of data sources that have structure to them. Tool uses and function calling
SWYX [01:16:53]: are just,
JONATHAN [01:16:53]: in my mind,
SWYX [01:16:55]: So I think basically what you're saying is like code is the God tool for Jonathan. Like, you know, SQL is so much the right abstraction for accessing all this data. One thing I do spend a lot of time thinking about is for the stuff that doesn't fit in a SQL table, you know, is knowledge graphs the answer? I think a lot of people are exploring that and I think every now and then people get a bout of knowledge graph religion and then it kind of doesn't work out. So I wonder, I wonder what the end state is. Like, is this an idea where it's a mirage? Or is this the idea where it sometime is going to work? It's about having the right tools
JOSH [01:17:27]: for the problems, right? Like as Jonathan was saying, SQL is sometimes definitely the right tool. Like you've got your, you know, order table or something and you want to know, you know, number of sales last month. Like you should be using SQL sum that column. OK, great. You're all set. Knowledge graphs also,
SWYX [01:17:40]: you know,
JOSH [01:17:40]: are sometimes the right tool for a particular problem. You have some like weird question about relationships between entities
SWYX [01:17:46]: that are modeled
JOSH [01:17:46]: on some particular ontology that you actually understand and it's like math to the real world. Great. Use a knowledge base. Like use a knowledge graph. This is fine. But I think in the real world, it gets a lot messier than like knowledge graph style of things where it's like, well, is there a relationship between these two nodes? Like, I don't know.
SWYX [01:18:04]: Like, is are these
JOSH [01:18:04]: two separate nodes? Like those kind of messy borders, I think, prevent it
SWYX [01:18:08]: from being a tool
JOSH [01:18:08]: that can like solve everything forever. And so I think it'll always be good for certain problems, just like SQL is good
SWYX [01:18:14]: for certain problems.
JOSH [01:18:14]: Like different abstractions are good for different problems. And yeah, I think this is why I'm excited about code. Like code lets you
SWYX [01:18:20]: kind of pick the right,
JOSH [01:18:20]: like let's use this library for this problem.
SWYX [01:18:22]: Let's use this library
JOSH [01:18:22]: for this other problem.
JONATHAN [01:18:24]: I think Josh said it and you said it well, like code is kind of the God tool. It unlocks literally everything. The challenge for me is always like,
SWYX [01:18:31]: you know, sometimes
JONATHAN [01:18:31]: unlocking too much power can sometimes inconvenient things can happen. And so it's all about balancing that
SWYX [01:18:37]: in some sense,
JONATHAN [01:18:37]: language is the God tool.
SWYX [01:18:39]: If only, you know,
JONATHAN [01:18:39]: we knew how to interpret it all the time. So code is has the really nice property
SWYX [01:18:44]: that at least you can
JONATHAN [01:18:44]: always execute it. And sometimes you just literally want your model to be able to do SQL calls and nothing else. And setting those boundaries properly for the problem,
SWYX [01:18:52]: I think is going to be, I think at least a lot of my customers
JONATHAN [01:18:54]: are going to be thinking very hard about that.
SWYX [01:18:56]: Like, should I give
JONATHAN [01:18:56]: the model access to the web?
SWYX [01:18:58]: Is that actually helpful
JONATHAN [01:18:58]: for this problem? It sounds great to just like flip yes on all the tools.
SWYX [01:19:02]: Is that actually going to mean
JONATHAN [01:19:02]: I'm going to get better solutions to my problems?
SWYX [01:19:04]: So I want to be mindful of time. I think that's basically our sort of recap of our discussion based on Imbue's releases today. I wanted to leave some time for what's next for both of you guys. Maybe Josh, as a guest of honor, you want to go first as to what happens next.
JOSH [01:19:19]: We have these releases. We're happy to put these things out. I think there's a lot of stuff
SWYX [01:19:22]: that we haven't released.
JOSH [01:19:22]: Like, this is not the only thing we've been working on. Most of our actual focus has been on kind of coding and reasoning. In particular, like the things that we're excited about are can we make these things useful? Like Jonathan is saying, right? Like, it's not about toy problems. It's like, can we use these today in our day-to-day workflow and actually have them accelerate us? And I think we have some kind of internal product prototypes and things that we're excited about. And so we're excited to share more about this in the coming, you know, months to quarters as we get it to a place where like other people could maybe get value out of this as well. But that's kind of our real focus right now is like, how do you take these really cool capabilities that are out there that our models have, et cetera. And like, make sure that they're actually useful today for us, like when we're doing real work and then for other people as well. In particular, focused on generating code, understanding code, testing code, verifying it, like starting with the like robust creation of software. Excellent.
SWYX [01:20:13]: Jonathan?
JONATHAN [01:20:14]: I never like to talk too much about the future because I think you've heard this from me before. I like for us to speak
SWYX [01:20:19]: through our work.
JONATHAN [01:20:19]: And so I don't, I don't like to tease too much. Our mission is, to Josh's point, to make this stuff useful to 12,000 customers. And not a lot of that ends up making it into the public eye
SWYX [01:20:30]: and not a lot of that
JONATHAN [01:20:30]: ends up getting released open source. So for this kind of forum where really, you know,
SWYX [01:20:34]: where we're talking
JONATHAN [01:20:34]: to the community, I'm asking myself right now, like, you know, what exciting things
SWYX [01:20:38]: are we going to have
JONATHAN [01:20:38]: to offer the community in the next little while? I think the most exciting part is just we're writing a lot of blog posts right now. We're trying to share more and more of our science because I feel like
SWYX [01:20:47]: we've been doing
JONATHAN [01:20:47]: these big pushes to create these really giant models.
SWYX [01:20:50]: I think, Josh,
JONATHAN [01:20:50]: I'm sure you had
SWYX [01:20:51]: the same experience.
JONATHAN [01:20:51]: It's exhausting and all-consuming and you get to the end
SWYX [01:20:54]: and you're like,
JONATHAN [01:20:54]: oh, I have all this stuff
SWYX [01:20:56]: I want to talk about.
JONATHAN [01:20:56]: Now I need to find the time to talk about it now that I've survived this huge push. And we're definitely in that mode right now. So there's going to be a lot of that coming in in the next little while. And, you know, we're always cooking up fun new models. I think the real question is, you know, releasing models open source is not our day-to-day bread and butter. It's kind of a fun reward that we get to do sometimes when we have something really cool to share and a little bit of time and spare GPUs in our hands. But for the most part,
SWYX [01:21:20]: everything is going
JONATHAN [01:21:20]: toward customers. You know, I think the joke is Databricks has been 18 months away from IPO for five years. So I guess Databricks
SWYX [01:21:26]: is 18 months away
JONATHAN [01:21:26]: from IPO still. But 18 months away from IPO means there's a lot of pressure to deliver for customers. And we're going to keep working on that. But I think you'll see hopefully some cool, interesting things
SWYX [01:21:36]: get dropped over the course
JONATHAN [01:21:36]: of the summer and into the fall. We'll find out when we get there.
SWYX [01:21:39]: I think that's the right way
JONATHAN [01:21:39]: to put it. I know we were talking earlier about kind of Abracadabra and Alakazam. And all I'll say is that, you know, the DBRX small model that we still haven't released yet was called Abra. DBRX was called Kadabra. And there's a third Pokémon in that evolution. And that's all I'll say for now. Cool stuff kind of popping up sometimes on Chatbot Arena. And, you know, keep your eyes out. Yep.
SWYX [01:21:59]: I'll leave the links and the hints in the show notes. That was a very fun way to leave some breadcrumbs for people to follow. Cool. I'll leave everything to sort of some calls to action. We're going to be releasing this next week. So I'll be deep in my conference, the AI Engineer World's Fair. So people can just go to AI.Engineer and livestream it. Do you guys have any other calls to action before you wrap?
JOSH [01:22:20]: The only one is, you know, we're definitely hiring. So if you're interested in working on coding, reasoning, interested in working on all of this stuff, you know, from the ground up and really deeply understanding not just how does the hardware work, but how do the models work and also designing these, you know, systems to actually be useful for yourself day to day, come say hi.
JONATHAN [01:22:36]: The only thing I'll say is, you know, and I like saying it these days, it feels like the field is so crowded and, you know, it requires so many resources to do impactful work. And, you know, on some days it feels like everything's been done or somebody else is doing everything before you can. At least I remember that feeling every single day of my PhD and even more so now. But I hope like what you heard from Josh today tells you there is so much enormously impactful work to do in the field. If only you take a step back and take a fresh look at some of these things and just talk about what you're doing. There's a huge amount left to do here and a huge amount of exciting work happening every day. And for those who are certainly feeling that exhaustion right now, and I count myself among those folks many days, it's refreshing to see these kinds of drops and see that there is so much more even in things that people feel like they understand how to set up a cluster. My God, you know, even in these evals that we think we understand, there is still more to understand and still more work to do. I hope everybody's keeping at it.
SWYX [01:23:32]: All right. Keep on keeping on. Well, thanks so much for your time, guys. That was a great discussion and we'll put the links in the show notes for people to read more. Thanks. Thanks a bunch.
JOSH [01:23:40]: Thank you so much.
The World’s Fair is officially sold out! Thanks for all the support and stay tuned for recaps of all the great goings on in this very special celebration of the AI Engineer!
Longtime listeners will remember the fan favorite Raza Habib, CEO of HumanLoop, on the pod:
Well, he’s caught the podcasting bug and is now flipping the tables on swyx!
Subscribe to High Agency wherever the finest Artificial Intelligence podcast are sold.
High Agency Pod Description
In this episode, I chatted with Shawn Wang about his upcoming AI engineering conference and what an AI engineer really is. It's been a year since he penned the viral essay "Rise of the AI Engineer' and we discuss if this new role will be enduring, the make up of the optimal AI team and trends in machine learning.
Timestamps
00:00 - Introduction and background on Shawn Wang (Swyx)03:45 - Reflecting on the "Rise of the AI Engineer" essay07:30 - Skills and characteristics of AI Engineers12:15 - Team composition for AI products16:30 - Vertical vs. horizontal AI startups23:00 - Advice for AI product creators and leaders28:15 - Tools and buying vs. building for AI products33:30 - Key trends in AI research and development41:00 - Closing thoughts and information on the AI Engineer World Fair Summit
Video
Editor’s note: One of the top reasons we have hundreds of companies and thousands of AI Engineers joining the World’s Fair next week is, apart from discussing technology and being present for the big launches planned, to hire and be hired!
Listeners loved our previous Elicit episode and were so glad to welcome 2 more members of Elicit back for a guest post (and bonus podcast) on how they think through hiring. Don’t miss their AI engineer job description, and template which you can use to create your own hiring plan!
How to Hire AI Engineers
James Brady, Head of Engineering @ Elicit (ex Spring, Square, Trigger.io, IBM)
Adam Wiggins, Internal Journalist @ Elicit (Cofounder Ink & Switch and Heroku)
If you’re leading a team that uses AI in your product in some way, you probably need to hire AI engineers. As defined in this article, that’s someone with conventional engineering skills in addition to knowledge of language models and prompt engineering, without being a full-fledged Machine Learning expert.
But how do you hire someone with this skillset? At Elicit we’ve been applying machine learning to reasoning tools since 2018, and our technical team is a mix of ML experts and what we can now call AI engineers. This article will cover our process from job description through interviewing. (You can also flip the perspectives here and use it just as easily for how to get hired as an AI engineer!)
My own journey
Before getting into the brass tacks, I want to share my journey to becoming an AI engineer.
Up until a few years ago, I was happily working my job as an engineering manager of a big team at a late-stage startup. Like many, I was tracking the rapid increase in AI capabilities stemming from the deep learning revolution, but it was the release of GPT-3 in 2020 which was the watershed moment. At the time, we were all blown away by how the model could string together coherent sentences on demand. (Oh how far we’ve come since then!)
I’d been a professional software engineer for nearly 15 years—enough to have experienced one or two technology cycles—but I could see this was something categorically new. I found this simultaneously exciting and somewhat disconcerting. I knew I wanted to dive into this world, but it seemed like the only path was going back to school for a master’s degree in Machine Learning. I started talking with my boss about options for taking a sabbatical or doing a part-time distance learning degree.
In 2021, I instead decided to launch a startup focused on productizing new research ideas on ML interpretability. It was through that process that I reached out to Andreas—a leading ML researcher and founder of Elicit—to see if he would be an advisor. Over the next few months, I learned more about Elicit: that they were trying to apply these fascinating technologies to the real-world problems of science, and with a business model that aligned it with safety goals. I realized that I was way more excited about Elicit than I was about my own startup ideas, and wrote about my motivations at the time.
Three years later, it’s clear this was a seismic shift in my career on the scale of when I chose to leave my comfy engineering job at IBM to go through the Y Combinator program back in 2008. Working with this new breed of technology has been more intellectually stimulating, challenging, and rewarding than I could have imagined.
Deep ML expertise not required
It’s important to note that AI engineers are not ML experts, nor is that their best contribution to a tech team.
In our article Living documents as an AI UX pattern, we wrote:
It’s easy to think that AI advancements are all about training and applying new models, and certainly this is a huge part of our work in the ML team at Elicit. But those of us working in the UX part of the team believe that we have a big contribution to make in how AI is applied to end-user problems.
We think of LLMs as a new medium to work with, one that we’ve barely begun to grasp the contours of. New computing mediums like GUIs in the 1980s, web/cloud in the 90s and 2000s, and multitouch smartphones in the 2000s/2010s opened a whole new era of engineering and design practices. So too will LLMs open new frontiers for our work in the coming decade.
To compare to the early era of mobile development: great iOS developers didn’t require a detailed understanding of the physics of capacitive touchscreens. But they did need to know the capabilities and limitations of a multi-touch screen, the constrained CPU and storage available, the context in which the user is using it (very different from a webpage or desktop computer), etc.
In the same way, an AI engineer needs to work with LLMs as a medium that is fundamentally different from other compute mediums. That means an interest in the ML side of things, whether through their own self-study, tinkering with prompts and model fine-tuning, or following along in #llm-paper-club. But this understanding is so that they can work with the medium effectively versus, say, spending their days training new models.
Language models as a chaotic medium
So if we’re not expecting deep ML expertise from AI engineers, what are we expecting? This brings us to what makes LLMs different.
We’ll assume already that our ideal candidate is already inspired by, and full of ideas about, all the new capabilities AI can bring to software products.
But the flip side is all the things that make this new medium difficult to work with. LLM calls are annoying due to high latency (measured in tens of seconds sometimes, rather than milliseconds), extreme variance on latency, high error rates even under normal operation. Not to mention getting extremely different answers to the same prompt provided to the same model on two subsequent calls!
The net effect is that an AI engineer, even working at the application development level, needs to have a skillset comparable to distributed systems engineering. Handling errors, retries, asynchronous calls, streaming responses, parallelizing and recombining model calls, the halting problem, and fallbacks are just some of the day-in-the-life of an AI engineer. Chaos engineering gets new life in the era of AI.
Skills and qualities in candidates
Let’s put together what we don’t need (deep ML expertise) with what we do (work with capabilities and limitations of the medium). Thus we start to see what Elicit looks for in AI engineers:
* Conventional software engineering skills. Especially back-end engineering on complex, data-intensive applications.
* Professional, real-world experience with applications at scale.
* Deep, hands-on experience across a few back-end web frameworks.
* Light devops and an understanding of infrastructure best practices.
* Queues, message buses, event-driven and serverless architectures, … there’s no single “correct” approach, but having a deep toolbox to draw from is very important.
* A genuine curiosity and enthusiasm for the capabilities of language models.
* One or more serious projects (side projects are fine) of using them in interesting ways on a unique domain.
* …ideally with some level of factored cognition, e.g. breaking the problem down into chunks, making thoughtful decisions about which things to push to the language model and which stay within the realm of conventional heuristics and compute capabilities.
* Personal studying with resources like Elicit’s ML reading list. Part of the role is collaborating with the ML engineers and researchers on our team. To do so, the candidate needs to “speak their language” somewhat, just as a mobile engineer needs some familiarity with backends in order to collaborate effectively on API creation with backend engineers.
* An understanding of the challenges that come along with working with large models (high latency, variance, etc.) leading to a defensive, fault-first mindset.
* Careful and principled handling of error cases, asynchronous code (and ability to reason about and debug it), streaming data, caching, logging and analytics for understanding behavior in production.
* This is a similar mindset that one can develop working on conventional apps which are complex, data-intensive, or large-scale apps. The difference is that an AI engineer will need this mindset even when working on relatively small scales!
On net, a great AI engineer will combine two seemingly contrasting perspectives: knowledge of, and a sense of wonder for, the capabilities of modern ML models; but also the understanding that this is a difficult and imperfect foundation, and the willingness to build resilient and performant systems on top of it.
Here’s the resulting AI engineer job description for Elicit. And here’s a template that you can borrow from for writing your own JD.
Hiring process
Once you know what you’re looking for in an AI engineer, the process is not too different from other technical roles. Here’s how we do it, broken down into two stages: sourcing and interviewing.
Sourcing
We’re primarily looking for people with (1) a familiarity with and interest in ML, and (2) proven experience building complex systems using web technologies. The former is important for culture fit and as an indication that the candidate will be able to do some light prompt engineering as part of their role. The latter is important because language model APIs are built on top of web standards and—as noted above—aren’t always the easiest tools to work with.
Only a handful of people have built complex ML-first apps, but fortunately the two qualities listed above are relatively independent. Perhaps they’ve proven (2) through their professional experience and have some side projects which demonstrate (1).
Talking of side projects, evidence of creative and original prototypes is a huge plus as we’re evaluating candidates. We’ve barely scratched the surface of what’s possible to build with LLMs—even the current generation of models—so candidates who have been willing to dive into crazy “I wonder if it’s possible to…” ideas have a huge advantage.
Interviewing
The hard skills we spend most of our time evaluating during our interview process are in the “building complex systems using web technologies” side of things. We will be checking that the candidate is familiar with asynchronous programming, defensive coding, distributed systems concepts and tools, and display an ability to think about scaling and performance. They needn’t have 10+ years of experience doing this stuff: even junior candidates can display an aptitude and thirst for learning which gives us confidence they’ll be successful tackling the difficult technical challenges we’ll put in front of them.
One anti-pattern—something which makes my heart sink when I hear it from candidates—is that they have no familiarity with ML, but claim that they’re excited to learn about it. The amount of free and easily-accessible resources available is incredible, so a motivated candidate should have already dived into self-study.
Putting all that together, here’s the interview process that we follow for AI engineer candidates:
* 30-minute introductory conversation. Non-technical, explaining the interview process, answering questions, understanding the candidate’s career path and goals.
* 60-minute technical interview. This is a coding exercise, where we play product manager and the candidate is making changes to a little web app. Here are some examples of topics we might hit upon through that exercise:
* Update API endpoints to include extra metadata. Think about appropriate data types. Stub out frontend code to accept the new data.
* Convert a synchronous REST API to an asynchronous streaming endpoint.
* Cancellation of asynchronous work when a user closes their tab.
* Choose an appropriate data structure to represent the pending, active, and completed ML work which is required to service a user request.
* 60–90 minute non-technical interview. Walk through the candidate’s professional experience, identifying high and low points, getting a grasp of what kinds of challenges and environments they thrive in.
* On-site interviews. Half a day in our office in Oakland, meeting as much of the team as possible: more technical and non-technical conversations.
The frontier is wide open
Although Elicit is perhaps further along than other companies on AI engineering, we also acknowledge that this is a brand-new field whose shape and qualities are only just now starting to form. We’re looking forward to hearing how other companies do this and being part of the conversation as the role evolves.
We’re excited for the AI Engineer World’s Fair as another next step for this emerging subfield. And of course, check out the Elicit careers page if you’re interested in joining our team.
Podcast version
Timestamps
* [00:00:24] Intros
* [00:05:25] Defining the Hiring Process
* [00:08:42] Defensive AI Engineering as a chaotic medium
* [00:10:26] Tech Choices for Defensive AI Engineering
* [00:14:04] How do you Interview for Defensive AI Engineering
* [00:19:25] Does Model Shadowing Work?
* [00:22:29] Is it too early to standardize Tech stacks?
* [00:32:02] Capabilities: Offensive AI Engineering
* [00:37:24] AI Engineering Required Knowledge
* [00:40:13] ML First Mindset
* [00:45:13] AI Engineers and Creativity
* [00:47:51] Inside of Me There Are Two Wolves
* [00:49:58] Sourcing AI Engineers
* [00:58:45] Parting Thoughts
Transcript
[00:00:00] swyx: Okay, so welcome to the Latent Space Podcast. This is another remote episode that we're recording. This is the first one that we're doing around a guest post. And I'm very honored to have two of the authors of the post with me, James and Adam from Elicit. Welcome, James. Welcome, Adam.
[00:00:22] James Brady: Thank you. Great to be here.
[00:00:23] Hey there.
[00:00:24] Intros
[00:00:24] swyx: Okay, so I think I will do this kind of in order. I think James, you're, you're sort of the primary author. So James, you are head of engineering at Elicit. You also, We're VP Eng at Teespring and Spring as well. And you also , you have a long history in sort of engineering. How did you, , find your way into something like Elicit where, , it's, you, you are basically traditional sort of VP Eng, VP technology type person moving into a more of an AI role.
[00:00:53] James Brady: Yeah, that's right. It definitely was something of a Sideways move if not a left turn. So the story there was I'd been doing, as you said, VP technology, CTO type stuff for around about 15 years or so, and Notice that there was this crazy explosion of capability and interesting stuff happening within AI and ML and language models, that kind of thing.
[00:01:16] I guess this was in 2019 or so, and decided that I needed to get involved. , this is a kind of generational shift. And Spent maybe a year or so trying to get up to speed on the state of the art, reading papers, reading books, practicing things, that kind of stuff. Was going to found a startup actually in in the space of interpretability and transparency, and through that met Andreas, who has obviously been on the, on the podcast before asked him to be an advisor for my startup, and he countered with, maybe you'd like to come and run the engineering team at Elicit, which it turns out was a much better idea.
[00:01:48] And yeah, I kind of quickly changed in that direction. So I think some of the stuff that we're going to be talking about today is how actually a lot of the work when you're building applications with AI and ML looks and smells and feels much more like conventional software engineering with a few key differences rather than really deep ML stuff.
[00:02:07] And I think that's one of the reasons why I was able to transfer skills over from one place to the other.
[00:02:12] swyx: Yeah, I
[00:02:12] James Brady: definitely
[00:02:12] swyx: agree with that. I, I do often say that I think AI engineering is about 90 percent software engineering with like the, the 10 percent of like really strong really differentiated AI engineering.
[00:02:22] And that might, that obviously that number might change over time. I want to also welcome Adam onto my podcast because you welcomed me onto your podcast two years ago.
[00:02:31] Adam Wiggins: Yeah, that was a wonderful episode.
[00:02:32] swyx: That was, that was a fun episode. You famously founded Heroku. You just wrapped up a few years working on Muse.
[00:02:38] And now you've described yourself as a journalist, internal journalist working on Elicit.
[00:02:43] Adam Wiggins: Yeah, well I'm kind of a little bit in a wandering phase here and trying to take this time in between ventures to see what's out there in the world and some of my wandering took me to the Elicit team. And found that they were some of the folks who were doing the most interesting, really deep work in terms of taking the capabilities of language models and applying them to what I feel like are really important problems.
[00:03:08] So in this case, science and literature search and, and, and that sort of thing. It fits into my general interest in tools and productivity software. I, I think of it as a tool for thought in many ways, but a tool for science, obviously, if we can accelerate that discovery of new medicines and things like that, that's, that's just so powerful.
[00:03:24] But to me, it's a. It's kind of also an opportunity to learn at the feet of some real masters in this space, people who have been working on it since it was, before it was cool, if you want to put it that way. So for me, the last couple of months have been this crash course, and why I sometimes describe myself as an internal journalist is I'm helping to write some, some posts, including Supporting James in this article here we're doing for latent space where I'm just bringing my writing skill and that sort of thing to bear on their very deep domain expertise around language models and applying them to the real world and kind of surface that in a way that's I don't know, accessible, legible, that, that sort of thing.
[00:04:03] And so, and the great benefit to me is I get to learn this stuff in a way that I don't think I would, or I haven't, just kind of tinkering with my own side projects.
[00:04:12] swyx: I forgot to mention that you also run Ink and Switch, which is one of the leading research labs, in my mind, of the tools for thought productivity space, , whatever people mentioned there, or maybe future of programming even, a little bit of that.
[00:04:24] As well. I think you guys definitely started the local first wave. I think there was just the first conference that you guys held. I don't know if you were personally involved.
[00:04:31] Adam Wiggins: Yeah, I was one of the co organizers along with a few other folks for, yeah, called Local First Conf here in Berlin.
[00:04:36] Huge success from my, my point of view. Local first, obviously, a whole other topic we can talk about on another day. I think there actually is a lot more what would you call it , handshake emoji between kind of language models and the local first data model. And that was part of the topic of the conference here, but yeah, topic for another day.
[00:04:55] swyx: Not necessarily. I mean , I, I selected as one of my keynotes, Justine Tunney, working at LlamaFall in Mozilla, because I think there's a lot of people interested in that stuff. But we can, we can focus on the headline topic. And just to not bury the lead, which is we're talking about hire, how to hire AI engineers, this is something that I've been looking for a credible source on for months.
[00:05:14] People keep asking me for my opinions. I don't feel qualified to give an opinion and it's not like I have. So that's kind of defined hiring process that I'm super happy with, even though I've worked with a number of AI engineers.
[00:05:25] Defining the Hiring Process
[00:05:25] swyx: I'll just leave it open to you, James. How was your process of defining your hiring, hiring roles?
[00:05:31] James Brady: Yeah. So I think the first thing to say is that we've effectively been hiring for this kind of a role since before you, before you coined the term and tried to kind of build this understanding of what it was.
[00:05:42] So, which is not a bad thing. Like it's, it was a, it was a good thing. A concept, a concept that was coming to the fore and effectively needed a name, which is which is what you did. So the reason I mentioned that is I think it was something that we kind of backed into, if you will. We didn't sit down and come up with a brand new role from, from scratch of this is a completely novel set of responsibilities and skills that this person would need.
[00:06:06] However, it is a A kind of particular blend of different skills and attitudes and and curiosities interests, which I think makes sense to kind of bundle together. So in the, in the post, the three things that we say are most important for a highly effective AI engineer are first of all, conventional software engineering skills, which is Kind of a given, but definitely worth mentioning.
[00:06:30] The second thing is a curiosity and enthusiasm for machine learning and maybe in particular language models. That's certainly true in our case. And then the third thing is to do with basically a fault first mindset, being able to build systems that can handle things going wrong in, in, in some sense.
[00:06:49] And yeah, the I think the kind of middle point, the curiosity about ML and language models is probably fairly self evident. They're going to be working with, and prompting, and dealing with the responses from these models, so that's clearly relevant. The last point, though, maybe takes the most explaining.
[00:07:07] To do with this fault first mindset and the ability to, to build resilient systems. The reason that is, is so important is because compared to normal APIs, where normal, think of something like a Stripe API or a search API or something like this. The latency when you're working with language models is, is wild, like you can get 10x variation.
[00:07:32] I mean, I was looking at the stats before, actually, before, before the podcast. We do often, normally, in fact, see a 10x variation in the P90 latency over the course of, Half an hour, an hour when we're prompting these models, which is way higher than if you're working with a, more kind of conventional conventionally backed API.
[00:07:49] And the responses that you get, the actual content and the responses are naturally unpredictable as well. They come back with different formats. Maybe you're expecting JSON. It's not quite JSON. You have to handle this stuff. And also the, the semantics of the messages are unpredictable too, which is, which is a good thing.
[00:08:08] Like this is one of the things that you're looking for from these language models, but it all adds up to needing to. Build a resilient, reliable, solid feeling system on top of this fundamentally, well, certainly currently fundamentally shaky foundation. The models do not behave in the way that you would like them to.
[00:08:28] And yeah, the ability to structure the code around them such that it does give the user this warm, reassuring, Snappy, solid feeling is is really what we're driving for there.
[00:08:42] Defensive AI Engineering as a chaotic medium
[00:08:42] Adam Wiggins: What really struck me as we, we dug in on the content for this article was that third point there. The, the language models is this kind of chaotic medium, this, this dragon, this wild horse you're, you're, you're riding and trying to guide in the direction that is going to be useful and reliable to users, because I think.
[00:08:58] So much of software engineering is about making things not only high performance and snappy, but really just making it stable, reliable, predictable, which is literally the opposite of what you get from from the language models. And yet, yeah, the output is so useful, and indeed, some of their Creativity, if you want to call it that, which is, is precisely their value.
[00:09:19] And so you need to work with this medium. And I guess the nuanced or the thing that came out of Elissa's experience that I thought was so interesting is quite a lot of working with that is things that come from distributed systems engineering. But you have really the AI engineers as we're defining them or, or labeling them on the illicit team is people who are really application developers.
[00:09:39] You're building things for end users. You're thinking about, okay, I need to populate this interface with some response to user input. That's useful to the tasks they're trying to do, but you have this. This is the thing, this medium that you're working with that in some ways you need to apply some of this chaos engineering, distributed systems engineering, which typically those people with those engineering skills are not kind of the application level developers with the product mindset or whatever, they're more deep in the guts of a, of a system.
[00:10:07] And so it's, those, those skills and, and knowledge do exist throughout the engineering discipline, but sort of putting them together into one person that is That feels like sort of a unique thing and working with the folks on the Elicit team who have that skills I'm quite struck by that unique that unique blend.
[00:10:23] I haven't really seen that before in my 30 year career in technology.
[00:10:26] Tech Choices for Defensive AI Engineering
[00:10:26] swyx: Yeah, that's a Fascinating I like the reference to chaos engineering. I have some appreciation, I think when you had me on your podcast, I was still working at Temporal and that was like a nice Framework, if you live within Temporal's boundaries, you can pretend that all those faults don't exist, and you can, you can code in a sort of very fault tolerant way.
[00:10:47] What is, what is you guys solutions around this, actually? Like, I think you're, you're emphasizing having the mindset, but maybe naming some technologies would help? Not saying that you have to adopt these technologies, but they're just, they're just quick vectors into what you're talking about when you're, when you're talking about distributed systems.
[00:11:03] Like, that's such a big, chunky word, , like are we talking, are Kubernetes or, and I suspect we're not, , like we're, we're talking something else now.
[00:11:10] James Brady: Yeah, that's right. It's more at the application level rather than at the infrastructure level, at least, at least the way that it works for us.
[00:11:17] So there's nothing kind of radically novel here. It is more a careful application of existing concepts. So the kinds of tools that we reach for to handle these kind of slightly chaotic objects that Adam was just talking about, are retries and fallbacks and timeouts and careful error handling. And, yeah, the standard stuff, really.
[00:11:39] There's also a great degree of dependence. We rely heavily on parallelization because, , these language models are not innately very snappy, and , there's just a lot of I. O. going back and forth. So All these things I'm talking about when I was in my earlier stages of a career, these are kind of the things that are the difficult parts that most senior software engineers will be better at.
[00:12:01] It is careful error handling, and concurrency, and fallbacks, and distributed systems, and, , eventual consistency, and all this kind of stuff and As Adam was saying, the kind of person that is deep in the guts of some kind of distributed systems, a really high, high scale backend kind of a problem would probably naturally have these kinds of skills.
[00:12:21] But you'll find them on, on day one, if you're building a, , an ML powered app, even if it's not got massive scale. I think one one thing that I would mention that we do do yeah, maybe, maybe two related things, actually. The first is we're big fans of strong typing. We share the types all the way from the Backend Python code all the way to the to the front end in TypeScript and find that is I mean We'd probably do this anyway But it really helps one reason around the shapes of the data which can going to be going back and forth and that's really important When you can't rely upon You you're going to have to coerce the data that you get back from the ML if you want if you want for it to be structured basically speaking and The second thing which is related is we use checked exceptions inside our Python code base, which means that we can use the type system to make sure we are handling, properly handling, all of the, the various things that could be going wrong, all the different exceptions that could be getting raised.
[00:13:16] So, checked exceptions are not, not really particularly popular. Actually there's not many people that are big fans of them. For our particular use case, to really make sure that we've not just forgotten to handle, , This particular type of error we have found them useful to to, to force us to think about all the different edge cases that can come up.
[00:13:32] swyx: Fascinating. How just a quick note of technology. How do you share types from Python to TypeScript? Do you, do you use GraphQL? Do you use something
[00:13:39] James Brady: else? We don't, we don't use GraphQL. Yeah. So we've got the We've got the types defined in Python, that's the source of truth. And we go from the OpenAPI spec, and there's a, there's a tool that you work and use to generate types dynamically, like TypeScript types from those OpenAPI definitions.
[00:13:57] swyx: Okay, excellent. Okay, cool. Sorry, sorry for diving into that rabbit hole a little bit. I always like to spell out technologies for people to dig their teeth into.
[00:14:04] How do you Interview for Defensive AI Engineering
[00:14:04] swyx: One thing I'll, one thing I'll mention quickly is that a lot of the stuff that you mentioned is typically not part of the normal interview loop.
[00:14:10] It's actually really hard to interview for because this is the stuff that you polish out in, as you go into production, the coding interviews are typically about the happy path. How do we do that? How do we, how do we design, how do you look for a defensive fault first mindset?
[00:14:24] Because you can defensive code all day long and not add functionality. to your to your application.
[00:14:29] James Brady: Yeah, it's a great question and I think that's exactly true. Normally the interview is about the happy path and then there's maybe a box checking exercise at the end of the candidate says of course in reality I would handle the edge cases or something like this and that unfortunately isn't isn't quite good enough when when the happy path is is very very narrow and yeah there's lots of weirdness on either side so basically speaking, it's just a case of, of foregrounding those kind of concerns through the interview process.
[00:14:58] It's, there's, there's no magic to it. We, we talk about this in the, in the po in the post that we're gonna be putting up on, on Laton space. The, there's two main technical exercises that we do through our interview process for this role. The first is more coding focus, and the second is more system designy.
[00:15:16] Yeah. White whiteboarding a potential solution. And in, without giving too much away in the coding exercise. You do need to think about edge cases. You do need to think about errors. The exercise consists of adding features and fixing bugs inside the code base. And in both of those two cases, it does demand, because of the way that we set the application up and the interview up, it does demand that you think about something other than the happy path.
[00:15:41] But your thinking is the right prompt of how do we get the candidate thinking outside of the, the kind of normal Sweet spot, smooth smooth, smoothly paved path. In terms of the system design interview, that's a little easier to prompt this kind of fault first mindset because it's very easy in that situation just to say, let's imagine that, , this node dies, how does the app still work?
[00:16:03] Let's imagine that this network is, is going super slow. Let's imagine that, I don't know, like you, you run out of, you run out of capacity in, in, in this database that you've sketched out here, how do you handle that, that, that sort of stuff. So. It's, in both cases, they're not firmly anchored to and built specifically around language models and ways language models can go wrong, but we do exercise the same muscles of thinking defensively and yeah, foregrounding the edge cases, basically.
[00:16:32] Adam Wiggins: James, earlier there you mentioned retries. And this is something that I think I've seen some interesting debates internally about things regarding, first of all, retries are, can be costly, right? In general, this medium, in addition to having this incredibly high variance and response rate, and, , being non deterministic, is actually quite expensive.
[00:16:50] And so, in many cases, doing a retry when you get a fail does make sense, but actually that has an impact on cost. And so there is Some sense to which, at least I've seen the AI engineers on our team, worry about that. They worry about, okay, how do we give the best user experience, but balance that against what the infrastructure is going to, , is going to cost our company, which I think is again, an interesting mix of, yeah, again, it's a little bit the distributed system mindset, but it's also a product perspective and you're thinking about the end user experience, but also the.
[00:17:22] The bottom line for the business, you're bringing together a lot of a lot of qualities there. And there's also the fallback case, which is kind of, kind of a related or adjacent one. I think there was also a discussion on that internally where, I think it maybe was search, there was something recently where there was one of the frontline search providers was having some, yeah, slowness and outages, and essentially then we had a fallback, but essentially that gave people for a while, especially new users that come in that don't the difference, they're getting a They're getting worse results for their search.
[00:17:52] And so then you have this debate about, okay, there's sort of what is correct to do from an engineering perspective, but then there's also what actually is the best result for the user. Is giving them a kind of a worse answer to their search result better, or is it better to kind of give them an error and be like, yeah, sorry, it's not working right at the moment, try again.
[00:18:12] Later, both are obviously non optimal, but but this is the kind of thing I think that that you run into or, or the kind of thing we need to grapple with a lot more than you would other kinds of, of mediums.
[00:18:24] James Brady: Yeah, that's a really good example. I think it brings to the fore the two different things that you could be optimizing for of uptime and response at all costs on one end of the spectrum and then effectively fragility, but kind of, if you get a response, it's the best response we can come up with at the other end of the spectrum.
[00:18:43] And where you want to land there kind of depends on, well, it certainly depends on the app, obviously depends on the user. I think it depends on the, feature within the app as well. So in the search case that you, that you mentioned there, in retrospect, we probably didn't want to have the fallback. And we've actually just recently on Monday, changed that to Show an error message rather than giving people a kind of degraded experience in other situations We could use for example a large language model from a large language model from provider B rather than provider A and Get something which is within the A few percentage points performance, and that's just a really different situation.
[00:19:21] So yeah, like any interesting question, the answer is, it depends.
[00:19:25] Does Model Shadowing Work?
[00:19:25] swyx: I do hear a lot of people suggesting I, let's call this model shadowing as a defensive technique, which is, if OpenAI happens to be down, which, , happens more often than people think then you fall back to anthropic or something.
[00:19:38] How realistic is that, right? Like you, don't you have to develop completely different prompts for different models and won't the, won't the performance of your application suffer from whatever reason, right? Like it may be caused differently or it's not maintained in the same way. I, I think that people raise this idea of fallbacks to models, but I don't think it's, I don't, I don't see it practiced very much.
[00:20:02] James Brady: Yeah, it is, you, you definitely need to have a different prompt if you want to stay within a few percentage points degradation Like I, like I said before, and that certainly comes at a cost, like fallbacks and backups and things like this It's really easy for them to go stale and kind of flake out on you because they're off the beaten track And In our particular case inside of Elicit, we do have fallbacks for a number of kind of crucial functions where it's going to be very obvious if something has gone wrong, but we don't have fallbacks in all cases.
[00:20:40] It really depends on a task to task basis throughout the app. So I can't give you a kind of a, a single kind of simple rule of thumb for, in this case, do this. And in the other, do that. But yeah, we've it's a little bit easier now that the APIs between the anthropic models and opening are more similar than they used to be.
[00:20:59] So we don't have two totally separate code paths with different protocols, like wire protocols to, to speak, which makes things easier, but you're right. You do need to have different prompts if you want to, have similar performance across the providers.
[00:21:12] Adam Wiggins: I'll also note, just observing again as a relative newcomer here, I was surprised, impressed, not sure what the word is for it, at the blend of different backends that the team is using.
[00:21:24] And so there's many The product presents as kind of one single interface, but there's actually several dozen kind of main paths. There's like, for example, the search versus a data extraction of a certain type, versus chat with papers, versus And each one of these, , the team has worked very hard to pick the right Model for the job and craft the prompt there, but also is constantly testing new ones.
[00:21:48] So a new one comes out from either, from the big providers or in some cases, Our own models that are , running on, on essentially our own infrastructure. And sometimes that's more about cost or performance, but the point is kind of switching very fluidly between them and, and very quickly because this field is moving so fast and there's new ones to choose from all the time is like part of the day to day, I would say.
[00:22:11] So it isn't more of a like, there's a main one, it's been kind of the same for a year, there's a fallback, but it's got cobwebs on it. It's more like which model and which prompt is changing weekly. And so I think it's quite, quite reasonable to to, to, to have a fallback that you can expect might work.
[00:22:29] Is it too early to standardize Tech stacks?
[00:22:29] swyx: I'm curious because you guys have had experience working at both, , Elicit, which is a smaller operation and, and larger companies. A lot of companies are looking at this with a certain amount of trepidation as, as, , it's very chaotic. When you have, when you have , one engineering team that, that, knows everyone else's names and like, , they, they, they, they meet constantly in Slack and knows what's going on.
[00:22:50] It's easier to, to sync on technology choices. When you have a hundred teams, all shipping AI products and all making their own independent tech choices. It can be, it can be very hard to control. One solution I'm hearing from like the sales forces of the worlds and Walmarts of the world is that they are creating their own AI gateway, right?
[00:23:05] Internal AI gateway. This is the one model hub that controls all the things and has our standards. Is that a feasible thing? Is that something that you would want? Is that something you have and you're working towards? What are your thoughts on this stuff? Like, Centralization of control or like an AI platform internally.
[00:23:22] James Brady: Certainly for larger organizations and organizations that are doing things which maybe are running into HIPAA compliance or other, um, legislative tools like that. It could make a lot of sense. Yeah. I think for the TLDR for something like Elicit is we are small enough, as you indicated, and need to have full control over all the levers available and switch between different models and different prompts and whatnot, as Adam was just saying, that that kind of thing wouldn't work for us.
[00:23:52] But yeah, I've spoken with and, um, advised a couple of companies that are trying to sell into that kind of a space or at a larger stage, and it does seem to make a lot of sense for them. So, for example, if you're trying to sell If you're looking to sell to a large enterprise and they cannot have any data leaving the EU, then you need to be really careful about someone just accidentally putting in, , the sort of US East 1 GPT 4 endpoints or something like this.
[00:24:22] I'd be interested in understanding better what the specific problem is that they're looking to solve with that, whether it is to do with data security or centralization of billing, or if they have a kind of Suite of prompts or something like this that people can choose from so they don't need to reinvent the wheel again and again I wouldn't be able to say without understanding the problems and their proposed solutions , which kind of situations that be better or worse fit for but yeah for illicit where really the The secret sauce, if there is a secret sauce, is which models we're using, how we're using them, how we're combining them, how we're thinking about the user problem, how we're thinking about all these pieces coming together.
[00:25:02] You really need to have all of the affordances available to you to be able to experiment with things and iterate rapidly. And generally speaking, whenever you put these kind of layers of abstraction and control and generalization in there, that, that gets in the way. So, so for us, it would not work.
[00:25:19] Adam Wiggins: Do you feel like there's always a tendency to want to reach for standardization and abstractions pretty early in a new technology cycle?
[00:25:26] There's something comforting there, or you feel like you can see them, or whatever. I feel like there's some of that discussion around lang chain right now. But yeah, this is not only so early, but also moving so fast. , I think it's . I think it's tough to, to ask for that. That's, that's not the, that's not the space we're in, but the, yeah, the larger an organization, the more that's your, your default is to, to, to want to reach for that.
[00:25:48] It, it, it's a sort of comfort.
[00:25:51] swyx: Yeah, I find it interesting that you would say that , being a founder of Heroku where , you were one of the first platforms as a service that more or less standardized what, , that sort of early developer experience should have looked like.
[00:26:04] And I think basically people are feeling the differences between calling various model lab APIs and having an actual AI platform where. , all, all their development needs are thought of for them. , it's, it's very much, and, and I, I defined this in my AI engineer post as well.
[00:26:19] Like the model labs just see their job ending at serving models and that's about it. But actually the responsibility of the AI engineer has to fill in a lot of the gaps beyond that. So.
[00:26:31] Adam Wiggins: Yeah, that's true. I think, , a huge part of the exercise with Heroku, which It was largely inspired by Rails, which itself was one of the first frameworks to standardize the SQL database.
[00:26:42] And people had been building apps like that for many, many years. I had built many apps. I had made my own templates based on that. I think others had done it. And Rails came along at the right moment. We had been doing it long enough that you see the patterns and then you can say look let's let's extract those into a framework that's going to make it not only easier to build for the experts but for people who are relatively new the best practices are encoded into you.
[00:27:07] That framework, , Model View Controller, to take one example. But then, yeah, once you see that, and once you experience the power of a framework, and again, it's so comforting, and you can develop faster, and it's easier to onboard new people to it because you have these standards. And this consistency, then folks want that for something new that's evolving.
[00:27:29] Now here I'm thinking maybe if you fast forward a little to, for example, when React came on the on the scene, , a decade ago or whatever. And then, okay, we need to do state management. What's that? And then there's, , there's a new library every six months. Okay, this is the one, this is the gold standard.
[00:27:42] And then, , six months later, that's deprecated. Because of course, it's evolving, you need to figure it out, like the tacit knowledge and the experience of putting it in practice and seeing what those real What those real needs are are, are critical, and so it's, it is really about finding the right time to say yes, we can generalize, we can make standards and abstractions, whether it's for a company, whether it's for, , a library, an open source library, for a whole class of apps and it, it's very much a, much more of a A judgment call slash just a sense of taste or , experience to be able to say, Yeah, we're at the right point.
[00:28:16] We can standardize this. But it's at least my, my very, again, and I'm so new to that, this world compared to you both, but my, my sense is, yeah, still the wild west. That's what makes it so exciting and feels kind of too early for too much. too much in the way of standardized abstractions. Not that it's not interesting to try, but , you can't necessarily get there in the same way Rails did until you've got that decade of experience of whatever building different classes of apps in that, with that technology.
[00:28:45] James Brady: Yeah, it's, it's interesting to think about what is going to stay more static and what is expected to change over the coming five years, let's say. Which seems like when I think about it through an ML lens, it's an incredibly long time. And if you just said five years, it doesn't seem, doesn't seem that long.
[00:29:01] I think that, that kind of talks to part of the problem here is that things that are moving are moving incredibly quickly. I would expect, this is my, my hot take rather than some kind of official carefully thought out position, but my hot take would be something like the You can, you'll be able to get to good quality apps without doing really careful prompt engineering.
[00:29:21] I don't think that prompt engineering is going to be a kind of durable differential skill that people will, will hold. I do think that, The way that you set up the ML problem to kind of ask the right questions, if you see what I mean, rather than the specific phrasing of exactly how you're doing chain of thought or few shot or something in the prompt I think the way that you set it up is, is probably going to be remain to be trickier for longer.
[00:29:47] And I think some of the operational challenges that we've been talking about of wild variations in, in, in latency, And handling the, I mean, one way to think about these models is the first lesson that you learn when, when you're an engineer, software engineer, is that you need to sanitize user input, right?
[00:30:05] It was, I think it was the top OWASP security threat for a while. Like you, you have to sanitize and validate user input. And we got used to that. And it kind of feels like this is the, The shell around the app and then everything else inside you're kind of in control of and you can grasp and you can debug, etc.
[00:30:22] And what we've effectively done is, through some kind of weird rearguard action, we've now got these slightly chaotic things. I think of them more as complex adaptive systems, which , related but a bit different. Definitely have some of the same dynamics. We've, we've injected these into the foundations of the, of the app and you kind of now need to think with this defined defensive mindset downwards as well as upwards if you, if you see what I mean.
[00:30:46] So I think it would gonna, it's, I think it will take a while for us to truly wrap our heads around that. And also these kinds of problems where you have to handle things being unreliable and slow sometimes and whatever else, even if it doesn't happen very often, there isn't some kind of industry wide accepted way of handling that at massive scale.
[00:31:10] There are definitely patterns and anti patterns and tools and whatnot, but it's not like this is a solved problem. So I would expect that it's not going to go down easily as a, as a solvable problem at the ML scale either.
[00:31:23] swyx: Yeah, excellent. I would describe in, in the terminology of the stuff that I've written in the past, I describe this inversion of architecture as sort of LLM at the core versus LLM or code at the core.
[00:31:34] We're very used to code at the core. Actually, we can scale that very well. When we build LLM core apps, we have to realize that the, the central part of our app that's orchestrating things is actually prompt, prone to, , prompt injections and non determinism and all that, all that good stuff.
[00:31:48] I, I did want to move the conversation a little bit from the sort of defensive side of things to the more offensive or, , the fun side of things, capabilities side of things, because that is the other part. of the job description that we kind of skimmed over. So I'll, I'll repeat what you said earlier.
[00:32:02] Capabilities: Offensive AI Engineering
[00:32:02] swyx: It's, you want people to have a genuine curiosity and enthusiasm for the capabilities of language models. We just, we're recording this the day after Anthropic just dropped Cloud 3. 5. And I was wondering, , maybe this is a good, good exercise is how do people have Curiosity and enthusiasm for capabilities language models when for example the research paper for cloud 3.
[00:32:22] 5 is four pages
[00:32:23] James Brady: Maybe that's not a bad thing actually in this particular case So yeah If you really want to know exactly how the sausage was made That hasn't been possible for a few years now in fact for for these new models but from our perspective as when we're building illicit What we primarily care about is what can these models do?
[00:32:41] How do they perform on the tasks that we already have set up and the evaluations we have in mind? And then on a slightly more expansive note, what kinds of new capabilities do they seem to have? Can we elicit, no pun intended, from the models? For example, well, there's, there's very obvious ones like multimodality , there wasn't that and then there was that, or it could be something a bit more subtle, like it seems to be getting better at reasoning, or it seems to be getting better at metacognition, or Or it seems to be getting better at marking its own work and giving calibrated confidence estimates, things like this.
[00:33:19] So yeah, there's, there's plenty to be excited about there. It's just that yeah, there's rightly or wrongly been this, this, this shift over the last few years to not give all the details. So no, but from application development perspective we, every time there's a new model release, there's a flow of activity in our Slack, and we try to figure out what's going on.
[00:33:38] What it can do, what it can't do, run our evaluation frameworks, and yeah, it's always an exciting, happy day.
[00:33:44] Adam Wiggins: Yeah, from my perspective, what I'm seeing from the folks on the team is, first of all, just awareness of the new stuff that's coming out, so that's, , an enthusiasm for the space and following along, and then being able to very quickly, partially that's having Slack to do this, but be able to quickly map that to, okay, What does this do for our specific case?
[00:34:07] And that, the simple version of that is, let's run the evaluation framework, which Lissa has quite a comprehensive one. I'm actually working on an article on that right now, which I'm very excited about, because it's a very interesting world of things. But basically, you can just try, not just, but try the new model in the evaluations framework.
[00:34:27] Run it. It has a whole slew of benchmarks, which includes not just Accuracy and confidence, but also things like performance, cost, and so on. And all of these things may trade off against each other. Maybe it's actually, it's very slightly worse, but it's way faster and way cheaper, so actually this might be a net win, for example.
[00:34:46] Or, it's way more accurate. But that comes at its slower and higher cost, and so now you need to think about those trade offs. And so to me, coming back to the qualities of an AI engineer, especially when you're trying to hire for them, It's this, it's, it is very much an application developer in the sense of a product mindset of What are our users or our customers trying to do?
[00:35:08] What problem do they need solved? Or what what does our product solve for them? And how does the capabilities of a particular model potentially solve that better for them than what exists today? And by the way, what exists today is becoming an increasingly gigantic cornucopia of things, right? And so, You say, okay, this new model has these capabilities, therefore, , the simple version of that is plug it into our existing evaluations and just look at that and see if it, it seems like it's better for a straight out swap out, but when you talk about, for example, you have multimodal capabilities, and then you say, okay, wait a minute, actually, maybe there's a new feature or a whole new There's a whole bunch of ways we could be using it, not just a simple model swap out, but actually a different thing we could do that we couldn't do before that would have been too slow, or too inaccurate, or something like that, that now we do have the capability to do.
[00:35:58] I think of that as being a great thing. I don't even know if I want to call it a skill, maybe it's even like an attitude or a perspective, which is a desire to both be excited about the new technology, , the new models and things as they come along, but also holding in the mind, what does our product do?
[00:36:16] Who is our user? And how can we connect the capabilities of this technology to how we're helping people in whatever it is our product does?
[00:36:25] James Brady: Yeah, I'm just looking at one of our internal Slack channels where we talk about things like new new model releases and that kind of thing And it is notable looking through these the kind of things that people are excited about and not It's, I don't know the context, the context window is much larger, or it's, look at how many parameters it has, or something like this.
[00:36:44] It's always framed in terms of maybe this could be applied to that kind of part of Elicit, or maybe this would open up this new possibility for Elicit. And, as Adam was saying, yeah, I don't think it's really a I don't think it's a novel or separate skill, it's the kind of attitude I would like to have all engineers to have at a company our stage, actually.
[00:37:05] And maybe more generally, even, which is not just kind of getting nerd sniped by some kind of technology number, fancy metric or something, but how is this actually going to be applicable to the thing Which matters in the end. How is this going to help users? How is this going to help move things forward strategically?
[00:37:23] That kind of, that kind of thing.
[00:37:24] AI Engineering Required Knowledge
[00:37:24] swyx: Yeah, applying what , I think, is, is, is the key here. Getting hands on as well. I would, I would recommend a few resources for people listening along. The first is Elicit's ML reading list, which I, I found so delightful after talking with Andreas about it.
[00:37:38] It looks like that's part of your onboarding. We've actually set up an asynchronous paper club instead of my discord for people following on that reading list. I love that you separate things out into tier one and two and three, and that gives people a factored cognition way of Looking into the, the, the corpus, right?
[00:37:55] Like yes, the, the corpus of things to know is growing and the water is slowly rising as far as what a bar for a competent AI engineer is. But I think, , having some structured thought as to what are the big ones that everyone must know I think is, is, is key. It's something I, I haven't really defined for people and I'm, I'm glad that this is actually has something out there that people can refer to.
[00:38:15] Yeah, I wouldn't necessarily like make it required for like the job. Interview maybe, but , it'd be interesting to see like, what would be a red flag. If some AI engineer would not know, I don't know what, , I don't know where we would stoop to, to call something required knowledge, , or you're not part of the cool kids club.
[00:38:33] But there increasingly is something like that, right? Like, not knowing what context is, is a black mark, in my opinion, right?
[00:38:40] I think it, I think it does connect back to what we were saying before of this genuine Curiosity about and that. Well, maybe it's, maybe it's actually that combined with something else, which is really important, which is a self starting bias towards action, kind of a mindset, which again, everybody needs.
[00:38:56] Exactly. Yeah. Everyone needs that. So if you put those two together, or if I'm truly curious about this and I'm going to kind of figure out how to make things happen, then you end up with people. Reading, reading lists, reading papers, doing side projects, this kind of, this kind of thing. So it isn't something that we explicitly included.
[00:39:14] We don't have a, we don't have an ML focused interview for the AI engineer role at all, actually. It doesn't really seem helpful. The skills which we are checking for, as I mentioned before, this kind of fault first mindset. And conventional software engineering kind of thing. It's, it's 0. 1 and 0.
[00:39:32] 3 on the list that, that we talked about. In terms of checking for ML curiosity and there are, how familiar they are with these concepts. That's more through talking interviews and culture fit types of things. We want for them to have a take on what Elisa is doing. doing, certainly as they progress through the interview process.
[00:39:50] They don't need to be completely up to date on everything we've ever done on day zero. Although, , that's always nice when it happens. But for them to really engage with it, ask interesting questions, and be kind of bought into our view on how we want ML to proceed. I think that is really important, and that would reveal that they have this kind of this interest, this ML curiosity.
[00:40:13] ML First Mindset
[00:40:13] swyx: There's a second aspect to that. I don't know if now's the right time to talk about it, which is, I do think that an ML first approach to building software is something of a different mindset. I could, I could describe that a bit now if that, if that seems good, but yeah, I'm a team. Okay. So yeah, I think when I joined Elicit, this was the biggest adjustment that I had to make personally.
[00:40:37] So as I said before, I'd been, Effectively building conventional software stuff for 15 years or so, something like this, well, for longer actually, but professionally for like 15 years. And had a lot of pattern matching built into my brain and kind of muscle memory for if you see this kind of problem, then you do that kind of a thing.
[00:40:56] And I had to unlearn quite a lot of that when joining Elicit because we truly are ML first and try to use ML to the fullest. And some of the things that that means is, This relinquishing of control almost, at some point you are calling into this fairly opaque black box thing and hoping it does the right thing and dealing with the stuff that it sends back to you.
[00:41:17] And that's very different if you're interacting with, again, APIs and databases, that kind of a, that kind of a thing. You can't just keep on debugging. At some point you hit this, this obscure wall. And I think the second, the second part to this is the pattern I was used to is that. The external parts of the app are where most of the messiness is, not necessarily in terms of code, but in terms of degrees of freedom, almost.
[00:41:44] If the user can and will do anything at any point, and they'll put all sorts of wonky stuff inside of text inputs, and they'll click buttons you didn't expect them to click, and all this kind of thing. But then by the time you're down into your SQL queries, for example, as long as you've done your input validation, things are pretty pretty well defined.
[00:42:01] And that, as we said before, is not really the case. When you're working with language models, there is this kind of intrinsic uncertainty when you get down to the, to the kernel, down to the core. Even, even beyond that, there's all that stuff is somewhat defensive and these are things to be wary of to some degree.
[00:42:18] Though the flip side of that, the really kind of positive part of taking an ML first mindset when you're building applications is that you, If you, once you get comfortable taking your hands off the wheel at a certain point and relinquishing control, letting go then really kind of unexpected powerful things can happen if you lean on the, if you lean on the capabilities of the model without trying to overly constrain and slice and dice problems with to the point where you're not really wringing out the most capability from the model that you, that you might.
[00:42:47] So, I was trying to think of examples of this earlier, and one that came to mind was we were working really early when just after I joined Elicit, we were working on something where we wanted to generate text and include citations embedded within it. So it'd have a claim, and then a, , square brackets, one, in superscript, something, something like this.
[00:43:07] And. Every fiber in my, in my, in my being was screaming that we should have some way of kind of forcing this to happen or Structured output such that we could guarantee that this citation was always going to be present later on that the kind of the indication of a footnote would actually match up with the footnote itself and Kind of went into this symbolic.
[00:43:28] I need full control kind of kind of mindset and it was notable that Andreas Who's our CEO, again, has been on the podcast, was was the opposite. He was just kind of, give it a couple of examples and it'll probably be fine. And then we can kind of figure out with a regular expression at the end. And it really did not sit well with me, to be honest.
[00:43:46] I was like, but it could say anything. I could say, it could literally say anything. And I don't know about just using a regex to sort of handle this. This is a potent feature of the app. But , this is that was my first kind of, , The starkest introduction to this ML first mindset, I suppose, which Andreas has been cultivating for much longer than me, much longer than most, of yeah, there might be some surprises of stuff you get back from the model, but you can also It's about finding the sweet spot, I suppose, where you don't want to give a completely open ended prompt to the model and expect it to do exactly the right thing.
[00:44:25] You can ask it too much and it gets confused and starts repeating itself or goes around in loops or just goes off in a random direction or something like this. But you can also over constrain the model. And not really make the most of the, of the capabilities. And I think that is a mindset adjustment that most people who are coming into AI engineering afresh would need to make of yeah, giving up control and expecting that there's going to be a little bit of kind of extra pain and defensive stuff on the tail end, but the benefits that you get as a, as a result are really striking.
[00:44:58] The ML first mindset, I think, is something that I struggle with as well, because the errors, when they do happen, are bad. , they will hallucinate, and your systems will not catch it sometimes if you don't have large enough of a sample set.
[00:45:13] AI Engineers and Creativity
[00:45:13] swyx: I'll leave it open to you, Adam. What else do you think about when you think about curiosity and exploring capabilities?
[00:45:22] Do people are there reliable ways to get people to push themselves? for joining us on Capabilities, because I think a lot of times we have this implicit overconfidence, maybe, of we think we know what it is, what a thing is, when actually we don't, and we need to keep a more open mind, and I think you do a particularly good job of Always having an open mind, and I want to get that out of more engineers that I talk to, but I, I, I, I struggle sometimes.
[00:45:45] Adam Wiggins: I suppose being an engineer is, at its heart, this sort of contradiction of, on one hand, yeah, systematic, almost very literal, yeah, wanting to control exactly what James described understand everything, model it in your mind, Precision, yeah, systematizing but fundamentally it is a, It is a creative endeavor, at least.
[00:46:09] I got into creating with computers because I saw them as a canvas for creativity, for making great things, and for making a medium for making things that are, , so multidimensional that it goes beyond any medium humanity's ever had for creating things. So I think, or hope, that a lot of engineers are drawn to it.
[00:46:31] Partially because you need both of those. You need that systematic controlling side and then the creative open ended, almost like artistic side. And I, and I think it is, I think it is exactly the same here. In fact, if anything, I feel like there's a theme running through everything James has said here, which is in many ways, what we're looking for in an AI engineer is not.
[00:46:52] Really all that fundamentally different from other, , call it conventional engineering or other types of engineering, but working with this strange new medium that has these different qualities. But in the end there, there, a lot of the things are an amalgamation of past engineering skills.
[00:47:07] And I think that, that mix of, yeah, curiosity, artistic, open ended, what can we do with this, with a desire to systematize, control, make reliable, make repeatable is, is the mix you need and trying to trying to find that balance, I think is, is probably where it's at. But fundamentally, I think people who are, are getting into this field to work on this is because it is an exciting, , they're excited by the promise and the potential of the technology.
[00:47:34] So to, to not have that kind of creative open ended curiosity side would be well would, would be surprising. Like what, why, why do it otherwise? So I think that, that blend is always what you're looking for. What you're looking for broadly, but here, now we're just scoping it to this new world of language models.
[00:47:51] Inside of Me There Are Two Wolves
[00:47:51] James Brady: I think the default first mindset and the ML curiosity attitude Could be somewhat intention, right? Because for example, the, the stereotypical, stereotypical version of someone that is great at building fault tolerant systems has probably been doing it for a decade or two. They've been principal engineer at some massive scale technology company.
[00:48:14] And that kind of a person might be less I think it's really important that people are able to turn on a dime and be under linkage control and be creative and take on this different mindset. Whereas someone who's very early in their career is much more able to do that kind of exploration and follow their curiosity kind of a thing.
[00:48:33] And they might be a little bit less creative. Practiced in how to, , serve terabytes of traffic every day, obviously. So
[00:48:43] Adam Wiggins: Yeah, the stereotype that comes to mind for me with those two you just described is the, the principal engineer, , fault tolerance, , handle unpredictable, is kind of grumpy and always skeptical of anything new and, , it's probably not going to work and that sort of thing.
[00:48:58] Whereas that, yeah, fresh face early in their career maybe more application focused and it's always thinking about the happy path and the optimistic and oh don't worry about the edge case that probably won't happen i i don't write code with bugs i don't know whatever like this but but really need both together i think in or both of those attitudes or personalities if that's even the right way to put it together in one I think
[00:49:21] James Brady: people can come from either end of the spectrum to be, to be clear.
[00:49:23] , not all grizzled principal engineers are the way that I'm described. Thankfully some, some probably are, and not all, , junior engineers are allergic to writing, , careful software or, or unable and unexcited to pick that up. So yeah, , it could be someone that's in the middle of the career and naturally has a bit of both.
[00:49:41] Could be someone at either end and just. , once they kind of round out their skill set and lean into the thing that they're a bit weaker on any of the, any of the above would work well for us. , a fair
[00:49:49] swyx: amount of like, actually we, I think we've accidentally defined AI engineering along the way as well, because you kind of have to do that in order to to hire and interview for people.
[00:49:58] Sourcing AI Engineers
[00:49:58] swyx: The last piece I wanted to And the last thing I would offer to our audience is sourcing a very underappreciated part because people just tend to rely on recruiters and, , assume that candidates fall from the sky. But I think the two of you have had plenty of experience with like really good sourcing and I just want to give leave some time open for what is AI engineer sourcing look like?
[00:50:19] Is it being very loud on Twitter?
[00:50:21] James Brady: Well, I mean, that definitely helps. I am really quiet on Twitter, unfortunately, but a lot of my teammates are much more effective on that front which is deeply appreciated. I think in terms of in terms of, maybe I'll focus a little bit more on active outbound, if you will, rather than the kind of yes, Marketing, branding type of work that that Adam's been really effective with us on.
[00:50:44] So the kinds of things that I'm looking for are certainly side projects. It's, it's really easy still. We're early on in this, early enough on in this process that people can still do interesting work pretty much at the cutting edge, not in terms of training whole models, of course, but AI engineering. You can.
[00:51:02] Very much build interesting apps that have interesting ideas and work well just using a, , basic Open API, Open AI API key. So, people sharing that kind of stuff on Twitter is always really interesting, or in, , Discord or Slacks, things like this. In terms of the, the kind of caricature of the grizzled principal engineer kind of a person, It's, it's notable.
[00:51:27] I mean, I've spoken with a bunch of people coming from that kind of perspective. They're fairly easy to find. They tend to be on LinkedIn. They tend to be really obvious on LinkedIn because they're maybe a bit more senior. They've got a ton of connections. They're probably expected to kind of post thought leadership kinds of things on LinkedIn.
[00:51:46] Everyone's favorite. And , some of those, some of those people are interested in picking up new skills and jumping into ML and, and large language models. And sometimes it's obvious from a profile. Sometimes you just need to reach out and introduce yourself and say, hey, this is what we're doing.
[00:52:00] We think we could use your skills and a bunch of them will, will, will bite your hand off actually, because it is such an interesting area. So that's how, that's how we've found success at sourcing on the kind of more experienced end of the spectrum. I think on the, on the less experienced end of the spectrum, having lots of hooks in the ocean seems to be a good strategy if I think about what's worked for us.
[00:52:25] So, it's, it tends to be much harder to find those people because they have less of an online presence in terms of like active outbound. So, things like blog posts, hot takes on Twitter, things like challenges that we might have Those are the kind of vectors through which you can find these keen, full of energy, less experienced people and bring them towards you.
[00:52:50] Yeah. Adam, do you have anything? You're pretty good on Twitter compared to me, at least. What's your, what's your take on yeah, the kind of more like throwing stuff out there and have people come towards you for this kind of a role.
[00:53:03] Adam Wiggins: Yeah, I do typically think of sourcing as being the one two punch of one, raise the beacon, let the world know that you are working on interesting problems, and you're expanding your team, and maybe there's a place for someone like them on that team, and that can come in a variety of forms, whether it's, , going to a job fair and having a booth, obviously it's job descriptions posted to your site, it's obviously things like, In some cases, yeah, blog posts about stuff you're working on, releasing open source, Anything that goes out into the world and people find out about what you're doing, Not at the very surface level of here's what the product is, And, I don't know, we have a couple job descriptions on the site, But a layer deeper of like, here's the kind, here's what it actually looks like.
[00:53:50] So, I think that's, that's one piece of it. And then the other piece of it is, as you said, is the outbound. I think it's not enough to especially when you're small. I think it's, it changes a lot when you're a bigger company with a strong brand or if the product you're working on is more in a technical space.
[00:54:05] And so, therefore, maybe your customer, there's actually among your customers, there's the sorts of people that you might might like to work for you. I don't know if you're a GitHub, then probably all of your users and customers, , the people you want to hire are among your user base, which is a nice combination, but for most products, that's not going to be the case.
[00:54:20] So then now the outbound is a big piece of it. And part of that is, as you said, getting out into the world, whether it's going to meetups, whether it's going to conferences, whether it's being on Twitter and just genuinely being out there and part of the field and having conversations with people and seeing people who are doing interesting things and making connections with them.
[00:54:37] Hopefully not in a. Transactional way, or you're always just, , sniffing around for who's available to hire. But you just generally, if you like this work and you want to be part of the field and you want to follow along with people who are doing interesting things, and then by the way, you will discover when they post, oh, I'm wrapping up my , my job here and thinking about the next thing and, , that's a good time to, to ping them and be like, oh, cool, , actually we, we have maybe some things that you, you might be interested in here on the team and that, that kind of, that kind of outbound, but I think it also pairs well, it's, it's not just that you need both, it's that they, they reinforce each other, so if someone has seen, for example, the open source project you've released, And they're like, Oh, that's cool.
[00:55:17] And they briefly looked at your company and then you follow each other on Twitter or whatever, and then they post, Hey, I'm thinking about my next thing and then you write them and they already have some context of like, Oh, I liked that project you did and I liked. , I kind of have some ambient awareness of what you're doing.
[00:55:31] Yeah. Let's have a conversation. This isn't totally cold. So I think those, those two together are important. The other footnote I would put again on the specifics, that's, I think, general sourcing for any kind of role, but for AI engineering specifically, you're not looking for professional experience at this stage.
[00:55:47] You're not always looking for professional experience with language models. It's just too early. So it's totally fine that someone has the professional experience with the Conventional engineering skills but yeah, the interest, the, the, the curiosity, that sort of thing expressed through side projects, hackathons, blog posts, whatever it is.
[00:56:06] swyx: Yeah, absolutely. I often tell people, a lot of people are asking me for San Francisco AI engineers because they want, there's this sort of wave or reaction against the remote mindset, which I know that you guys probably differ in opinion on, but a lot of people are trying to, , go back to office.
[00:56:20] And so my, my only option for people is just find them at the hackathons. Like they're, , the, the most self driven motivated people, Who can work on things quickly and ship fast are already in hackathons. And just go through the list of winners. And then self interestedly, , if, for example, someone's hosting an AI conference from June 25th to June 27th on San Francisco, you might want to show up there and see, for example, who might be available.
[00:56:45] So, and that is true, , not, , it's not something I want to advertise to the employers, the people who come, but a lot of people change jobs at conferences. This is a known thing so.
[00:56:54] Adam Wiggins: Yeah, of course. But I think it's the same as engaging on Twitter, engaging in open source, attending conferences, 100%, this is a great way both to find new opportunities if you're a job seeker, Find people for your team if you're a hiring manager, but if you come at it too networky and transactional, that's just gross for everyone.
[00:57:12] Hopefully, we're all people that got into this work largely because we love it, and it's nice to connect with other people that have the same, , skills and struggle with the same problems in their work. And you make genuine connections and you learn from each other, and by the way, from that can come as a, well, not quite a side effect, but an, an effect on the list is pairing together people who are looking for opportunities with people who have interesting problems to work on.
[00:57:38] swyx: Yeah, most important part of employer branding, , have, have a great mission have great teammates. , if you can show that off in, in whatever way you can you'll, you'll be, you'll be starting off on the right foot. On
[00:57:46] James Brady: that note, we have. Been really successful with hiring a number of people from From targeted job boards, maybe, maybe is the right way of saying it.
[00:57:55] So not some kind of generic Indeed. com or something, not to trash them, but something that's a bit more tied to your mission, tied to what you're doing, something which is really relevant, something which is going to cut down the search space for what you're looking at, what the candidate's looking at. So we're definitely, , affiliated with the AI safety, effective altruists kind of movement.
[00:58:19] I've gone to a few EA Globals and have hired people effectively through the 80, 000 hours list as, as well. So, , that's not the only reason why people would want to join Elicit, but as an example of, if you're interested in, in AI safety or, , whatever your take is on this stuff, then there's probably something, there's a sub stack, there's a podcast, there's a, there's a mailing list, there's a job board, there's something which lets you zoom in on the kind of particular take that, That you agree with.
[00:58:45] Parting Thoughts
[00:58:45] swyx: Cool. I will leave it there. Any, any last comments about just hiring in general advice to other technology leaders in AI? , one, one thing I'm trying to do for my conference as well is to create a forum for technology leaders to, to share thoughts, right?
[00:58:59] James Brady: Yeah, a couple of thoughts here. So firstly, when I think back to how I was when I was in my early 20s, when I was at, when I was at college or university, the maturity and capabilities and just kind of general put togetherness of people at that age now is strikingly different to, to, to where I was then.
[00:59:24] And I, I think this is. Not because I was especially lexadesical or something when I was, when I was young. I think it's I hear the same thing echoed in other people about my, about my age. So the takeaway from that is finding a way of presenting yourself to and identifying and bringing in really high capability young people into your organization.
[00:59:46] I mean, it's always been true, but I think it's even more true now. They're kind of more professional, more capable, more committed more driven. have more of a sense of what they're all about than certainly I did 20 years ago. So that's, that's the first thing. I think the second thing is in terms of the interview process, this is somewhat a general take, but it definitely applies to AI engineer roles.
[01:00:07] And I think more so to AI engineer roles. I really have a strong dislike and distaste for interview questions, which are arbitrary and kind of strip away all the context from what it really is to do the work. We try to make the interview process that's illicit. A simulation of working together. The only people that we go into an interview process with.
[01:00:29] are pretty obviously extraordinary really, really capable. They must have done something for them to have moved into the proper interview process. So it is a check on technical capability and in the ways that we've described, but it's at least as much them sizing us up. Like, is this something which is worth my time?
[01:00:49] Is it something that I'm going to really be able to dedicate myself to? So being able to show them, this is really what it's like working at Elicit. This is the people you're going to work with. These are the kinds of tasks that you're going to be doing. This is the sort of environment that we work in.
[01:01:00] These are the tools we use. All that kind of stuff is really, really important from a candidate experience, but it also gives us a ton more signal as well about, , what is it actually like to work with this person? Not just can they do really well on some kind of leak code style, style problem.
[01:01:15] I think the reason that it bears a particularly on the AI engineer role is because it is something of an emerging category, if you will. So there isn't a very kind of. Well established do these that nobody's written the book yet Maybe this is the beginning of us writing the book and how to get hired as an AI engineer but that book doesn't exist at the moment and Yeah, It's an empirical job as, as much as any other kind of software engineering.
[01:01:41] It's, it's less about having kind of book learning and more about being able to apply that in a real world situation. So let's make the interview as close to a real world situation as possible.
[01:01:49] swyx: I do, I do co sign a lot of that. Yeah, I think this is a really great overview of just the, the, the sort of state of, Hiring AI engineers.
[01:01:56] And I honestly, that's just what, what AI engineering even is, which it really is like, when I was thinking about this as an industrial movement it was very much around, around the labor market, actually and the economic forces that give rise to, to a role like this both on the incentives of the model labs, as well as the demand and supply of engineers and the interest level of companies And the engineers working on these problems.
[01:02:20] So I definitely see you guys as pioneers. Thank you so much for putting together this piece, which is something I've been seeking for a long time. You even shared your job description, your reading list, and your interview loop. So, , if anyone's looking to hire AI engineers, I expect this to be the definitive piece and definitive podcast covering it.
[01:02:39] So thank you so much for taking the time to do this.
[01:02:43] Adam Wiggins: It was fun. Thanks for having us. Thanks a
[01:02:44] James Brady: lot. Really enjoyed the conversation. And I appreciate you naming something which we all had in our heads, but but couldn't put a label on.
[01:02:51] swyx: It was going to be named anyway. So I actually, I never, I never actually personally say that I coined a term because I'm sure someone else used the term before me.
[01:02:59] All I did was write a popular piece on it. All right. So I I'm happy to help because I know that it contributed to job creation at a bunch of companies I respect and, and, and help people find each other, which is my whole goal here. So, yeah, thanks for helping me do this.
In April 2023 we released an episode named “Mapping the future of *truly* open source models” to talk about Dolly, the first open, commercial LLM.
Mike was leading the OSS models team at Databricks at the time. Today, Mike is back on the podcast to give us the “one year later” update on the evolution of large language models and how he’s been using them to build Brightwave, an an AI research assistant for investment professionals.
Today they are announcing a $6M seed round (led by Alessio and Decibel!), and sharing some of the learnings from serving customers with >$120B of assets under management in production in the last 4 months since launch.
Losing faith in long context windows
In our recent “Llama3 1M context window” episode we talked about the amazing progress we have done in context window size, but it’s good to remember that Dolly’s original context size was 1,024 tokens, and this was only 14 months ago.
But while understanding length has increased, models are still not able to generate very long answers. His empirical intuition (which matches ours while building smol-podcaster) is that most commercial LLMs, as well as Llama, tend to generate responses <=1,200 tokens most of the time. While Needle in a Haystack tests will pass with flying colors at most context sizes, the granularity of the summary decreases as the context increases as it tries to fit the answer in the same tokens range, rather than returning tokens close to the 4,096 max_output, for example.
Recently Rob Mulla from Dreadnode highlighted how LMSys Arena results prefer longer responses by a large margin, so both LLMs and humans have a well documented length bias which doesn’t necessarily track the quality of answer:
The way Mike and team solved this is by breaking down the task in multiple subtasks, and then merging them back together. For example, have a book summarized chapter by chapter to preserve more details, and then put those summaries together. In Brightwave’s case, it’s creating multiple subsystems that accomplish different tasks on a large corpus of text separately, and then bringing them all together in a report. For example understanding intent of the question, extracting relations between companies, figuring out if it’s a positive / negative, etc.
Mike’s question is whether or not we’ll be able to imbue better synthesis capabilities in the models: can you have synthesis-oriented demonstrations at training time rather than single token prediction?
“LLMs as Judges” Strategies
In our David Luan episode he mentioned they don’t use any benchmarks for their models, because the benchmarks don’t reflect their customer needs. Brightwave shared some tips on leveraging LLMs as Judges:
* Human vs LLM reviews: while they work with human annotators to create high quality datasets, that data isn’t just used to fine tune models but also as a reference basis for future LLM reviews. Having a set of trusted data to use as calibration helps you trust the LLM judgement even more.
* Ensemble consistency checking: rather than using an LLM as judge for one output, you use different LLMs to generate a result for the same task, and then use another LLM to highlight where those generations differ. Do the two outputs differ meaningfully? Do they have different beliefs about the implications of something? If there are a lot of discrepancies between generations coming from different models, you then do additional passes to try and resolve them.
* Entailment verification: for each unique insight that they generate, they take the output and separately ask LLMs to verify factuality of information based on the original sources. In the actual product, user can then highlight any piece of text and ask it to 1) “Tell Me More” 2) “Show Sources”. Since there’s no way to guarantee factuality of 100% of outputs, and humans have good intuition for things that look out of the ordinary, giving the user access to the review tool helps them build trust in it.
It’s all about the data
During his time at Databricks, they had created dolly-15k, a dataset of instruction-following records written by thousands of their employees. Since then, no other company has replicated that type of effort even though the data wars are in full effect. It’s been clear in the last year that the half-life of a model is much shorter than the half-life of a dataset. The Pile by Eleuther (see Datasets 101) came out in 2020 and is still widely used; if you had trained an LLM in 2020, you would have definitely replaced it by now as they have gotten better and cheaper.
On the age old “RAG v Fine-Tuning” question, Mike shared a great example that we’ll just quote:
I think of language models kind of like a stem cell, and then under fine tuning, they differentiate into different kinds of specific cells. I don't think that unbounded agentic behaviors are useful, and that instead, a useful LLM system is more like a finite state machine where the behavior of the system is occupying one of many different behavioral regimes and making decisions about what state should I occupy next in order to satisfy the goal. As you think about the graph of those states that your system is moving through, once you develop conviction that one behavior is useful and repeatable and worthwhile to differentiate down into a specific kind of subsystem, that's where like fine tuning and specifically generating the training data, like having human annotators produce a corpus that is useful enough to get a specific class of behaviors, that's kind of how we use fine tuning rather than trying to imbue net new information into these systems.
There are a lot of other nuggets in the episode around knowledge graphs extraction, private vs public data, user intent extraction, etc, but we only have so much room in the writeup so go listen! And if you’re interested in working on these problems, Brightwave is hiring 👀
Watch on YouTube
We like Mike. The camera likes Mike. Our audience loooves Mike.
Show Notes
* Brightwave
* Mike Conover
* Mike on Latent Space #1
* Nature paper on S&P 500 talent movement
* Dolly announcement
* Dolly 15K dataset
* Bard blog post on double-checking generation
* RLHF 201 episode
* David Luan Episode
* Red Pajama
* Snorkel
* Renaissance
Timestamps
* [00:00:00] Introductions
* [00:02:40] Social media's polarization influence on LLMs
* [00:04:09] What's Brightwave?
* [00:05:13] How to hire for a vertical AI startup
* [00:09:34] How $20B+ hedge funds use Brightwave
* [00:11:23] Evolution of context sizes in language models
* [00:14:36] Summarizing vs Ideating with AI
* [00:18:26] Collecting feedback in a field with no truth
* [00:20:49] Evaluation strategies and the importance of custom datasets
* [00:23:43] Should more companies make employees label data?
* [00:25:32] Retrieval for highly temporal and hierarchical data
* [00:30:05] Context-aware prompting for private vs. public data
* [00:32:01] Knowledge graph extraction and structured information retrieval
* [00:33:49] Fine-tuning vs RAG
* [00:36:16] Anthropomorphizing language models
* [00:38:20] Why Brightwave doesn't do spreadsheets
* [00:42:24] Will there be fully autonomous hedge funds?
* [00:47:58] State of open source AI
* [00:53:53] Hiring and team expansion at Brightwave
Transcript
Alessio [00:00:01]: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO in Residence at Decibel Partners, and I have no co-host today. Swyx is in Vienna at ICLR having fun in Europe, and we're in the brand new studio. As you might see, if you're on YouTube, there's still no sound panels on the wall. Mike tried really hard to put them up, but the glue is a little too old for that. So if you hear any echo or anything like that, sorry, but we're doing the best that we can. And today we have our first repeat guest, Mike Conover. Welcome Mike, who's now the founder of Brightwave, not Databricks anymore.
Mike [00:00:40]: That's right. Yeah. Pleased to be back.
Alessio [00:00:42]: Our last episode was one of the fan favorites, and I think this will be just as good. So for those that have not listened to the first episode, which might be many because the podcast has grown a lot since then, thanks to people like Mike who have interesting conversations on it. You spent a bunch of years doing ML at some of the best companies on the internet, things like Workday, you know, Skipflag, LinkedIn, most recently at Databricks where you were leading the open source large language models team working on Dolly. And now you're doing Brightwave, which is in the financial services space. But this is not something new, I think when you and I first talked about Brightwave, I was like, why is this guy doing a financial services company? And then you look at your background and you were doing papers on The Nature Magazine about LinkedIn data predicting S&P 500 stock movement, like many, many years ago. So what are some of the tying elements in your background that maybe people are overlooking that brought you to do this?
Mike [00:01:36]: Yeah, sure. Yeah. So my PhD research was funded by DARPA and we had access to the Twitter data set early in the natural history of the availability of that data set, and it was focused on the large scale structure of propaganda and misinformation campaigns. And LinkedIn, we had planet scale descriptions of the structure of the global economy. And so primarily my work was homepage news feed relevant. So when you go to LinkedIn.com, you'd see updates from one of our machine learning models. But additionally, I was a research liaison as part of the economic graph challenge and had this Nature Communications paper where we demonstrated that 500 million jobs transitions can be hierarchically clustered as a network of labor flows and could predict next quarter S&P 500 market gap changes. And at Workday, I was director of financials machine learning. You start to see how organizations are organisms. And I think of the way that like an accountant or the market encodes information in databases similar to how social insects, for example, organize their work and make collective decisions about where to allocate resources or time and attention. And that especially with the work on Twitter, we would see network structures relating to polarization emerge organically out of the interactions of many individual components. And so like much of my professional work has been focused on this idea that our lives are governed by systems that we're unable to see from our locally constrained perspective. And when humans interact with technology, they create digital trace data that allows us to observe the structure of those systems as though through a microscope or a telescope. And particularly as regards finance, I think the markets are the ultimate manifestation and record of that collective decision making process that humans engage in.
Alessio [00:03:21]: Just to start going off script right away, how do you think about some of these interactions creating the polarization and how that reflects in the language models today because they're trained on this data? Like do you think the models pick up on these things on their own as well?
Mike [00:03:34]: Absolutely. Yeah. I think they are a compression of the world as it existed at the point in time when they were pre-trained. And so I think absolutely. And you see this in Word2Vec too. I mean, just the semantics of how we think about gender as it relates to professions are encoded in the structure of these models and like language models, I think are much more sort of complete representation of human sort of beliefs.
Alessio [00:04:01]: So we left you at Databricks last time you were building Dolly. Tell us a bit more about Brightwave. This is the first time you're really talking about it publicly.
Mike [00:04:09]: Yeah. Yeah. And it's a pleasure. So Brightwave is a $6 million seed round, led by Decibel, that we love working with, and including participation from Point72, one of the largest hedge funds in the world and Moonfire Ventures. And if you think of the job of an active asset manager, the work to be done is to understand something about the market that nobody else has seen in order to identify a mispriced asset. And it's our view that that is not a task that is well suited to human intellect or attention span. And so much as I was gesturing towards the ability of these models to perceive more than a human is able to, we think that there's a historically unique opportunity to expand individual's ability to reason about the structure of the economy and the markets. It's not clear that you get superhuman reasoning capabilities from human level demonstrations of skill. And by that I mean the pre-training corpus, but then additionally the fine tuning corpuses. I think you largely mimic the demonstrations that are present at model training time. But from a working memory standpoint, these models outclass humans in their ability to reason about these systems.
Alessio [00:05:13]: And you started Brightwave with Brandon. What's the story? You two worked together at Workday, but he also has a really relevant background.
Mike [00:05:20]: Yes. So Brandon Kotara is my co-founder, the CTO, and he's a very special human. So he has a deep background in finance. He was the former CTO of a federally regulated derivatives exchange, but his first deep learning patent was filed in 2018. And so he spans worlds. He has experience building mission critical infrastructure in highly regulated environments for finance use cases, but also was very early to the deep learning party and understand. He led at Workday, was the tech lead for semantic search over hundreds of millions of resumes and job listings. And so just has been working with information retrieval and neural information retrieval methods for a very long time. And so was an exceptional person, and I'm glad to count him among the people that we're doing this with.
Alessio [00:06:07]: Yeah. And a great fisherman.
Mike [00:06:09]: Yeah. Very talented.
Alessio [00:06:11]: That's always important.
Mike [00:06:12]: Very enthusiastic.
Alessio [00:06:13]: And then you have a bunch of amazing engineers, then you have folks like JP who used to work at Goldman Sachs. Yeah. How should people think about team building in this more vertical domain? Obviously you come from a deep ML background, but you also need some of the industry side. What's the right balance?
Mike [00:06:28]: I think one of the things that's interesting about building verticalized solutions in AI in 2024 is that historically, you need the AI capability, you need to understand both how the models behave and then how to get them to interact with other kinds of machine learning subsystems that together perform the work of a system that can reason on behalf of a human. There are also material systems engineering problems in there. So I saw, I forget who this is attributed to, but a tweet that made reference to all of the traditional software companies are trying to hire AI talent and all the AI companies are trying to hire systems engineers, and that is 100% the case. Getting these systems to behave in a predictable and repeatable and observable way is equally challenging to a lot of the methodological challenges. But then you bring in, whether it's law or medicine or public policy or in our case finance, I think a lot of the most valuable, like Grammarly is a good example of a company that has generative work product that is valuable by most humans. Whereas in finance, the character of the insight, the depth of insight and the non-consensusness of the insight really requires fairly deep domain expertise. And even operating an exchange, I mean, when we went to raise it around, a lot of people said, why don't you start a hedge fund? And it's like, there are many, many separate skills that are unrelated to AI in that problem. And so we've brought into the fold domain experts in finance who can help us evaluate the character and sort of steer the system.
Alessio [00:07:59]: So that's the team. What does the system actually do? What's the Brightwave product?
Mike [00:08:03]: Yeah. I mean, it does many, many things, but it acts as a partner in thought to finance professionals. So you can ask Brightwave a question like, how is NVIDIA's position in the GPU market impacted by rare earth metal shortages? And it will identify as thematic contributors to an investment decision or developing your thesis that in response to export controls on A100 cards, China has put in place licensors on the transfer of germanium and gallium, which are not rare earth metals, but they're semiconductor production inputs and has expanded its control of African and South American mining operations. And so we see, if you think about, we have a $20 billion crossover hedge fund. Their equities team uses this tool to go deep on a thesis. So I was describing this like multiple steps into the value chain or supply chain for companies. We see wealth management professionals using Brightwave to get up to speed extremely quickly as they step into nine conversations tomorrow with clients who are assessing like, do you know something that I don't? Can I trust you to be a steward of my financial wellbeing? We see investor relations teams using Brightwave. You just think about the universe of coverage that a person working in finance needs to be aware of, the ability to rip through filings and transcripts and have a very comprehensive view of the market. It's extremely rate limited by how quickly a person is able to read and not just read, but like solve the blank page problem of knowing what to say about a factor of finding.
Alessio [00:09:34]: So you mentioned the $20 billion hedge fund. What's like the range of customers that you work with as far as AUM goes?
Mike [00:09:41]: I mean, we have customers across the spectrum. So from $500 million owner operated RIAs to organizations with tens and tens of billions of dollars in asset center management.
Alessio [00:09:52]: What else can you share about customers that you're working with?
Mike [00:09:55]: Yeah. So we have seen traction that far exceeded our expectations from the market. You sit somebody down with a system that can take any question and generate tight, actionable financial analysis on that subject and the product kind of sells itself. So we see many, many different funds, firms, and strategies that are making use of Brightwave. So you've got 10 person owner operated registered investment advisor, the classical wealth manager, you know, $500 million in AUM. We have crossover hedge funds that have tens and tens of billions of dollars in assets center management, very different use case. So that's more investment research, whereas the wealth managers can use this to step into client interactions, just exceptionally well prepared. We see investor relations teams. We see corporate strategy types that are needing to understand very quickly new markets, new themes, and just the ability to very quickly develop a view on any investment theme or sort of strategic consideration is broadly applicable to many, many different kinds of personas.
Alessio [00:10:56]: Yeah. I can attest to the product selling itself, given that I'm a user. Let's jump into some of the technical challenges and work behind it, because there are a lot of things. As I mentioned, you were on the podcast about a year ago. Yep. You had released Dolly from Databricks, which was one of the first open source LLMs. Yep. Dolly had a whopping 1,024 tokens of context size. And today, you know, I think a thousand tokens, a model would be unusable.
Mike [00:11:23]: You lose that much out.
Alessio [00:11:24]: Yeah, exactly. How did you think about the evolution of context sizes as you built the company and where we are today? What are things that people get wrong? Any commentary there?
Mike [00:11:34]: Sure. We very much take a systems of systems approach. When I started the company, I think I had more faith in the ability of large context windows to generally solve problems relating to synthesis. And actually, if you think about the attention mechanism and the way that it computes similarities between tokens at a distance, I, on some level, believed that as you would scale that up, you would have the ability to simultaneously perceive and draw conclusions across vast, disparate bodies of content. And I think that does not empirically seem to be the case. So when, for example, you, and this is something anybody can try, take a very long document, like needle in a haystack. I think, sure, we can do information retrieval on specific fact-finding activities pretty easily. I kind of think about it like summarizing, if you write a book report on an entire book versus a synopsis of each individual chapter, there is a characteristic output length for these models. Let's say it's about 1,200 tokens. It is very difficult to get any of the commercial LLMs or LLAMA to write 5,000 tokens. And you think about it as, what is the conditional probability that I generate an end token? It just gets higher the more tokens are in the context window prior to that sort of next inference step. And so if I have 1,000 words in which to say something, the level of specificity and the level of depth when I am assessing a very large body of content is going to necessarily be less than if I am saying something specific about a sub-passage. I mean, if you think about drawing a parallel to consumer internet companies like LinkedIn or Facebook, there are many different subsystems with it. So let's take the Facebook example. Facebook almost certainly has, I mean, you can see this in your profile, your inferred interests. What are the things that it believes that you care about? Those assessments almost certainly feed into the feed relevance algorithms that would judge what you are, you know, am I going to show you snowboarding content? I'm going to show you aviation content. It's the outputs of one machine learning system feeding into another machine learning system. And I think with modern rag and sort of agent-based reasoning, it is really about creating subsystems that do specific tasks well. And I think the problem of deciding how to decompose large documents into more kind of atomic reasoning units is still very important. Now, it's an open question whether that is a model that is addressable by pre-training or instruction tuning. Like, can you have synthesis-oriented demonstrations at training time? And now this problem is more robustly solved because synthesis is quite different from complete the next word in the great Gatsby. I think empirically is not the case that you can just throw all of the SCC filings in a million token context window and get deep insight that is useful out the other end.
Alessio [00:14:36]: Yeah. And I think that's the main difference about what you're doing. It's not about summarizing. It's about coming up with different ideas and kind of like thought threads to pull on.
Mike [00:14:47]: Yeah. You know, if I think that GLP-1s are going to blow up the diet industry, identifying and putting in context a negative result from a human clinical trial, or for example, that adherence rates to Ozempic after a year are just 35%, what are the implications of this? So there's an information retrieval component. And then there's a not just presenting me with a summary of like, here's here are the facts, but like, what does this entail? And how does this fit into my worldview, my fund strategy? Broadly, I think that, you know, I mean, this idea, I think, is very eloquently puts it, which is, and this is not my insight, but that language models, and help me know who said this. You may be familiar, but language models are not tools for creating new knowledge. They're tools for helping me create new knowledge. Like they themselves do not do that. I think that that's presently the right way to think about it.
Alessio [00:15:36]: Yeah. I've read a tweet about Needle in the Haystack actually being harmful to some of this work because now the model is like too focused on recalling everything versus saying, oh, that doesn't matter. Like ignoring some of the things, if you think about a S1 filing, like 85% is like boilerplate. It's like, you know, previous performance doesn't guarantee future performance. Like the company might not be able to turn a profit in the future, blah, blah, blah. All these things, they always come up again.
Mike [00:16:02]: COVID and currency fluctuations.
Alessio [00:16:03]: Yeah, yeah, yeah. Yada, yada, yada. We have a large workforce and all of that. Have you had to do any work at the model level to kind of like make it okay to forget these things? Or like have you found that making it a smaller problem than putting them back together kind of solves for that?
Mike [00:16:19]: Absolutely. And I think this is where having domain expertise around the structure of these documents. So if you look at the different chunking strategies that you can employ to understand like what is the intent of this clause or phrase, and then really be selective at retrieval time in order to get the information that is most relevant to a user query based on the semantics of that unique document. And I think it's certainly not just a sliding window over that corpus.
Alessio [00:16:45]: And then the flip side of it is obviously factuality. You don't want to forget things that were there. How do you tackle that?
Mike [00:16:52]: Yeah, I mean, of course, it's a very deep problem. And I think I'll be a little circumspect about the specific kinds of methods we use. This sort of multiple passes over the material and saying, how convicted are you that what you're saying is in fact true? And you can take generations from multiple different models and compare and contrast and say, do these both reach the same conclusion? You can treat it like a voting problem. We train our own models to assess. You can think of this like entailment. Is this supported by the underlying primary sources? And I think that you have methodological approaches to this problem, but then you also have product affordances. There was a great blog post on Bard from the Bard team. It was sort of a design-led product innovation that allows you to ask the model to double-check the work. So if you have a surprising finding, we can let the user discretionarily spend more compute to double-check the work. And I think that you want to build product experiences that are fault tolerant. And the difference between hallucination and creativity is fuzzy. Do you ever get language models with Next Token Prediction as the loss function that are guaranteed to not contain factual misstatements? That is not clear. Now, maybe being able to invoke Code Interpreter, like code generation and then execution in a secure way, helps to solve some of these problems, especially for quantitative reasoning. That may be the case, but for right now, I think you need to have product affordances that allow you to live with the reality that these things are fallible.
Alessio [00:18:26]: We did our RLHF 201 episode, just talking about different methods and whatnot. How do you think about something like this, where it's maybe unclear in the short term, even if the product is right? It might give an insight that might be right, but it might not prove until later. So it's kind of hard for the users to say, that's wrong, because actually it might be like, you think it's wrong. Like an investment, that's kind of what it comes down to. Some people are wrong. Some people are right. How do you think about some of the product features that you need and something like this to bring user feedback into the mix and maybe how you approach it today and how you think about it long term?
Mike [00:19:01]: Yeah, well, I mean, I think that your point about the model may make a statement which is not actually verifiable. It's like, this may be the case. I think that is where the reason we think of this as a partner in thought, is that humans are always going to have access to information that has not been digitized. And so in finance, you see that, especially with regards to expert call networks, the unstated investment theses that a portfolio manager may have, like, we just don't do biotech. Or we think that Eli Lilly is actually very exposed because of how unpleasant it is to take examples. Right. Those are things that are beliefs about the world, but that may not be like falsifiable right now. And so I think you can, again, take pages from the consumer web playbook and think about personalization. So it is getting a person to articulate everything that they believe is not a realistic task. Netflix doesn't ask you to describe what kinds of movies you like and they give you the option to vote, but nobody does this. And so what I think you do is you observe people's revealed preferences. So one of the capabilities that our system exposes is, given everything that Brightwave has read and assessed, and like the sort of synthesized financial analysis, what are the natural next questions that a person investigating this subject should ask? And you can think of this chain of thought and this deepening kind of investigative process and the direction in which the user steers the attention of this system reveals information about what do they care about, what do they believe, what kinds of things are important. And so at the individual level, but then also at the fund and firm level, you can develop like an implicit representation of your beliefs about the world in a way that you just you're never going to get somebody to write everything down.
Alessio [00:20:49]: How does that tie into one of our other favorite topics, e-mails? We had David Luan from Adapt and he mentioned they don't care about benchmarks because their customers don't work on benchmarks, they work on business results. How do you think about that for you? And maybe as you build a new company, when is the time to like still focus on the benchmark versus when it's time to like move on to your own evaluation using maybe labelers or whatnot?
Mike [00:21:14]: We use a fair bit of LLM supervision to evaluate multiple different subsystems. And I think that one of the reasons that we pay human annotators to evaluate the quality of the generative outputs, and I think that that is always the reference standard, but we frequently first turn to LLM supervision as a way to have, whether it's at fine-tuning time or even for subsystems that are not generative, what is the quality of the system? I think we will generate a small corpus of high-quality domain expert annotations and always compare that against how well is either LLM supervision or even just a heuristic. A simple thing you can do, this is a technique that we do not use, but as an example, do not generate any integers or any numbers that are not present in the underlying source data. If they're doing rag, you can just say you can't name numbers that are not, it's very sort of heavy-handed, but you can take the annotations of a human evaluator and then compare that. I mean, Snorkel kind of takes a similar perspective, like multiple different weak sort of supervision data sets can give you substantially more than any one of them does on their own. And so I think you want to compare the quality of any evaluation against human-generated sort of benchmark. But at the end of the day, especially for things that are nuanced, is this transcendent poetry, there's just no way to multiple choice your way out of that, you know? And so really where I think a lot of the flywheels for some of the large LLM companies are, it's methodological, obviously, but it's also just data generation. And you think about like, you know, for anybody who's done crowdsource work, and this I think applies to the high-skilled human annotators as well, like you look at the Google search quality evaluator guidelines, it's like a 90 or 120-page rubric describing like, what is a high-quality Google search result? And it's like very difficult to get on a human level people to reproducibly follow a rubric. And so what is your process for orchestrating that motion? Like how do you articulate what is high-quality insight? I think that's where a lot of the work actually happens, and that it's sort of the last resort. Ideally, you want to automate everything, but ultimately the most interesting problems right now are those that are not especially automatable.
Alessio [00:23:43]: One thing you did at Databricks was the, well, not that you did specifically, but the team there was like the Dolly 15K dataset. You mentioned people misvalue the value of this data. Why has no other company done anything similar with like creating this employee-led dataset? You can imagine some of these Goldman Sachs, they got like thousands and thousands of people in there. Obviously they have different privacy and whatnot requirements. Do you think more companies should do it? Do you think there's like a misunderstanding of how valuable that is?
Mike [00:24:15]: So I think Databricks is a very special company and led by people who are very sort of courageous, I guess is one word for it. Just like, let's just ship it. And I think it's unusual. And it's also because I think most companies will recognize, like if they go to the effort to produce something like that, they recognize that it is competitive advantage to have it and to be the only company that has it. And I think Databricks is in an unusual position in that they benefit from more people having access to these kinds of sources, but you also saw scale, I guess they haven't released it.
Alessio [00:24:49]: Well, yeah. I'm sure they have it because they charge people a lot of money.
Mike [00:24:51]: They created that alternative to GSM 8K, I believe was how that's said. I guess they too are not releasing that.
Alessio [00:25:01]: It's interesting because I talked to a lot of enterprises and a lot of them are like, man, I spent so much money on Scale. And I'm like, why don't you just do it? And they're like, what?
Mike [00:25:11]: So I think this again gets to the human process orchestration. It's one thing to do like a single monolithic push to create a training data set like that or an evaluation corpus. But I think it's another to have a repeatable process. And a lot of that realistically is pretty unsexy, like people management work. So that's probably a big part of it.
Alessio [00:25:32]: So we have these four wars of AI framework, the data quality war, we kind of touched on a little bit now. About RAG, that's like the other battlefield, RAG and context sizes and kind of like all these different things. You work in a space that has a couple of different things. One, temporality of data is important because every quarter there's new data and like the new data usually overrides the previous one. So you cannot just like do semantic search and hope that you get the latest one. And then you have obviously very structured numbers thing that are very important to the token level. Like, you know, 50% gross margins and 30% gross margins are very different, but you know, this organization is not that different. Any thoughts on like how to build a system to handle all of that as much as you can share, of course?
Mike [00:26:19]: Yeah, absolutely. So I think this again, rather than having open ended retrieval, open ended reasoning, our approach is to decompose the problem into multiple different subsystems that have specific goals. And so, I mean, temporality is a great example. When you think about time, I mean, just look at all of the libraries for managing calendars. Time is kind of at the intersection of language and math. And this is one of the places where, without taking specific technical measures to ensure that you get high quality narrative overlays of statistics that are changing over time and have a description of how a PE multiple is increasing or decreasing, and like a retrieval system that is aware of the time, sort of the time intent of the user query, right? So if I'm asking something about breaking news, that's going to be very different than if I'm looking for a thematic account of the past 18 months in Fed interest rate policy. You have to have retrieval systems that are, to your point, like if I just look for something that is a nearest neighbor without any of that temporal or other qualitative metadata overlay, you're just going to get a kind of a bag of facts. I think that that is explicitly not helpful, because the worst failure state for these systems is that they are wrong in a convincing way. And so I think, at least presently, you have to have subsystems that are aware of the semantics of the documents, or aware of the semantics of the intent behind the question, and then have multiple, we have multiple evaluation steps. Once you have the generated outputs, we assess it multiple different ways to know, is this a factual statement given the sort of content that's been retrieved?
Alessio [00:28:10]: Yep. And what about, I think people think of financial services, they think of privacy, confidentiality. What's kind of like customer's interest in that, as far as like sharing documents and like, how much of a deal breaker is that if you don't have them? I don't know if you want to share any about that and how you think about architecting the product.
Mike [00:28:29]: Yeah, so one of the things that gives our customers a high degree of confidence is the fact that Brandon operated a federally regulated derivatives exchange. That experience in these highly regulated environments, I mean, additionally, at Workday, I worked with the financials product, and without going into specifics, it's exceptionally sensitive data, and you have multiple tenants, and it's just important that you take the right approach to being a steward of that material. And so, from the start, we've built in a way that anticipates the need for controls on how that data is managed, and who has access to it, and how it is treated throughout the lifecycle. And so that, for our customer base, where frequently the most interesting and alpha-generating material is not publicly available, has given them a great degree of confidence in sharing. Some of this, the most sensitive and interesting material, with systems that are able to combine it with content that is either publicly or semi-publicly available, to create non-consensus insight into some of the most interesting and challenging problems in finance.
Alessio [00:29:40]: Yeah, we always say it breaks our recommendation systems for LLMs. How do you think about that when you have private versus public data, where sometimes you have public data as one thing, but then the private is like, well, actually, we got this insight model, with this insight scoop that we're going to figure out. How do you think in the RAC system about a value of these different documents? I know a lot of it is secret sauce, but- No, no, it's fine.
Mike [00:30:05]: I mean, I think that there is, so I will gesture towards this by way of saying context-aware prompting. So you can have prompts that are composable, and that have different command units that may or may not be present based on the semantics of the content that is being populated into the RAG context window. And so that's something we make great use of, which is, where is this being retrieved from? What does it represent? And what should be in the instruction set in order to treat and respect the underlying contents, not just as like, here's a bunch of text, you figure it out, but this is important in the following way, or this aspect of the SEC filings are just categorically uninteresting, or this is sell-side analysis from a favored source. And so it's that creating it, much like you have with the qualitative, the problem of organizing the work of humans, you have the problem of organizing the work of all of these different AI subsystems, and getting them to propagate what they know through the rest of the stack, so that if you have multiple seven, 10 sequence inference calls, that all of the relevant metadata is propagated through that system, and that you are aware of, where did this come from? How convicted am I that it is a source that should be trusted? I mean, you see this also just in analysis, right? So different, like Seeking Alpha is a good example of just a lot of people with opinions, and some of them are great, some of them are really mid, and how do you build a system that is aware of the user's preferences for different sources? I think this is all related to how, we talked about systems engineering, it's all related to how you actually build the systems.
Alessio [00:31:51]: And then, just to kind of wrap on the rec side, how should people think about knowledge graphs and kind of like extraction from documents, versus just like semantic search over the documents?
Mike [00:32:01]: Knowledge graph extraction is an area where we're making a pretty substantial investment, and so I think that it is underappreciated how powerful, there's the generative capabilities of language models, but there's also the ability to program them to function as arbitrary machine learning systems, basically for marginally zero cost. And so, the ability to extract structured information from huge, sort of unfathomably large bodies of content in a way that is single pass, so rather than having to reanalyze a document every time that you perform inference or respond to a user query, we believe quite firmly that you can also, in an additive way, perform single pass extraction over this body of text and then bring that into the RAG context window. And this really sort of levers off of my experience at LinkedIn, where you had this structured graph representation of the global economy, where you said, person A works at company B, we believe that there's an opportunity to create a knowledge graph that has resolution that greatly exceeds what any, whether it's Bloomberg or LinkedIn, currently has access to, where we're getting as granular as person X submitted congressional testimony that was critical of organization Y, and this is the language that is attached to that testimony, and then you have a structured data artifact that you can pivot through and reason over that is complementary to the generative capabilities that language models expose. And so it's the same technology being applied to multiple different ends. And this is manifest in the product surface, where it's a highly facetable, pivotable product, but it also enhances the reasoning capability of the system.
Alessio [00:33:49]: Yeah, you know, when you mentioned you don't wanna re-query like the same thing over and over, a lot of people may say, well, I'll just fine tune this information in the model, you know? How do you think about that? That was one thing when we started working together, you were like, we're not building foundation models. A lot of other startups were like, oh, we're building the finance financial model, the finance foundation model, or whatever. When is the right time for people to do fine tuning versus RAG? Any heuristics that you can share that you use to think about it?
Mike [00:34:19]: So we, in general, I do not, I'll just say like, I don't have a strong opinion about how much information you can imbue into a model that is not present in pre-training through large-scale fine tuning. The benefit of rag is the capability around grounded reasoning. So the, you know, forcing it to attend to a collection of facts that are known and available at inference time, and sort of like materially, like only using these facts. At least in my view, the role of fine tuning is really more around, I think of like language models kind of like a stem cell, and then under fine tuning, they differentiate into different kinds of specific cells, so kidney or an eye cell. And if you think about specifically, like, I don't think that unbounded agentic behaviors are useful, and that instead, a useful LLM system is more like a finite state machine where the behavior of the system is occupying one of many different behavioral regimes and making decisions about what state should I occupy next in order to satisfy the goal. As you think about the graph of those states that your system is moving through, once you develop conviction that one behavior is useful and repeatable and worthwhile to differentiate down into a specific kind of subsystem, that's where like fine tuning and like specifically generating the training data, like having human annotators produce a corpus that is useful enough to get a specific class of behaviors, that's kind of how we use fine tuning rather than trying to imbue net new information into these systems.
Alessio [00:36:00]: Yeah, and people always try to turn LLMs into humans. It's like, oh, this is my reviewer, this is my editor. I know you're not in that camp. So any thoughts you have on how people should think about, yeah, how to refer to models?
Mike [00:36:16]: I mean, we've talked a little bit about this, and it's notable that I think there's a lot of anthropomorphizing going on, and that it reflects the difficulty of evaluating the systems. Is it like, does the saying that you're the journal editor for Nature, does that help? Like you've got the editor, and then you've got the reviewer and you've got the, you're the private investigator. It's like, this is, I think, literally we wave our hands and we say, maybe if I tell you that I'm gonna tip you, that's gonna help. And it sort of seems to, and like maybe it's just like the more cycles, the more compute that is attached to the prompt and then the sort of like chain of thought at inference time, it's like, maybe that's all that we're really doing and that it's kind of like hidden compute. But our experience has been that you can get really, really high quality reasoning from roughly an agentic system without needing to be too cute about it. You can describe the task and within well-defined bounds, you don't need to treat the LLM like a person in order to get it to generate high quality outputs.
Alessio [00:37:24]: And the other thing is like all these agent frameworks are assuming everything is an LLM.
Mike [00:37:29]: Yeah, for sure. And I think this is one of the places where traditional machine learning has a real material role to play in producing a system that hangs together. And there are guaranteeable like statistical promises that classical machine learning systems to include traditional deep learning can make about what is the set of outputs and like what is the characteristic distribution of those outputs that LLMs cannot afford. And so like one of the things that we do is we, as a philosophy, try to choose the right tool for the job. And so sometimes that is a de novo model that has nothing to do with LLMs that does one thing exceptionally well. And whether that's retrieval or critique or multiclass classification, I think having many, many different tools in your toolbox is always valuable.
Alessio [00:38:20]: This is great. So there's kind of the missing piece that maybe people are wondering about. You do a financial services company and you didn't do anything in Excel. What's the story behind why you're doing partner in thought versus, hey, this is like a AI enabled model that understands any stock and all that?
Mike [00:38:37]: Yeah, and to be clear, Brightwave does a fair amount of quantitative reasoning. I think what is an explicit non-goal for the company is to create Excel spreadsheets. And I think when you look at products that work in that way, you can spend hours with an Excel spreadsheet and not notice a subtle bug. And that is a highly non-fault tolerant product experience where you encounter a misstatement in a financial model in terms of how a formula is composed and all of your assumptions are suddenly violated. And now it's effectively wasted effort. So as opposed to the partner in thought modality, which is yes and, like if the model says something that you don't agree with, you can say, take it under consideration. This is not interesting to me. I'm going to pivot to the next finding or claim. And it's more like a dialogue. The other piece of this is that the financial modeling is often very, when we talk to our users, it's very personal. So they have a specific view of how a company is structured. They have the one key driver of asset performance that they think is really, really important. It's kind of like the difference between writing an essay and having an essay, I guess. Like the purpose of homework is to actually develop what do I think about this? And so it's not clear to me that like push a button, have a financial model is solving the actual problem that the financial model affords. That said, we take great efforts to have exceptionally high quality quantitative reasoning. So we think about, and I won't get into too many specifics about this, but we deal with a fair number of documents that have tabular data that is really important to making informed decisions. And so the way that our RAG systems operate over and retrieve from tabular data sources is it's something that we place a great degree of emphasis on it's just, I think the medium of Excel spreadsheets is just, I think not the right play for this class of technologies as they exist in 2024.
Alessio [00:40:40]: Yeah, what about 2034?
Mike [00:40:42]: 2034?
Alessio [00:40:43]: Are people still going to be making Excel models or like, yeah, I think to me, the most interesting thing is like, how are the models abstracting people away from some of these more syntax driven thing and making them focus on what matters to them?
Mike [00:40:58]: Yeah, I wouldn't be able to tell you what the future, 10 years from now it looks like. I think anybody who could convince you of that is not necessarily somebody to be trusted. I do think that, so let's draw the parallel to accountants in the 70s. So VisiCalc, I believe came out in 1979. And historically the core, you know, you would have as an accountant, as a finance professional in the 70s, like I'm the one who runs the, I run the numbers. I do the arithmetic and that's like my main job. And we think that, I mean, you just look now that's not a job anybody wants. And the sophistication of the analysis that a person is able to perform as a function of having access to powerful tools like computational spreadsheets is just much greater. And so I think that with regards to language models, it is probably the case that there is a play in the workflow where it is commenting on your analysis within that, you know, spreadsheet based context, or it is taking information from those models and sucking this into a system that does qualitative reasoning on top of that. But I think the, it is an open question as to whether the actual production of those models is still a human task. But I think the sophistication of the analysis that is available to us and the completeness of that analysis necessarily increases over time.
Alessio [00:42:24]: What about AI hedge funds? Obviously, I mean, we have quants today, right? But those are more kind of like momentum driven, kind of like signal driven and less about long thesis driven. Do you think that's a possibility?
Mike [00:42:35]: It's, this is an interesting question. I would put it back to you and say like, how different is that from what hedge funds do now? I think there is, the more that I have learned about how teams at hedge funds actually behave, and you look at like systematics desks or semi-systematic trading groups, man, it's a lot like a big machine learning team. And it's, I sort of think it's interesting, right? So like, if you look at video games and traditional like Bay Area tech, there's not a ton of like talent mobility between those two communities. You have people that work in video games and people that work in like SaaS software. And it's not that like cognitively they would not be able to work together. It's just like a different set of skill sets, a different set of relationships. And it's kind of like network clusters that don't interact. I think there's probably a similar phenomenon happening with regards to machine learning within the active asset allocation community. And so like, it's actually not clear to me that we don't have AI hedge funds now. The question of whether you have an AI that is operating a trading desk, that seems a little, maybe, like I don't have line of sight to something like that existing yet. No, I mean, I'm always curious.
Alessio [00:43:48]: I think about asset management on a few different ways, but venture capital is like extremely power law driven. It's really hard to do machine learning in power law businesses because, you know, the distribution of outcomes is like so small versus public equities. Most high-frequency trading is like very, you know, bell curve, normal distribution. It's like, even if you just get 50.5% at the right scale, you're gonna make a lot of money. And I think AI starts there, right? And today, most high-frequency trading is already AI driven. You know, Renaissance started a long time ago using these models. But I'm curious how it's gonna move closer and closer to like power law businesses, right? I would say some boutique hedge funds, their pitch is like, hey, we're differentiated because we only do kind of like these long-only strategies that are like thesis driven versus, you know, movement driven. And most venture capitalists will tell you, well, our fund is different because we have this unique thesis on this market. And I think like five years ago, I've read this blog post about why machine learning would never work in venture because the things that you're investing in today, they're just like no precedent that should tell you this will work. You know, most new companies, a model will tell you this is not gonna work, you know, versus the closer you get to the public companies, the more any innovation is like, okay, this is kind of like this thing that happened. And I feel like these models are quite good at generalizing and thinking, again, going back to the partnering thought, like thinking about second order.
Mike [00:45:13]: Yeah, and that's maybe where concrete example, I think it certainly is the case that we tell retrospective, to your point about venture, we tell retrospective stories where it's like, well, here was the set of observable facts. This was knowable at the time, and these people made the right call and were able to cross correlate all of these different sources and said, this is the bet we're gonna make. I think that process of idea generation is absolutely automatable. And the question of like, do you ever get somebody who just sets the system running and it's making all of its own decisions like that, and it is truly like doing thematic investing or more of the like what a human analyst would be kind of on the hook for, as opposed to like HFT. But the ability of models to say, here is a fact pattern that is noteworthy, and we should pay more attention here. Because if you think about the matrix of like all possible relationships in the economy, it grows with the square of the number of facts you're evaluating, like polynomial with the number of facts you're evaluating. And so if I want to make bets on AI, I think it's like, what are ways to profit from the rise of AI? It is very straightforward to take a model and say, parse through all of these documents and find second order derivative bets and say, oh, it turns out that energy is like very, very adjacent to investments in AI and may not be priced in the same way that GPUs are. And a derivative of energy, for example, is long duration energy storage. And so you need a bridge between renewables, which have fluctuating demands, and the compute requirements of these data centers. And I think, and I'm telling this story as like, having witnessed Brightwave do this work, you can take a premise and say like, what are second and third order bets that we can make on this topic? And it's going to come back with, here's a set of reasonable theses. And then I think a human's role in that world is to assess like, does this make sense given our fund strategy? Does this, is this coherent with the calls that I've had with the management teams? There's this broad body of knowledge that I think humans sort of are the ultimate like, synthesizers and deciders. And like, maybe I'm wrong. Maybe the world of the future looks like, and the AI that truly does everything, I think it is kind of a singularity vector where it's like really hard to reason about like, what that world looks like. And like, you asked me to speculate, but I'm actually kind of hesitant to do so because it's just the forecast, the hurricane path just diverges far too much to have a real conviction about what that looks like.
Alessio [00:47:58]: Awesome, I know we've already taken up a lot of your time, but maybe one thing to touch on before wrapping is open source LLMs. Obviously you were at the forefront of it. We recorded our episode the day that Red Pajama was open source and we were like, oh man, this is mind blowing. This is going to be crazy. And now we're going to have an open source dense transformer model that is 400 billion parameters. I don't know if one year ago you could have told me that that was going to happen. So what do you think matters in open source? What do you think people should work on? What are like things that people should keep in mind to evaluate? Okay, is this model actually going to be good? Or is it just like cheating some benchmarks to look good? It's like, is there anything there? Like, yeah, this is the part of the podcast where people already dropped off if they wanted to. So they want to hear the hot things right now.
Mike [00:48:46]: I mean, I do think that that's another reason to have your own private evaluation corpuses is so that you can objectively and out of sample measure the performance of these models. And again, sometimes that just looks like giving everybody on the team 250 annotations and saying, we're just going to grind through this. And you have to tell, does this meet? The other thing about doing the work yourself is that you get to articulate your loss function precisely. What is the thing that, what do I actually want the system to behave like? Do I prefer this system or this model or this other model? Yeah, and I think the work around overfitting on the test I think is like that 100% is happening. One notable, in contrast to a year ago, say, the incentives, the economic incentives for companies to train their own foundation models, I think are diminishing. So the window in which you are the dominant pre-train, and let's say that you spend five to $40 million for like a, call it kind of a commodity-ish pre-train, not 400 billion would be another sort of-
Alessio [00:49:50]: It costs more than 40 million. Another leap.
Mike [00:49:52]: But the kind of thing that, like a small multi-billion dollar mom and pop shop might be able to pull off. The benefit that you get from that is like, I think, diminishing over time. And so I think fewer companies are going to make that capital outlay. And I think that there's probably some material negatives to that. But the other piece is that we're seeing that, at least in the past two and a half, three months, there's a convergence towards like, well, these models all behave fairly similarly. And it's probably that the training data on which they are pre-trained is substantially overlapping. And so it's generalizing a model that generalizes to that training data. And so it's unclear to me that you have this sort of balkanization where there are many different models, each of which is good in its own unique way, versus something like Lama becomes like, listen, this is a fine standard to build off of. We'll see, it's just like the upfront cost is so high. And I think for the people that have the money, the benefit of doing the pre-train is now less. Where I think it gets really interesting is how do you differentiate these and all of these different behavioral regimes? And I think the cost of producing instruction tuning and fine tuning data that creates specific kinds of behaviors, I think that's probably where the next generation of really interesting work starts to happen. If you see that the same model architecture trained on much more training data can exhibit substantially improved performance, it's the value of modeling innovations. For fundamental machine learning and AI research, there is still so much to be done. But I think that much lower hanging fruit, I guess, is developing new kinds of training data corpuses that elicit new behaviors from these models in a specific way. And so that's where, when I think about the availability to like a year ago, you had to have access to fairly high performance GPUs that were hard to get in order to get the experience of multiple reps fine tuning these models. And what you're doing when you take a corpus and then fine tune the model and then see across many inference passes, what is the qualitative character of the output, you're developing your own internal mental model of how does the composition of the training corpus shape the behavior of the model in a qualitative way. A year ago, it was very expensive to get that experience. And now you can just recompose multiple different training corpuses and see like, well, what do I do if I insert this set of demonstrations or I ablate that set of demonstrations? And that I think is a very, very valuable skill and one of the ways that you can have models and products that other people don't have access to. And so I think as more people, as those sensibilities proliferate because more people have that experience, you're gonna see teams that release data corpuses that just imbue the models with new behaviors that are especially interesting and useful. And I think that may be where some of the next sets of kind of innovation differentiation come from.
Alessio [00:53:03]: Yeah, yeah, when people ask me, I always tell them the half-life of a model, it's much shorter than a half-life of a dataset.
Mike [00:53:08]: Yes, absolutely.
Alessio [00:53:09]: I mean, the pile is still around and like core to most of these training runs versus all the models people trained a year ago. It's like, they're at the bottom of the LMC's litter board.
Mike [00:53:20]: It's kind of crazy, like I don't, just the parallels to other kinds of computing technology where like the work involved in producing the artifact is so significant and the like shelf life is like a week. You know, I'm sure there's a precedent, but it is remarkable.
Alessio [00:53:37]: Yeah, I remember when Dolly was the best open source model.
Mike [00:53:42]: Dolly was never the best open source model, but it demonstrated something that was not obvious to many people at the time. Yeah, but we always were clear that it was never state-of-the-art.
Alessio [00:53:53]: State-of-the-art or whatever that means, right? This is great, Mike. Anything that we forgot to cover that you want to add? Any call, I know you're, you know, thinking about growing the team.
Mike [00:54:03]: We are hiring across the board, AI engineering, classical machine learning, systems engineering and distributed systems, front-end engineering, design. We have many open roles on the team. We hire exceptional people. We fit the job to the person as a philosophy and would love to work with more incredible humans. Awesome.
Alessio [00:54:25]: Thank you so much for coming on, Mike.
Mike [00:54:26]: Thanks, Alessio.
Our second wave of speakers for AI Engineer World’s Fair were announced! The conference sold out of Platinum/Gold/Silver sponsors and Early Bird tickets! See our Microsoft episode for more info and buy now with code LATENTSPACE.
This episode is straightforwardly a part 2 to our ICLR 2024 Part 1 episode, so without further ado, we’ll just get right on with it!
Timestamps
[00:03:43] Section A: Code Edits and Sandboxes, OpenDevin, and Academia vs Industry — ft. Graham Neubig and Aman Sanger
* [00:07:44] WebArena
* [00:18:45] Sotopia
* [00:24:00] Performance Improving Code Edits
* [00:29:39] OpenDevin
* [00:47:40] Industry and Academia
[01:05:29] Section B: Benchmarks
* [01:05:52] SWEBench
* [01:17:05] SWEBench/SWEAgent Interview
* [01:27:40] Dataset Contamination Detection
* [01:39:20] GAIA Benchmark
* [01:49:18] Moritz Hart - Science of Benchmarks
[02:36:32] Section C: Reasoning and Post-Training
* [02:37:41] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
* [02:51:00] Let’s Verify Step By Step
* [02:57:04] Noam Brown
* [03:07:43] Lilian Weng - Towards Safe AGI
* [03:36:56] A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
* [03:48:43] MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
[04:00:51] Bonus: Notable Related Papers on LLM Capabilities
Section A: Code Edits and Sandboxes, OpenDevin, and Academia vs Industry — ft. Graham Neubig and Aman Sanger
* Guests
* Graham Neubig
* Aman Sanger - Previous guest and NeurIPS friend of the pod!
* WebArena
*
* Sotopia (spotlight paper, website)
*
* Learning Performance-Improving Code Edits
* OpenDevin
* Junyang Opendevin
* Morph Labs, Jesse Han
* SWE-Bench
* SWE-Agent
* Aman tweet on swebench
* LiteLLM
* Livecodebench
* the role of code in reasoning
* Language Models of Code are Few-Shot Commonsense Learners
* Industry vs academia
* the matryoshka embeddings incident
* other directions
* Unlimiformer
Section A timestamps
* [00:00:00] Introduction to Guests and the Impromptu Nature of the Podcast
* [00:00:45] Graham's Experience in Japan and Transition into Teaching NLP
* [00:01:25] Discussion on What Constitutes a Good Experience for Students in NLP Courses
* [00:02:22] The Relevance and Teaching of Older NLP Techniques Like Ngram Language Models
* [00:03:38] Speculative Decoding and the Comeback of Ngram Models
* [00:04:16] Introduction to WebArena and Zotopia Projects
* [00:05:19] Deep Dive into the WebArena Project and Benchmarking
* [00:08:17] Performance Improvements in WebArena Using GPT-4
* [00:09:39] Human Performance on WebArena Tasks and Challenges in Evaluation
* [00:11:04] Follow-up Work from WebArena and Focus on Web Browsing as a Benchmark
* [00:12:11] Direct Interaction vs. Using APIs in Web-Based Tasks
* [00:13:29] Challenges in Base Models for WebArena and the Potential of Visual Models
* [00:15:33] Introduction to Zootopia and Exploring Social Interactions with Language Models
* [00:16:29] Different Types of Social Situations Modeled in Zootopia
* [00:17:34] Evaluation of Language Models in Social Simulations
* [00:20:41] Introduction to Performance-Improving Code Edits Project
* [00:26:28] Discussion on DevIn and the Future of Coding Agents
* [00:32:01] Planning in Coding Agents and the Development of OpenDevon
* [00:38:34] The Changing Role of Academia in the Context of Large Language Models
* [00:44:44] The Changing Nature of Industry and Academia Collaboration
* [00:54:07] Update on NLP Course Syllabus and Teaching about Large Language Models
* [01:00:40] Call to Action: Contributions to OpenDevon and Open Source AI Projects
* [01:01:56] Hiring at Cursor for Roles in Code Generation and Assistive Coding
* [01:02:12] Promotion of the AI Engineer Conference
Section B: Benchmarks
* Carlos Jimenez & John Yang (Princeton) et al: SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR Oral, Paper, website)
* “We introduce SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.
Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks.
Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.”
* Yonatan Oren et al (Stanford): Proving Test Set Contamination in Black-Box Language Models (ICLR Oral, paper, aman tweet on swebench contamination)
* “We show that it is possible to provide provable guarantees of test set contamination in language models without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchangeable benchmark should be equally likely. In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples.
* We demonstrate that our procedure is sensitive enough to reliably prove test set contamination in challenging situations, including models as small as 1.4 billion parameters, on small test sets of only 1000 examples, and datasets that appear only a few times in the pretraining corpus.”
* Outstanding Paper mention: “A simple yet elegant method to test whether a supervised-learning dataset has been included in LLM training.”
* Thomas Scialom (Meta AI-FAIR w/ Yann LeCun): GAIA: A Benchmark for General AI Assistants (paper)
* “We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency.
* GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92% vs. 15% for GPT-4 equipped with plugins.
* GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer.
*
* Mortiz Hardt (Max Planck Institute): The emerging science of benchmarks (ICLR stream)
* “Benchmarks are the keystone that hold the machine learning community together. Growing as a research paradigm since the 1980s, there’s much we’ve done with them, but little we know about them. In this talk, I will trace the rudiments of an emerging science of benchmarks through selected empirical and theoretical observations. Specifically, we’ll discuss the role of annotator errors, external validity of model rankings, and the promise of multi-task benchmarks. The results in each case challenge conventional wisdom and underscore the benefits of developing a science of benchmarks.”
Section C: Reasoning and Post-Training
* Akari Asai (UW) et al: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR oral, website)
* (Bad RAG implementations) indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation.
* We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) that enhances an LM's quality and factuality through retrieval and self-reflection.
* Our framework trains a single arbitrary LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its generations using special tokens, called reflection tokens. Generating reflection tokens makes the LM controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements.
* Self-RAG (7B and 13B parameters) outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA, reasoning, and fact verification tasks, and it shows significant gains in improving factuality and citation accuracy for long-form generations relative to these models.
* Hunter Lightman (OpenAI): Let’s Verify Step By Step (paper)
* “Even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step.
* We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision.
* To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.
*
* Noam Brown - workshop on Generative Models for Decision Making
* Solving Quantitative Reasoning Problems with Language Models (Minerva paper)
* Describes some charts taken directly from the Let’s Verify Step By Step paper listed/screenshotted above
.
* Lilian Weng (OpenAI) - Towards Safe AGI (ICLR talk)
* OpenAI Model Spec
* OpenAI Instruction Hierarchy: The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Section D: Agent Systems
* Izzeddin Gur (Google DeepMind): A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR oral, paper)
* [Agent] performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias on HTML.
* We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions.
* WebAgent plans ahead by decomposing instructions into canonical sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those.
* We design WebAgent with Flan-U-PaLM, for grounded code generation, and HTML-T5, new pre-trained LLMs for long HTML documents using local and global attention mechanisms and a mixture of long-span denoising objectives, for planning and summarization.
* We empirically demonstrate that our modular recipe improves the success on real websites by over 50%, and that HTML-T5 is the best model to solve various HTML understanding tasks; achieving 18.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation.
* Sirui Hong (DeepWisdom): MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR Oral, Paper)
* We introduce MetaGPT, an innovative meta-programming framework incorporating efficient human workflows into LLM-based multi-agent collaborations. MetaGPT encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. MetaGPT utilizes an assembly line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks into subtasks involving many agents working together.
Bonus: Notable Related Papers on LLM Capabilities
This includes a bunch of papers we wanted to feature above but could not.
* Lukas Berglund (Vanderbilt) et al: The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A” (ICLR poster, paper, Github)
* We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form ''A is B'', it will not automatically generalize to the reverse direction ''B is A''. This is the Reversal Curse.
* The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as ''Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]'' and the reverse ''Who is Mary Lee Pfeiffer's son?''. GPT-4 correctly answers questions like the former 79\% of the time, compared to 33\% for the latter.
*
* Omar Khattab (Stanford): DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines (ICLR Spotlight Poster, GitHub)
* presented by Krista Opsahl-Ong
* “Existing LM pipelines are typically implemented using hard-coded “prompt templates”, i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, or imperative computational graphs where LMs are invoked through declarative modules.
* DSPy modules are parameterized, meaning they can learn how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques.
* We design a compiler that will optimize any DSPy pipeline to maximize a given metric, by creating and collecting demonstrations.
* We conduct two case studies, showing that succinct DSPy programs can express and optimize pipelines that reason about math word problems, tackle multi-hop retrieval, answer complex questions, and control agent loops.
* Within minutes of compiling, DSPy can automatically produce pipelines that outperform out-of-the-box few-shot prompting as well as expert-created demonstrations for GPT-3.5 and Llama2-13b-chat. On top of that, DSPy programs compiled for relatively small LMs like 770M parameter T5 and Llama2-13b-chat are competitive with many approaches that rely on large and proprietary LMs like GPT-3.5 and on expert-written prompt chains.
*
* MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
* Scaling Laws for Associative Memories
* DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
* Efficient Streaming Language Models with Attention Sinks
<150 Early Bird tickets left for the AI Engineer World’s Fair in SF! Prices go up soon.
Note that there are 4 tracks per day and dozens of workshops/expo sessions; the livestream will air <30% of the content this time. Basically you should really come if you dont want to miss out on the most stacked speaker list/AI expo floor of 2024.
Apply for free/discounted Diversity Program and Scholarship tickets here. We hope to make this the definitive technical conference for ALL AI engineers.
Exactly a year ago, we declared the Beginning of Context=Infinity when Mosaic made their breakthrough training an 84k token context MPT-7B.
A Brief History of Long Context
Of course right when we released that episode, Anthropic fired the starting gun proper with the first 100k context window model from a frontier lab, spawning smol-developer and other explorations. In the last 6 months, the fight (and context lengths) has intensified another order of magnitude, kicking off the "Context Extension Campaigns" chapter of the Four Wars:
* In October 2023, Claude's 100,000 token windows was still SOTA (we still use it for Latent Space’s show notes to this day).
* On November 6th, OpenAI launched GPT-4 Turbo with 128k context.
* On November 21st, Anthropic fired back extending Claude 2.1 to 200k tokens.
* Feb 15 (the day everyone launched everything) was Gemini's turn, announcing the first LLM with 1 million token context window.
* In May 2024 at Google I/O, Gemini 1.5 Pro announced a 2m token context window
In parallel, open source/academia had to fight its own battle to keep up with the industrial cutting edge. Nous Research famously turned a reddit comment into YaRN, extending Llama 2 models to 128k context. So when Llama 3 dropped, the community was ready, and just weeks later, we had Llama3 with 4M+ context!
A year ago we didn’t really have an industry standard way of measuring context utilization either: it’s all well and good to technically make an LLM generate non-garbage text at 1m tokens, but can you prove that the LLM actually retrieves and attends to information inside that long context? Greg Kamradt popularized the Needle In A Haystack chart which is now a necessary (if insufficient) benchmark — and it turns out we’ve solved that too in open source:
Today's guest, Mark Huang, is the co-founder of Gradient, where they are building a full stack AI platform to power enterprise workflows and automations. They are also the team behind the first Llama3's 1M+ and 4M+ context window finetunes.
Long Context Algorithms: RoPE, ALiBi, and Ring Attention
Positional encodings allow the model to understand the relative position of tokens in the input sequence, present in what (upcoming guest!) Yi Tay affectionately calls the OG “Noam architecture”. But if we want to increase a model’s context length, these encodings need to gracefully extrapolate to longer sequences.
ALiBi, used in models like MPT (see our "Context=Infinity" episode with the MPT leads, Jonathan Frankle and Abhinav), was one of the early approaches to this space. It lets the context window stretch as it grows, using a linearly decreasing penalty between attention weights of different positions; the further two tokens are, the higher the penalty. Of course, this isn’t going to work for usecases that actually require global attention across a long context.
In more recent architectures and finetunes, RoPE (Rotary Position Embedding) encoding is more commonly used and is also what Llama3 was based on. RoPE uses a rotational matrix to encode positions, which empirically performs better for longer sequences.
The main innovation from Gradient was to focus on tuning the theta hyperparameter that governs the frequency of the rotational encoding.
Audio note: If you want the details, jump to 15:55 in the podcast (or scroll down to the transcript!)
By carefully increasing theta as context length grew, they were able to scale Llama3 up to 1 million tokens and potentially beyond.
Once you've scaled positional embeddings, there's still the issue of attention's quadratic complexity, and how longer and longer sequences impacts models speed and scaling abilities. Getting to 1-4M context window requires a fairly large amount of compute, so efficiency matters.
Ring Attention was the other "one small trick that GPU clouds hate" that improves GPU utilization by allowing parallel computation and communication between GPUs. Gradient started from the EasyContext library as implementation of Ring Attention in PyTorch, since the original one was in JAX.
Long Context Data: Curriculum Learning and Progressive Extension
The use of curriculum learning when extending context was another new approach; rather than training Llama3 on the full 1 million token context from the start, they progressively increased the sequence length over the course of training. Intuitively, it allows the model to first learn to utilize shorter contexts before tackling the full length, but it only works if data gets more and more "tricky" in long context situation.
For the generic pre-training corpus they used SlimPajama as a base, and concatenated texts to reach the target length, while monitoring for diversity in the data. Datasets that only required attending to the last few tokens, for instance, would fail to teach long-range reasoning. To fix that, they used synthetic data (another one of our Four Wars of AI!) with GPT-4 to augment their datasets by prompting it to expand on information or rephrase excerpts. Another paper we previously mentioned in this space is "Rephrasing The Web".
Long Context Benchmarking: Beyond Needles
Long context is cool, but does it work? Greg’s now-famous "needle in a haystack" (NIAH) test, which measures a model's ability to extract a piece of information embedded in a long context, is a clean standard that everyone uses to start, but it is a little simplistic and the community has since created many options to extend it:
* RULER: Outside of various NIAH tests (single value, multiple values, etc) it also tests for things like "most frequent words" and "variable tracking", which is very helpful especially in coding use cases.
* LooGLE: Focuses on three main area: scientific papers, Wikipedia articles, movie and TV scripts. "Timeline reorder" is an interesting challenge in their benchmark, which asks model to create a timeline out of events that happened out of order in the text.
* Infinite Bench: First created in November 2023, most avg input tokens tasks are in the 100-200k tokens range across retrieval, Q&A, and code debugging.
* ZeroSCROLLS: this comes with a public leaderboard where you can see models performance, as well as tasks that you can browse to get an idea.
The 4M context size seemed to be the limit where things started to fall apart as far as performance goes, which is quite impressive!
Show Notes
* Mark Huang
* Gradient
* Chris Chang
* HuggingFace Hub with Llama3 finetunes
* Mad Men
* Crusoe
* Greg Kamradt's Needle in a Haystack
* Chameleon paper
* Charles Goddard (Mentioned in context with model merging)
* Matei Zaharia
* Phil Wang (lucidrains)
* Wing Lian
* Zhang Peiyuan
* Yi
* Scaling Laws of RoPE-based Extrapolation
* ALiBi
* YaRN
* Ring Attention
* Easy Context
* StrongCompute
* LoRa
* RULER: What's the Real Context Size of Your Long-Context Language Models?
* LooGLE: Can Long-Context Language Models Understand Long Contexts?
* Infinite Bench
* BAMBOO
* ZeroSCROLLS: Zero-Shot CompaRison Over Long Language Sequences
* DeepSeek paper
* Multi-head Latent Attention
Chapters
* [00:00:01] Introductions
* [00:01:28] Founding story of Gradient and its mission
* [00:03:50] "Minimum viable agents"
* [00:07:37] Differentiating ML and AI, focusing on out-of-domain generalization
* [00:08:19] Extending Llama3 to 1M tokens
* [00:11:41] Technical challenges with long context sequences
* [00:14:30] Data quality and the importance of diverse datasets
* [00:16:07] What's a theta value?
* [00:18:27] RoPE vs Ring Attention vs ALiBi vs YaARN
* [00:20:23] Why RingAttention matters
* [00:22:47] How to refine datasets for context extension
* [00:27:28] Multi-stage training data and avoiding overfitting to recent data
* [00:28:10] The potential of using synthetic data in training
* [00:31:21] Applying LoRa adapters to extend model capabilities
* [00:34:45] Benchmarking long context models and evaluating their performance
* [00:38:38] Pushing to 4M context and output quality degradation
* [00:40:49] What do you need this context for?
* [00:42:54] Impact of long context in chat vs Docs Summarization
* [00:45:35] Future directions for long context models and multimodality
* [00:48:01] How do you know what research matters?
* [00:50:31] Routine for staying updated with AI research and industry news
* [00:52:39] Deciding which AI developments to invest time in
* [00:56:08] Request for collaboration and data set construction for long context
Transcript
Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO-in-Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI.
Swyx [00:00:14]: Hey, and today we're in the remote studio with Mark Wang from Gradient. Welcome Mark.
Mark [00:00:19]: Hey, glad to be here. It's really a great experience to be able to talk with you all. I know your podcast is really, really interesting and I always am listening to it every time you guys have a release.
Alessio [00:00:31]: He's not a paid actor. He said that out of his own will.
Swyx [00:00:34]: We'll give you the check later. So you're unusual in the sense that you and I go back to college. I don't exactly remember where we overlapped, but you know, we both went to Wharton. We went into the sort of quantitative developer realm.
Mark [00:00:46]: Yeah, exactly. Kind of crazy, right? So it all goes full circle. I was a quant for quite a few years and then made it out into Silicon Valley and now we intersect again when it kind of feels like more or less the same, right? Like the AI wars, the trading wars back in the day too, to a certain extent and the grab for talent.
Swyx [00:01:07]: I think there's definitely a few of us ex-finance people moving into tech and then finding ourselves gravitating towards data and AI. Seems like you did that. You were at a bunch of sort of quant trading shops, but then as you moved to tech, you were a lead data scientist at Box and staff ML scientist at Splunk. And then before working on the startup that eventually became Gradient. You want to tell that story?
Mark [00:01:28]: Yeah, I think part of the reason why I came over from the quant finance world is to get more collaboration, learn about what big data and scaling machine learning really looks like when you're not in this bubble, right? And working at Box, I worked mostly in a cross-functional role, helping product analytics and go to market. And then at Splunk, it was a lot more specific role where I was helping with streaming analytics and search and deep learning. And for Gradient, like really why we started it was whether it was in finance or whether it was in tech, I always noticed that there was a little bit more to give in terms of what AI or ML could contribute to the business. And we came at a really good time with respect to wanting to bring the full value of what that could be into the enterprise. And then obviously, OpenAI created this huge vacuum into the industry to allow for that, right? So I myself felt like really, really empowered to actually ship product and ship stuff that I could think could really help people.
Alessio [00:02:35]: And maybe just to touch a little bit on Gradient, I know we have a lot of things to go through Gradient, Llama3 context extension, there's a lot, but what exactly is Gradient? And you have an awesome design on your website, it's like really retro. I think people that are watching Fallout on Amazon Prime right now can maybe feel nostalgia just looking at it. What exactly is it? Because I know you have the foundry, you have the agent SDK, there's like a lot of pieces into it.
Mark [00:03:00]: Yeah, for sure. And appreciate the call out for the design. I know my co-founder, Chris, spent a lot of thought in terms of how he wanted the aesthetic to look like. And it reminds me a lot about Mad Men. So that was the initial emotional shape that I felt when I saw it. Quite simply, Gradient, we're a full stack AI platform. And what we really want to do is we want to enable all of the RPA workloads or the codified automation workloads that existed in enterprise before. We really want to enable people to transition into more autonomous, agentic workflows that are less brittle, feel more seamless as an interface to able to empower what we really think the new AI workforce should look like. And that kind of required us to build a fairly horizontal platform for those purposes.
Alessio [00:03:50]: We have this discussion in our AI in Action club on Discord, like the minimum viable agent or like kind of how you define an agent. In your mind, what is like the minimum thing that you can call actually an agent and not just like a for loop? And how do you see the evolution over time, especially as people adopt it more and more?
Mark [00:04:08]: So I kind of stage it where everybody, first of all, at the lowest level thinks about like non-determinism with respect to how the pipeline looks like when it's executed. But even beyond that, this goes back into effectively evaluations. It's like on each stage of the node, you're going to have to see a marginal improvement in the probability of success for that particular workload because of non-determinism. So I think it is an overloaded term to a certain extent because like everything is an agent if it calls a language model or any sort of multimodal model these days. But for us, it's like, you know, my background is statistics. So I want to see improvements in the probability of the success event or outcome happening because of more nodes.
Swyx [00:04:52]: Yeah, I think, you know, the one thing that makes this sort of generative AI era very different from the sort of data science-y type era is that it is very non-deterministic and it's hard to control. What's the founding story of Gradient? Like of all the problems that you chose, why choose this one? How did you get together your co-founders, anything like that, bring us up to the present day?
Mark [00:05:13]: Yeah. So one of my co-founders is Chris and he's a really good friend of mine as well. I don't know if you intersected with him at Penn as well, but... Chris Chang? Yeah, yeah. Chris Chang, who did banking for maybe one or two years and then, you know, was a software engineer at Meta, also was at Google. And then most recently, he was like a director at Netflix and product. And we always wanted to do something together, but we felt what really came to fruition was wanting to develop something that is enterprise facing for once, mostly because of our experience with internal tooling, inability for something to like basically exist through like a migration, right? All the time with every ML platform that I've ever had to experience or he had to experience, it's like a rebuild and you rip it out and you have a new workflow or automation come in and it's this huge multi-quarter, maybe even multi-year project to do that. And we also teamed up with former coworker Chris's from Open Door Forest, who was also on Google Cloud Platform and him seeing the scale and actually the state of the art in terms of Google was using AI for systems before everybody else too, right? They invented a transformer and their internal set of tooling was just so far superior to everything else. It's really hard for people to go back after seeing that. So what we really wanted was to reduce that friction for like actually shipping workloads in product value when you have all these types of operational frictions that happen inside of these large enterprises. And then really like the main pivot point for all of it was like you said, things that can handle out of domain problems. So like out of domain data that comes in, having the flexibility to not fall over and having something that you build over time that continues to improve. Like machine learning is about learning and I feel like a lot of systems back in the place, they were learning a very specific objective function, but they weren't really natively learning with the user. So like that's the whole, you know, we use the term assistant all the time, but my vision for the assistant was always for the system to grow alongside me, right? Almost like an embodied second limb or something that will be able to get better as you also learn yourself.
Swyx [00:07:37]: Yeah. You know, people always trying to define the difference between ML and AI. And I think in AI, we definitely care a lot more about out of domain generalization and that's all under the umbrella of learning, but it is a very specific kind of learning. I'm going to try to make a segue into today's main topic of conversation that's something that you've been blowing up on, which is the long context learning, right? Which is also some form of out of distribution generalization. And in this context, you're extending the context window of an existing open source model. Maybe if you want to like just bring us all the way back to it, towards like why got you interested in long context? Why did you find it like an interesting investment to work on? And then the story of how you did your first extensions.
Mark [00:08:19]: For Llama3, it's specifically, we chose that model because of the main criticisms about it before, when it first got released, 8,000 context lengths just seemed like it was too short because it seemed like Mistral and even Yi came out with like a 2,000 token context length model. Really, the inception of all of it was us fine tuning so many models and working on regs so much and having this, and it still exists today, this basically pedagogical debate with everybody who's like, Hey, is it fine tuning versus reg? Is it this versus that? And at the end of the day, it's just all meta learning, right? Like all we want is like the best meta learning workflow or meta learning set up possible to be able to adapt a model to do anything. So naturally, long context had a place in that, but nobody had really pushed the limits of it, right? You would see like 10 shot, maybe 100 shot prompting or improving the model's capabilities, but it wasn't until Google comes out with Gemini with the first 1 million context length model that a lot of people's jaws dropped in that hunger for understanding what that could really facilitate and the new workflows came about. So we're staged to actually train other open source models to do that. But the moment Llama3 came out, we just went ham against that specific model because the two things that were particularly appealing for that was the fact that I see a lot of these language models as compression algorithms to a certain extent, like the way we have 15 trillion tokens into a specific model. That definitely made me feel like it would have a lot of capabilities and be more adaptable towards extending that context length. So we went in there and the 1 million number, that was more of just like, put the North Star up there and see if we can get there and then see what was happening along the way as we did that. So also shout out to Crusoe who facilitated all that compute because I would be lying if I was to say like, anyone could just go out and do it. It does require quite a bit of compute. It requires a lot of preparation, but all the stars kind of aligned for that moment for us to go after that problem.
Swyx [00:10:32]: I'll take a side note on Crusoe since you just brought it up. Yeah. Like, can you explain what Crusoe is? I have this mental image of putting GPUs on top of oil rigs. What is it? What do they do? How do you work with them? You know, just anything nice. I'm sure they appreciate nice things that you say about them. Oh, for sure.
Mark [00:10:48]: For sure. So they came to us through a collaborative effort where we basically were in search of a GPU provider. I don't want to call cloud service provider quite yet because then, you know, you think about hyperscalers. But for them, you know, they're one of the biggest alternative GPU cloud providers. And they were offering up, like, we want to do a collaboration to showcase their technology. And it just made it really easy for us to, like, scale up with their L40Ss. And those are the specific GPU instances we used and coordinating that effort with them to get that dedicated cluster first to do the project. It became a really good relationship. And we still work with them today because, like, we're trying to evaluate more of these models and possibly train more of them. And anyone could go up to them and basically get your compute from them. And they have a lot of GPUs available for those type of projects.
Alessio [00:11:41]: I would love to maybe have you run people through why the models don't come with longer context sequences out of the box. Like, obviously, you know, the TLDR is like self-attention is like quadratic scaling of memory. So the longer the context size, the more compute you have to spend the training time. And that's why you have to get Crusoe to help you extend it. How do you actually train large language model that is like a very long context? And then how does that differ from just tacking it on on top later? And then maybe we'll dive into performance and some of those things. But I think for a lot of folks in our audience that are more AI engineers, they use models, but don't necessarily build the models themselves. A lot of time, it's hard to understand what goes into actually making a long context model.
Mark [00:12:23]: Yeah, in terms of, you know, all the literature out there, I would say, honestly, it's probably still TBD as to like the trade offs between the approach we did, which is more of a curriculum learning approach after the fact versus inherently training a model with a long context throughout, because I just don't think people have looked at the scaling properties of it in deep detail. But as stylistic facts exist out there with research papers from meta themselves, actually, they've already shown in a paper that if you train a model on a shorter context, and you progressively increase that context to like, you know, the final limit that you have, like 32k is usually the limit of Lama 2 was that long. It actually performs better than if you try to train 32k the whole time. And I like to think about it intuitively, as if you're trying to learn probability theory, you're not going to go and read the book cover to cover and then do all the exercises afterwards, what you're going to do is you're going to do each chapter, do an exercise, read the chapter, do an exercise, and then finish right with the final set of like holistic exercises, or examination. So attention is exactly what it sounds like, to a certain extent, you have a bunch of indices, and you are making the model attend to localize contexts and concepts across the entirety of its encoding, right, like whatever the text that the sequence that you're giving it. So when you're doing the curriculum learning aspect of things, you are kind of trying to give it the opportunity to also attend to all the concepts. So data actually, in the creation of that context, plays a huge role, because a lot of times people make the mistake of trying to extend the context length by just giving it raw text that doesn't have the necessity for the model to go all the way in the beginning of the sequence, and then connect an idea to the end of the sequence.
Alessio [00:14:30]: So data quality is one thing, but it sounds like what is the work like the 1 million context if Llama3 was 2k context size, like, is there like a minimum context size that you need to then be able to generalize? Or does it not not really matter in defined tuning kind of takes care of it?
Mark [00:14:47]: There's no minimum, I would say, or at least, I can't make such a strong statement as to say that that does not exist. But if you have a 4k, any regular model out there, like you can progressively increase the context length of it so long as it has shown really good perplexity scores prior to your context length extension. So if it hasn't shown good perplexity, you basically can't even predict the next token, you're kind of out of luck, right? But then from there, the other component that we actually just released a blog on maybe last Friday, it's like you got to pay attention to the theta value that the model starts off with. What was fairly unique about the Llama3 model was their choice of the theta parameter, which gave some suspicion as to how long the context could be extended for the model. So that aspect of we can go into, you know, a huge lesson in terms of positional encodings and in rope scaling and stuff. But those concepts and that aspect of things enables you to scale out the length much more easily.
Alessio [00:15:55]: What's the TLDR of what the theta is for a model? If I haven't built a model before? Yeah. I mean, obviously, I know what it is. But for people that don't know, right, I'm totally an expert.
Mark [00:16:07]: So not all models have it. But you know, some models will employ rope scaling. And Llama3 does that. But there's also other positional encoding and embedding mechanisms that other models employ. But TLDR is, if you think about most architectures, they employ, it's kind of like a sine or cosine curve. And you're thinking about, you know, you have the amplitudes that occur there to allow for the model to like, see different types of distributions of data. Really what the theta value does is it governs like, how often like a pattern is going to appear in the embedding space, you basically are able to shift that rotational curve by increasing the theta value and allow for different types of distributions to be seen as if they actually occurred in the training data before. It's super confusing. But it's like, there's positional extrapolation, and then there's interpolation, you want interpolation, it's been shown that just pure extrapolation makes the model a lot worse, and it's harder to attend to stuff. Whereas the interpolation is like you're squeezing everything back in to what the original contact length was to a certain extent, and then allowing for it to overlap different sequences that it's already seen, as if it actually occurred when you see a million contexts of sequence tokens. So yeah, I think that aspect, we didn't know how well it would scale. I think that's one thing. So like, I'm not gonna lie and tell you like, right off the bat, we're like, we're definitely gonna hit a million. It was more like, we're getting to 256 and it looked good. We did our evals, we scaled it more. And then what was really good was that we established the formula at the start. So like, it's actually a formula that we actually took from the paper, I think it's the rope scaling paper. And we looked at that particular formula, and then we backed out the values. And it's all empirical. So like, it's not like a mathematical tautology or proof, it's an empirical formula that actually worked really well. And then we just kept scaling it up and it held. It's kind of like the scaling laws, you know, the scaling laws exist, but you don't know if they're going to continue.
Swyx [00:18:27]: Yeah. Like, are you able to compare it with like other forms of scaling that people have been talking about? Alibi comes to mind, yarn is being talked about a lot by a news research. And then there's other forms which are like, not exactly directly related, but like ring attention comes up a lot that we had a really good session with StrongCompute in the Latent Space Discord talking about all these approaches. I just wonder if you want to compare and contrast like rope versus the other stuff.
Mark [00:18:51]: Yeah, I think Alibi, we haven't compared with that one specifically, mostly because I've noticed some of the newer architectures don't actually employ it a lot. I think the last architecture that actually really employed it was the Mosaic MPT model class. And then almost all the models these days are all rope scaling. And then effectively, you can use yarn with that as well. We just did the theta scaling specifically because of its empirical elegance, really easy and like it was well understood by us. The other one that I know that in the open source that people are applying, which uses more of a LoRa based approach, which is really interesting too, is the one that Wing has been employing, which is Pose. We sort of help them evaluate some of the models. With respect to like the performance of it, it does start to break down a little bit more on the longer, longer context. So like 500,000 to a million, it appeared that it doesn't hold as well specifically for like needle in the haystack. It's still TBD as evaluations. It's a sparse high dimensional space where you're just like evaluating performance across so many different things and then trying to map it back to like, hey, here's the thing that I actually cared about from the start and I have like a thousand different evaluations and they tell me something but not the entire picture. And as for like ring attention specifically, we employed ring attention in order to do the training. So we combined flash attention and ring attention together with like a really specific network topology on our GPUs to be able to maximize the memory bandwidth. Yeah.
Swyx [00:20:23]: As far as I understand, like ring attention, a lot of people credit it for Gemini's million token context, but actually it's just a better utilization of GPUs. Like, yeah, that's really what it is. You mentioned in our show notes, Zhang Peiyuan's easy context repo. I have seen that come up quite a bit. What does that do as, you know, like how important is it as ring attention implementation? I know there's like maybe another one that was done by Lucid Reins or one of the other open source people. But like, what is easy context? Is that the place to go? Like, did you evaluate a bunch of things to implement ring attention?
Mark [00:20:53]: Yeah, we evaluated all of them. I would say the original authors, you know, Matei and all the folks at Berkeley, they created the JAX implementation for it. And unfortunately, not to discredit, you know, TPUs or whatever, like the JAX implementation just does not work on GPUs very well. Like any naive setup that you do, like it just won't run out of the box very easily. And then unfortunately, that was probably the most mature repo with a lot more configurations to set up interesting network topologies for your cluster. And then the other PyTorch implementations outside of easy context, they just didn't really work. Maybe we weren't implementing one small aspect incorrectly, but like, there was an active development on it at a certain point, like even lucidrains, I think he's interesting because for once he was actually like, he was like taking a job somewhere and then just stopped doing commits. And as we were working to try to find it, we never really want to jump in on a repo where someone's like kind of actively committing breaking changes to it. Otherwise, we have to like eat that repo ourselves. And easy context was the first PyTorch implementation that applied it with native libraries that worked pretty well. And then we adapted it ourselves in order to configure it for our cluster network topology. So you know, shout out to Zhang Peiyuan for his open source contributions. I think that we look forward to possibly collaborating him and push that further in the future because I think more people if they do want to get started on it. I would recommend that to be the easiest way unless you want to, like, I don't know how many people know Jax. Me personally, I don't really know it that well. So I'm more of a PyTorch guy. So I think he provides a really good introduction to be able to try it out.
Alessio [00:22:47]: And so once you had the technical discovery, what about the actual customer interest, customers that you work with? I feel like sometimes the context size can be a bit of a marketing ploy, you know, people are like, oh, yeah, well, no, 1 million, 2 million, 3 million, 4 million. That's kind of the algorithms side of it. How do you power the training? But the other side is obviously the data that goes into it. There's both quantity and quality. I think that's how one of your tweets, you trained on about 200 million tokens for the AP model to the context extension. But what are the tokens? You know, how do you build them? What are like maybe some of the differences between pre-training data sets and context extension data sets? Yeah, any other color you give there will be great.
Mark [00:23:30]: So specifically for us, we actually staged two different updates to the model. So our initial layer that we trained was just basically like a pre-training layer. So continual pre-training where we took the slim pajamas data, and then we filtered it and concatenated it so that it would reach the context lengths that we were trying to extend out to. And then we took the UltraChat data set, filtered it down, or maybe some other, you know, second order derivative of the UltraChat data set that was curated in, and then filtered it down and then reformatted it for our chat use case. For those two data sets, you always have to really keep in mind for the pre-training data, whether or not you may be like cutting off tokens in weird ways, whether or not, you know, the content is actually diverse enough to retain the ability of the model. So slim pajamas tends to be one of the best ones, mostly because it's a diverse data set. And you can use embeddings too as a pre-filtering step as well, right? Like how diverse are your embeddings space to the original corpus of the model, and then train on top of that to retain its abilities. And then finally, for the chat data set, making sure that it's attending to all the information that would be expected to really stretch its capabilities, because you could create like a long context data set where every single time the last 200 tokens could answer the entire question, and that's never going to make the model attend to anything. So it's even something that we're doing right now is trying to think about like, how do we actually improve these models? And how do you ablate the data sets such that it can expose like even more nuanced capabilities that aren't easily measurable quite yet?
Alessio [00:25:26]: Is there a ratio between diversity of the data set versus diversity compared to what the model already knows? Like does the model already need to understand a good part of the new like the context extension data to function? Like can you put context extension data set that is like very far from like what was in the pre training? I'm just thinking as as the model get older, some of the data sets that we have might not be in the knowledge of the existing model that you're trying to extend.
Mark [00:25:54]: I think that's always a consideration. I think specifically, you really got to know how many tokens were expended into that particular model from the start. And all models these days are now double digit trillions, right? So it's kind of a drop in the bucket, if you really think I can just put, you know, a billion tokens in there. And I actually think that the model is going to truly learn new information. There is a lot of research out there between the differences with respect to full fine tuning, which we applied full fine tuning versus lower base fine tuning. It's a trade off. And my opinion of it is actually that you can test certain capabilities and you can kind of inject new knowledge into the model. But to this day, I've not seen any research that does like a strong, well scaled out empirical study on how do you increase the model's ability to understand like these decision boundaries with a new novel data. Most of it is holding on a portion of the data as like novel and then needing to recycle some of the old knowledge. So it just doesn't forget and get worse at everything else, right? Which was seen. We do have historical precedent, where the original code bomb was trained further from Mama 2, and it just lost all its language capability, basically, right? So I don't want to call that project like deem it as a failure, but it wasn't a really successful generalization exercise, because, you know, these models are about flexibility and being like generic to a certain extent.
Swyx [00:27:28]: One thing I see in the recent papers that have been coming out is this sort of concept of multi-stage training data. And if you're doing full fine tuning, maybe the move or the answer is don't train 500 billion tokens on just code, because then yeah, it's going to massively overfit to just code. Instead, maybe the move is to slowly change the mix over the different phases, right? So in other words, you still need to mix in some of your original source data set to make sure it doesn't deviate too much. I feel like that is a very crude solution. Maybe there's some smarter way to adjust like the loss function so that it doesn't deviate or overfit too much to more recent data. It seems like it's a solvable thing. That's what I'm saying. Like this overfitting to more recent data issue.
Mark [00:28:10]: Yeah, I do think solvable is hard. I think provably solvable is always something that I know is extremely difficult, but from a heuristical standpoint, as well as like having like some sort of statistical efficiency on like how you can converge to the downstream tasks and improve the performance that way in a targeted manner, I do think there are papers that try to do that. Like the Do-Re-Mi paper, I think it was released last year, it was really good about doing an empirical study on that. I think the one thing people struggle with though, is the fact that they always try to do it on pretty naive tasks. Like you target like a naive task, and then you create your data mixture and you try to show some sort of algorithm that can retain the performance for those downstream tasks. But then what do we all care about are actually like really, really interesting, complex tasks, right? And we barely have good evaluations for those. If you do a deep dive at the Gemini 1.5 technical paper, which they just updated, it was a fantastic paper with new updates. If you look at all of their long context evaluations there, like a lot of them are just not something that the open community can even do, because they just hired teachers to evaluate whether or not this model generated a huge lesson plan that is really coherent. Or like you hire a bunch of subject matter experts, or they taught the model how to do language translation for extinct language where only 200 people in the world know. It's kind of hard for us to do that same study as an early stage startup.
Swyx [00:29:50]: I mean, technically, now you can use Gemini as a judge, Gemini is touting a lot of their capabilities and low resource languages. One more thing before on that sort of data topic, did you have any exploration of synthetic data at all? You know, use Mistral to rephrase some existing part of your data sets, generate more tokens, anything like that, or any other form of synthetic data that you choose to mention? I think you also mentioned the large world model paper, right?
Mark [00:30:13]: We used GPT-4 to rephrase certain aspects of the chat data, reformatting it or kind of generating new types of tokens and language and types of data that the model could see. And also like trying to take the lower probability, right, or the lower correlated instances of out of domain data in that we wanted to inject it to the model too, as well. So I actually think a lot of the moat is in the data pipeline. You'll notice most papers just don't really go into deep detail about the data set creation because, I mean, there's some aspects that are uninteresting, right? Which is like, we paid a bunch of people and generated a lot of good data. But then the synthetic data generating pipeline itself, sometimes that could be like 25% or 50% of the entire data set that you've been used to depreciating.
Swyx [00:31:08]: Yeah, I think it's just for legal deniability.
Swyx [00:31:13]: No, it's just too boring. You know, I'm not going to say anything because it's too boring. No, it's actually really interesting. But in fact, it might be too interesting. So we're not going to say anything about it.
Alessio [00:31:21]: One more question that I had was on LoRa and taking some of these capabilities out and bringing them to other model. You mentioned Weng's work. He tweeted about we're going to take this LoRa adapter for the Gradient 1 million context extension, and you're going to be able to apply that to other model. Can you just generally explain to people how these things work with language models? I think people understand that with stable diffusion, you have these LoRa patches for different types of styles. Does that work similarly with LLMs? And is it about functionality? Can you do LoRa patches with specific knowledge? What's the state of the art there?
Mark [00:31:58]: Yeah, I think there's a huge resurgence in what I would call model alchemy to a certain extent, because you're taking all of these LoRa's and you're mixing them together. And then that's a lot of the model merging stuff that I think Charles Goddard does and a lot of others in the open community, right? Because it's a really easy way. You don't need training, and you can test and evaluate models and take the best skills and mix and match. I don't think there has been as much empirical study, like you're saying, for how shows the same type of... It's not as interpretable as stable diffusion to a certain extent. Because even we have experimented with taking deltas in the same methodology as Wing, where we'll take a delta of an already trained model, try to see how that has created, in a sense, an ROHF layer, right? Taking the LLAMA instruct layer, subtracting the base model from that, and then trying to apply that LoRa adapter to another model and seeing what it does to it. It does seem to have an effect, though. I will not lie to say I'm really surprised how effective it is sometimes. But I do notice that for more complex abilities, other than more stylistic stuff, it kind of falls through. Because maybe it requires a much deeper path in the neural network, right? All these things, these weights are just huge trees of paths that the interesting stuff is the road less traveled, to a certain extent. And when you're just merging things brute force together that way, you don't quite know what you'll get out all the time. There's a lot of other research that you have merged ties and you have all these different types of techniques to effectively just apply a singular value decomposition on top of weights and just get the most important ones and prevent interference across all the other layers. But I think that that is extremely interesting from developer community. And I want to see more of it, except it is to a certain extent, kind of polluting the leaderboards these days because it's so targeted. And now you can kind of game the metric by just finding all the best models and then just merging them together to do that. And I'll just add one last bit is basically the most interesting part about all that actually to me is when people are trying to take the lowers as a way of like, short circuiting the training process. So they take the lowers, they merge it in, and then they'll fine tune afterwards. So like the fine tuning and the reinitialization of a little bit of noise into all the new merged models provides like kind of a learning tactic for you to get to that capability a little bit faster.
Swyx [00:34:45]: There's a lot there. I really like the comparison of ties merging to singular value decomposition. I looked at the paper and I don't really think I understood it on that high level until you just said it. We have to move on to benchmarking. This is a very fun topic. Needle in a haystack. What are your thoughts and feelings? And then we can discuss the other benchmarks first, but needle in a haystack.
Mark [00:35:04]: You want to put me on the spot with that one? Yeah, I think needle in a haystack is definitely like the standard for presenting the work in a way that people can understand and also proving out. I view it as like a primitive that you have to pass in order to give the model any shot of doing something that combines both like a more holistic language understanding and instruction following, right? Honestly, like it's mostly about if you think about the practical applications of long context and what people complain most about models when you stuff a lot of context into it is either the language model just doesn't care about what you asked it to do, or it cannot differentiate context that you want it to use as a source to prevent hallucination versus like instructions. I think that when we were doing it, it was to make sure that we were on the right track. I think Greg did a really great job of creating metric and a benchmark that everybody could
Swyx [00:36:00]: understood.
Mark [00:36:00]: It was intuitive. Even he says himself, we have to move past it. But to that regard, it's a big reason why we did the evaluation on the ruler suite of benchmarks, which are way harder. They actually include needle in the haystack within those benchmarks too. And I would even argue is more comprehensive than the benchmark that Gemini released for their like multi-needle in the haystack. Yeah.
Swyx [00:36:26]: You mentioned quite a few. You mentioned RULER, LooGLE, infinite bench, bamboo, ZeroSCROLLS. Do you want to give us maybe two or three of those that you thought were particularly interesting or challenging and what made them stand out for you?
Mark [00:36:37]: There's just so many and they're so nuanced. I would say like, yeah, zero scrolls was the first one I'd ever heard of coming out last year. And it was just more of like tracking variable over long context. I'll go into ruler because that's the freshest in my mind. And we're just scrutinizing it so much and running the evaluation in the previous two
Swyx [00:36:56]: weeks.
Mark [00:36:56]: But like ruler has four different types of evaluations. So the first one is exactly needle in the haystack. It's like you throw multiple needles. So you got to retrieve multiple key value pairs. There's another one that basically you need to differentiate.
Swyx [00:37:13]: Multi-value, multi-query. Yeah, yeah.
Mark [00:37:15]: Multi-value, multi-query. That's the ablation. There's also a variable tracking one where you go, hey, if X equals this, Y equals this, Y equals Z, like what is this variable? And you have to track it through all of that context. And then finally, there's one that is more of like creating a summary statistic. So like the common words one, where you choose a word that goes across the entire context, and then you have to count it. So it's a lot more holistic and a little bit more difficult that way. And then there's a few other ones that escaped me at this moment. But ruler really pushes you. If I think about the progression of the evaluations, it start to force the model to actually understand like the totality of the context. Like everybody argues to say, couldn't I just use like a retrieval to like just grab that variable rather than pay $10 for one shot or something? Although it's not as expensive. The main thing that I struggled with, with even some of our use cases, were like when the context is scattered across multiple documents, and you have like really delicate plumbing for the retrieval step. But it only works for that one, that really specific instance, right? And then you throw in other documents and you're like, oh, great, my retrieval doesn't grab the relevant context anymore. So that's the dream, right? Of getting a model that can generalize really well that way.
Swyx [00:38:38]: Yeah, totally. And I think that probably is what Greg mentioned when saying that he has to move beyond Needle and Haystack. You also mentioned you extended from 1 million to 4 million token context recently. And you saw some degradation in the benchmarks too. Like you want to discuss that?
Mark [00:38:53]: So if you look at our theta value at that point, it's getting really big. So think about floating point precision and think about basically now you're starting to run into problems where in a deep enough network and having to do joint probabilities across so many tokens, you're hitting the kind of the upper bound on accuracy there. And there's probably some aspect of clamping down certain activations that we need to do within training. Maybe it happens at inference time as well with respect to like the theta value that we use in how do we ensure that it doesn't just explode. If you've ever had to come across like the exploding gradients or the vanishing gradient problem, you will know what I'm talking about. A lot of the empirical aspect of that and scaling up these things is experimentation and figuring out how do you kind of marshal these really complicated composite functions such that they don't just like do a divide over zero problem at one point. Awesome.
Alessio [00:39:55]: Just to wrap, there's the evals and then there's what people care about. You know, there's two things. Do you see people care about above 1 million? Because Jem and I had the 2 million announcement and I think people were like, okay, 1 million, 2 million, it's whatever. Like, do you think we need to get to 10 million to get people to care about again?
Swyx [00:40:13]: Yeah.
Alessio [00:40:14]: Do we need to get to 100 million?
Mark [00:40:16]: I mean, that's an open question. I would certainly say a million seemed like the number that got people really excited for us. And then, you know, the 4 million is kind of like, okay, rather than like a breakthrough milestone, it's just the next incremental checkpoint. I do think even Google themselves, they're evaluating and trying to figure out specifically, how do you measure the quality of these models? And how do you measure and map those to capabilities that you care about going down the line?
Swyx [00:40:49]: Right.
Mark [00:40:49]: And I think us as a company, we're figuring out how to saturate the context window in a way that's actually adding incremental value. So the obvious one is code because code repositories are huge. So like, can you stuff the entire context of a repo into a model and then make it produce some module that is useful or some suggestion that is useful? However, I would say there are other techniques like, you know, alpha coding and flow engineering that if you do iterative things in a more agentic manner, it may actually produce better quality. I would preface and I would actually counter that maybe start off with the use case that people are more familiar with right now, which is constantly evolving context in like a session. So like, whereas you're coding, right? If you can figure out evals that actually work where you're constantly providing it multiple turns in each incremental turn has a nuance aspect and you have a targeted generation that you know of making the model track state and have state management over time is really, really hard. And it's an incredibly hard evaluation will probably only really work when you have a huge context. So that's sort of what we're working on trying to figure out those types of aspects. You can also map that. It's not just code state management exists. You know, we work in the finance sector a lot, like investment management, having a state management of like a concept and stuff that evolves over like a long session. So I'm super excited to hear what other people think about the longer context. I don't think Google is probably investing to try to get a billion quite yet. I think they're trying to figure out how to fully leverage what they've done already.
Alessio [00:42:39]: And does this change in your mind for very long chats versus a lot of documents? The chat is kind of interactive, you know, and information changes. The documents are just trying to synthesize more and more things. Yeah. Any thoughts on how those two workloads differ?
Mark [00:42:54]: I would say like with the document aspect of things, you probably have a little bit more ability to tweak other methodologies. You can get around the long context sometimes where you can do retrieval augmented generation or you do hierarchical recursive summarization, whereas evolution in like a session, because that state variable could undergo pretty rapid changes. It's a little bit harder to you getting around that without codifying a really specific workflow or like some sort of state clause that is going back to like determinism. Right. And then finally, what I really think people are trying to do is figure out how did all these shots progress over time? How do you get away from the brittleness of the retrieval step? If you shove in a thousand shots or 2000 shots, will it just make the retrieval aspect of good examples irrelevant? Kind of like a randomly sampling is fine at that point. There's actually a paper on that that came out from CMU that they showed with respect to a few extraction or classification, high cardinality benchmarks, they tracked fine tuning versus in context learning versus many, many shot in context learning. And they basically showed that many, many shot in context learning helps to prevent as much sensitivity around the examples themselves, right? Like the distraction error that a lot of LLMs get where you give it irrelevant context and it literally can't do the task because it gets sort of like a person too, right? Like you got to be very specific about, I don't want to distract this person because then they're going to go down a rabbit hole and not be able to complete the task. Yeah.
Alessio [00:44:37]: Well, that's kind of the flip side of the needle in a haystack thing too in a bit. It's like now the models pay attention to like everything so well. Like sometimes it's hard to get them to like, I just said that once, please do not bring that up again. You know, it happens to me with code. Yeah. It happens to me with like CSS style sometimes or like things like that. If I have a long conversation, it tries to always reapply certain styles, even though I told it maybe that's not the right way to do it. But yeah, there's a lot again of empirical that people will do. And just, I know we kind of went through a lot of the technical side, but maybe the flip side is why is it worth doing? What are like the use cases that people have that make long context really useful? I think you have a lot of healthcare use cases. I saw on your Twitter, you just mentioned the finance use case, obviously some of the filings and documents that companies publish can be quite worthy. Any other things that you want to bring up, maybe how people are using gradient, anything like that, I think that will help have a clearer picture for people. Yeah.
Mark [00:45:35]: So beyond just using the context for, you know, sessions and evolving state management, it really comes down to something that's fairly obvious, which everybody's trying to do and work on is how do you ground the language model better? So I think when you think pure text, that's one thing, but then multimodality, it's going to be pivotal for long context, just because videos, when you're getting into the frames per second, and you're getting into lots of images and things that are a lot more embodied, you need to utilize and leverage way more, way more tokens. And that is probably where, you know, us as a company, we're exploring more and trying to open up the doors for a lot more use cases because I think in financial services, as well as healthcare, we've done a good job on the tech side, but we still need to push a little bit further when we combine, you know, a picture with words, like a chart with words or somebody's medical image with words, stuff like that. You definitely can do a better job. You know, it's timely too, because Meta just released the new chameleon paper that does multimodal training, and it shows that early fusion is more sample efficient, right? So having that kind of view towards the future is something that we want to be primed to do because, you know, it's similar to what Sam Altman says himself too, right? You need to just assume that these models are going to be 10x better in the next few years. And if you are primed for that, that's where you have kind of a business that, you know, you're not just pivoting after every release or every event, you know, that drops.
Swyx [00:47:12]: I think the thing about this 10x issue is that the 10x direction moves all the time. You know, some people were complaining about GPT-4.0 that the ELO scores for GPT-4.0 actually in reality, weren't that much higher than GPT-4.0 Turbo. And really the, you know, so it's not 10x better in reasoning, it's just 10x better in the integration of multiple modalities. By the way, look over here, there's a really sexy voice chat app that they accidentally made that they had to deprecate today. The 10x direction keeps moving. Now it's like, you know, fully in like sort of multi-modality land, right? And so can 10x in various ways, but like you, you guys have 10x context length, but like, are we chasing the last war? Because like, now like nobody cares about context length, now it's like multi-modality time, you know? I'm joking, obviously people do care about it. I wonder about this, how this comment about this 10x thing every single time.
Mark [00:48:01]: You know, that's honestly why we kind of have our eye on the community as well as you, right? Like with your community and the things that you hear, you know, you want to build where, you know, we're a product company, we're trying to build for users, trying to listen to understand what they actually need. Obviously, you know, you don't build everything that people ask you to build, but we know what's useful, right? Because I think that you're totally right there. If we want to make something 10x better in a certain direction, but nobody cares and it's not useful for somebody, then it wasn't really worth the while. And if anything, maybe that's the bitter lesson 2.0 for so many tech startups. It's like build technology that people care about and will actually 10x their value rather than build technology that's just 10x harder.
Swyx [00:48:48]: I mean, that's not a bitter lesson. That's just Paul Graham.
Swyx [00:48:53]: One more thing on the chameleon paper. I was actually just about to bring that up, you know? So on AI News, my daily newsletter, it was literally my most recent featured paper. And I always wonder if you can actually sort of train images onto the same latent space as words. That was kind of done with like, you know, what we now call late fusion models with lava and flamingo and, you know, all the others. But now the early fusion models like chameleon seem to be the way forward. Like obviously it's more native. I wonder if you guys can figure out some kind of weird technique where you can take an existing Lama 3 model and early fuse the images into the text encoder so that we just retroactively have the early fusion models. Yeah.
Mark [00:49:34]: Even before the chameleon paper came out, I think that was on our big board of next to do's to possibly explore or our backlog of ideas, right? Because as you said, even before this paper, I can't remember. I think Meta even had like a scaling laws for multimodality paper that does explore more early fusion. The moment we saw that, it was just kind of obvious to us that eventually it'll get to the point that becomes a little bit more mainstream. And yeah, that's a cool twist that we've been thinking about too as well, as well as other things that are kind of in the works that are a little bit more agentic. But if open collaboration interests you, we can always work on that together with the
Swyx [00:50:14]: community. Okay. Shout out there. You can leave that in the call to action at the end. We have a couple more questions to round this out. You mentioned a lot of papers in your work. You're also building a company. You're also looking at open source projects and community. What is your daily or weekly routine to keep on top of AI?
Mark [00:50:31]: So one, subscribe to AI News. He didn't have to pay me to say that. I actually really think it's a good aggregator. I think it's a good aggregator.
Swyx [00:50:40]: I'll tell you why.
Mark [00:50:41]: Most of the fastest moving research that's being done out there, it's mostly on Twitter. I wasn't a power Twitter user at all before three years ago, but I had to use it and I had to always check it in order to keep on top of early work that people wanted to talk about or present. Because nothing against submitting research papers to like ICLR or ICML, knowing the state of the art, those are like six months late, right? People have already dropped it on archive or they're just openly talking about it. And then being on Discord to see when the rubber hits the road, right? The implementations and the practices that are being done or the data sets, like you said. A lot of conversations about really good data sets and how do you construct them are done in the open in figuring that out. For people that don't have budgets of like $10 million, you just pay a bunch of annotators. So my routine daily is like, second thing I do when I wake up is to look on Twitter to see what the latest updates are from specific people that do really, really great work. Armin at Meta who did the chameleon paper, everything he writes on Twitter is like gold. So anytime he writes something there, I really try to figure out what he's actually saying there and then tie it to techniques and research papers out there. And then sometimes I try to use certain tools. I myself use AI itself to search for the latest papers on a specific topic, if that's the thing, on the top of my mind. And at the end of the day, trying out the products too. I think if you do not try out the tooling and some of the products out there, you are missing out on someone's compression algorithm. Like they compressed all the research out there and all the thought and all the state of the art into a product that they're trying to create for you. And then really backing out and reverse engineering what it took to build something like that. That's huge, right? If you can actually understand perplexity, for instance, you'll already be well ahead on the research.
Swyx [00:52:39]: Oh, by the way, you mentioned what is a good perplexity score? There's just a number, right? It's like five to eight or something. Do you have a number in mind when you said that? Yeah.
Mark [00:52:48]: I mean, flipping between train loss and perplexity is actually not native to me quite yet. But if you can get a four using the context length extension on LLAMA, you're in the right direction. And then obviously you'll see spikes. And specifically when the one trick you should pay attention to is you know that your context length and theta scaling is working right if the early steps in the perplexity go straight down. So when it wasn't correct, it would oscillate a lot in the beginning. And we just knew that we cut the training short and then retry a new theta scale.
Swyx [00:53:19]: You're properly continuing fine tuning or the full pre-training. Yeah, yeah.
Mark [00:53:23]: The model just saw something out of domain immediately and was like, I have no idea what to do. And you need it to be able to overlap that positional embedding on top of each other. One follow up, right?
Swyx [00:53:34]: Before we close out. I think being on Twitter and looking at all these new headlines is really helpful, but then it only gets you a very surface level understanding. Then you still need a process to decide which one to invest in. I'm trying to dig for what is your formula for deciding what to go deep on and what to kind of skip.
Mark [00:53:54]: From a practical standpoint, as a company, I already know there are three to five things that will be valuable and useful to us. And then there's other stuff that's out of scope for different reasons. Some stuff is out of scope from, hey, this is not going to impact or help us. And then other things are out of scope because we can't do it. A really good instance for that is specific algorithms for improving extremely large scale distributed training. We're not going to have the opportunity to get 2000 H100s. If we do, it'd be really cool. But I'm just saying, as for now, you got to reach for the things that would be useful. Things that would be useful for us, for everybody actually, to be honest, is evaluations, different post-training techniques, and then synthetic data construction. I'm always on the look for that. And then how do I figure out which new piece of news is actually novel? Well, that's sort of my mental cache to a certain extent. I've built up this state of, hey, I already know all the things that have already been written for the state of the art for certain topic areas. And then I know what's being recycled as an empirical study versus something that actually is very insightful. Underrated specific instance would be the DeepSeek paper where I'd never seen it before, but the multi-head latent attention. That was really unexpected to me because I thought I'd seen every way that people wanted to cut mixture of experts into interesting ways. And I never thought something would catch my eye to be like, oh, this is totally new. And it really does have a lot of value. That's mainly how I try to do it. And you talk to your network too. I just talk to the people and then know and make sure that I have certain subject matter experts on speed dial that I also like to share information with and understand, hey, does this catch your eye too? Do you think this is valuable or real? Because it's a noisy space we're in right now, which is cool because it's really interesting and people are excited about it. But at the same time, there is actually a 10X or more explosion of information coming in that all sounds really, really unique and new. And you could spend hours down a rabbit hole that isn't as useful. Awesome, Mark.
Alessio [00:56:08]: I know we kept you in the studio for a long time. Any final call to actions for folks that could be roles you're hiring for, requests for startups, anything that comes to mind that you want to share with the audience?
Mark [00:56:19]: We definitely have a call to action to get more people to work together with us for long context evaluations. That is sort of the it topic throughout even meta or Google or any of the other folk are focusing on because I think we lack an understanding of that within the community. And then can we as a community also help to construct other modalities of datasets that would be interesting, like pairwise datasets, right? Like you could get just straight video and then straight text, but getting them together for grounding purposes will be really useful for training the next set of models that I know are coming out. And the more people we have contributing to that would be really useful. Awesome.
Alessio [00:57:00]: Thank you so much for coming on, Mark.
Swyx [00:57:02]: This was a lot of fun.
Alessio [00:57:02]: Yeah, thanks a lot.
Mark [00:57:03]: Yeah, this is great.
Speakers for AI Engineer World’s Fair have been announced! See our Microsoft episode for more info and buy now with code LATENTSPACE — we’ve been studying the best ML research conferences so we can make the best AI industry conf!
Note that this year there are 4 main tracks per day and dozens of workshops/expo sessions; the free livestream will air much less than half of the content this time.
Apply for free/discounted Diversity Program and Scholarship tickets here. We hope to make this the definitive technical conference for ALL AI engineers.
UPDATE: This is a 2 part episode - see Part 2 here.
ICLR 2024 took place from May 6-11 in Vienna, Austria.
Just like we did for our extremely popular NeurIPS 2023 coverage, we decided to pay the $900 ticket (thanks to all of you paying supporters!) and brave the 18 hour flight and 5 day grind to go on behalf of all of you. We now present the results of that work!
This ICLR was the biggest one by far, with a marked change in the excitement trajectory for the conference:
Of the 2260 accepted papers (31% acceptance rate), of the subset of those relevant to our shortlist of AI Engineering Topics, we found many, many LLM reasoning and agent related papers, which we will cover in the next episode. We will spend this episode with 14 papers covering other relevant ICLR topics, as below.
As we did last year, we’ll start with the Best Paper Awards. Unlike last year, we now group our paper selections by subjective topic area, and mix in both Outstanding Paper talks as well as editorially selected poster sessions. Where we were able to do a poster session interview, please scroll to the relevant show notes for images of their poster for discussion. To cap things off, Chris Ré’s spot from last year now goes to Sasha Rush for the obligatory last word on the development and applications of State Space Models.
We had a blast at ICLR 2024 and you can bet that we’ll be back in 2025 🇸🇬.
Timestamps and Overview of Papers
[00:02:49] Section A: ImageGen, Compression, Adversarial Attacks
* [00:02:49] VAEs
* [00:32:36] Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models
* [00:37:25] The Hidden Language Of Diffusion Models
* [00:48:40] Ilya on Compression
* [01:01:45] Christian Szegedy on Compression
* [01:07:34] Intriguing properties of neural networks
[01:26:07] Section B: Vision Learning and Weak Supervision
* [01:26:45] Vision Transformers Need Registers
* [01:38:27] Think before you speak: Training Language Models With Pause Tokens
* [01:47:06] Towards a statistical theory of data selection under weak supervision
* [02:00:32] Is ImageNet worth 1 video?
[02:06:32] Section C: Extending Transformers and Attention
* [02:06:49] LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
* [02:15:12] YaRN: Efficient Context Window Extension of Large Language Models
* [02:32:02] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
* [02:44:57] ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
[02:54:26] Section D: State Space Models vs Transformers
* [03:31:15] Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
* [03:37:08] End of Part 1
A: ImageGen, Compression, Adversarial Attacks
* Durk Kingma (OpenAI/Google DeepMind) & Max Welling: Auto-Encoding Variational Bayes (Full ICLR talk)
* Preliminary resources: Understanding VAEs, CodeEmporium, Arxiv Insights
* Inaugural ICLR Test of Time Award! “Probabilistic modeling is one of the most fundamental ways in which we reason about the world. This paper spearheaded the integration of deep learning with scalable probabilistic inference (amortized mean-field variational inference via a so-called reparameterization trick), giving rise to the Variational Autoencoder (VAE).”
* Pablo Pernías (Stability) et al: Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR oral, poster)
* Hila Chefer et al (Google Research): Hidden Language Of Diffusion Models (poster)
* See also: Google Lumiere, Attend and Excite
* Christian Szegedy (X.ai): Intriguing properties of neural networks (Full ICLR talk)
* Ilya Sutskever: An Observation on Generalization
* on Language Modeling is Compression
* “Stating The Obvious” criticism
* Really good compression amounts to intelligence
* Lexinvariant Language models
* Inaugural Test of Time Award runner up: “With the rising popularity of deep neural networks in real applications, it is important to understand when and how neural networks might behave in undesirable ways. This paper highlighted the issue that neural networks can be vulnerable to small almost imperceptible variations to the input. This idea helped spawn the area of adversarial attacks (trying to fool a neural network) as well as adversarial defense (training a neural network to not be fooled). “
* with Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus
B: Vision Learning and Weak Supervision
* Timothée Darcet (Meta) et al : Vision Transformers Need Registers (ICLR oral, Paper)
* ICLR Outstanding Paper Award: “This paper identifies artifacts in feature maps of vision transformer networks, characterized by high-norm tokens in low-informative background areas. The authors provide key hypotheses for why this is happening and provide a simple yet elegant solution to address these artifacts using additional register tokens, enhancing model performance on various tasks. The insights gained from this work can also impact other application areas. The paper is very well-written and provides a great example of conducting research – identifying an issue, understanding why it is happening, and then providing a solution.“
* HN discussion: “According to the paper, the "registers" are additional learnable tokens that are appended to the input sequence of a Vision Transformer model during training. They are added after the patch embedding layer, with a learnable value, similar to the [CLS] token and then at the end of the Vision Transformer, the register tokens are discarded, and only the [CLS] token and patch tokens are used as image representations.
The register tokens provide a place for the model to store, process and retrieve global information during the forward pass, without repurposing patch tokens for this role.
Adding register tokens removes the artifacts and high-norm "outlier" tokens that otherwise appear in the feature maps of trained Vision Transformer models. Using register tokens leads to smoother feature maps, improved performance on dense prediction tasks, and enables better unsupervised object discovery compared to the same models trained without the additional register tokens. This is a neat result. For just a 2% increase in inference cost, you can significantly improve ViT model performance. Close to a free lunch.”
* Sachin Goyal (Google) et al: Think before you speak: Training Language Models With Pause Tokens (OpenReview)
* We operationalize this idea by performing training and inference on language models with a (learnable) pause token, a sequence of which is appended to the input prefix. We then delay extracting the model's outputs until the last pause token is seen, thereby allowing the model to process extra computation before committing to an answer. We empirically evaluate pause-training on decoder-only models of 1B and 130M parameters with causal pretraining on C4, and on downstream tasks covering reasoning, question-answering, general understanding and fact recall.
* Our main finding is that inference-time delays show gains when the model is both pre-trained and finetuned with delays. For the 1B model, we witness gains on 8 of 9 tasks, most prominently, a gain of 18% EM score on the QA task of SQuAD, 8% on CommonSenseQA and 1% accuracy on the reasoning task of GSM8k. Our work raises a range of conceptual and practical future research questions on making delayed next-token prediction a widely applicable new paradigm.
* Pulkit Tandon (Granica) et al: Towards a statistical theory of data selection under weak supervision (ICLR Oral, Poster, Paper)
* Honorable Mention: “The paper establishes statistical foundations for data subset selection and identifies the shortcomings of popular data selection methods.”
* Shashank Venkataramanan (Inria) et al: Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video (ICLR Oral, paper)
* First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hours-long, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning.
* Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image-based pretraining approaches to incorporate more frames. Instead, we advocate a "tracking to learn to recognize" approach. Our method called DoRA leads to attention maps that DiscOver and tRAck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks.
* Honorable Mention: “The paper proposes a novel path to self-supervised image pre-training, by learning from continuous videos. The paper contributes both new types of data and a method to learn from novel data.“
C: Extending Transformers and Attention
* Yukang Chen (CUHK) et al: LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR Oral, Poster)
* We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost. LongLoRA extends Llama2 7B from 4k context to 100k, or Llama2 70B to 32k on a single 8x A100 machine. LongLoRA extends models' context while retaining their original architectures, and is compatible with most existing techniques, like Flash-Attention2.
* Bowen Peng (Nous Research) et al: YaRN: Efficient Context Window Extension of Large Language Models (Poster, Paper)
* Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. The models fine-tuned using YaRN has been made available and reproduced online up to 128k context length.
* Mentioned papers: Kaikoendev on TILs While Training SuperHOT, LongRoPE, Ring Attention, InfiniAttention, Textbooks are all you need and the Synthetic Data problem
* Suyu Ge et al: Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (aka FastGen. ICLR Oral, Poster, Paper)
* “We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs). Different from the conventional KV cache that retains key and value vectors for all context tokens, we conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens. In our experiments across various asks, FastGen demonstrates substantial reduction on GPU memory consumption with negligible generation quality loss. ”
* 40% memory reduction for Llama 67b
* Honorable Mention: “The paper targets the critical KV cache compression problem with great impact on transformer based LLMs, reducing the memory with a simple idea that can be deployed without resource intensive fine-tuning or re-training. The approach is quite simple and yet is shown to be quite effective.”
* Guanhua Wang (DeepSpeed) et al, ZeRO++: Extremely Efficient Collective Communication for Giant Model Training (paper, poster, blogpost)
* Zero Redundancy Optimizer (ZeRO) has been used to train a wide range of large language models on massive GPUs clusters due to its ease of use, efficiency, and good scalability. However, when training on low-bandwidth clusters, or at scale which forces batch size per GPU to be small, ZeRO's effective throughput is limited because of high communication volume from gathering weights in forward pass, backward pass, and averaging gradients. This paper introduces three communication volume reduction techniques, which we collectively refer to as ZeRO++, targeting each of the communication collectives in ZeRO.
* Collectively, ZeRO++ reduces communication volume of ZeRO by 4x, enabling up to 2.16x better throughput at 384 GPU scale.
* Mentioned: FSDP + QLoRA
Poster Session Picks
We ran out of airtime to include these in the podcast, but we recorded interviews with some of these authors and could share audio on request.
* Summarization
* BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR Oral)
* Uncertainty
* Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
* Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models
* MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs
* Language Model Cascades: Token-Level Uncertainty And Beyond
* Tabular Data
* CABINET: Content Relevance-based Noise Reduction for Table Question Answering
* Squeezing Lemons with Hammers: An Evaluation of AutoML and Tabular Deep Learning for Data-Scarce Classification Applications
* Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space
* Making Pre-trained Language Models Great on Tabular Prediction
* How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data
* Watermarking (there were >24 papers on watermarking, both for and against!!)
* Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense
* Provable Robust Watermarking for AI-Generated Text
* Attacking LLM Watermarks by Exploiting Their Strengths
* Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
* Is Watermarking LLM-Generated Code Robust?
* On the Reliability of Watermarks for Large Language Models
* Watermark Stealing in Large Language Models
* Misc
* Massively Scalable Inverse Reinforcement Learning in Google Maps
* Zipformer: A faster and better encoder for automatic speech recognition
* Conformal Risk Control
D: State Space Models vs Transformers
* Sasha Rush’s State Space Models ICLR invited talk on workshop day
* Ido Amos (IBM) et al: Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR Oral)
* Modeling long-range dependencies across sequences is a longstanding goal in machine learning and has led to architectures, such as state space models, that dramatically outperform Transformers on long sequences.
* However, these impressive empirical gains have been by and large demonstrated on benchmarks (e.g. Long Range Arena), where models are randomly initialized and trained to predict a target label from an input sequence. In this work, we show that random initialization leads to gross overestimation of the differences between architectures.
* In stark contrast to prior works, we find vanilla Transformers to match the performance of S4 on Long Range Arena when properly pretrained, and we improve the best reported results of SSMs on the PathX-256 task by 20 absolute points.
* Subsequently, we analyze the utility of previously-proposed structured parameterizations for SSMs and show they become mostly redundant in the presence of data-driven initialization obtained through pretraining. Our work shows that, when evaluating different architectures on supervised tasks, incorporation of data-driven priors via pretraining is essential for reliable performance estimation, and can be done efficiently.
* Outstanding Paper Award: “This paper dives deep into understanding the ability of recently proposed state-space models and transformer architectures to model long-term sequential dependencies. Surprisingly, the authors find that training transformer models from scratch leads to an under-estimation of their performance and demonstrates dramatic gains can be achieved with a pre-training and fine-tuning setup. The paper is exceptionally well executed and exemplary in its focus on simplicity and systematic insights.”
The podcast currently has 78 episodes available.
950 Listeners
472 Listeners
436 Listeners
279 Listeners
280 Listeners
163 Listeners
199 Listeners
78 Listeners
175 Listeners
100 Listeners
62 Listeners
139 Listeners
55 Listeners
307 Listeners
10 Listeners