This paper talks about how to measure the quality of large language models (LLMs) in a way that considers both how well they perform and how efficient they are. The authors introduce a new idea called "capacity density," which is like figuring out how much "brainpower" an LLM has compared to its size. Imagine two students who get the same grade on a test – the student who studied less has a higher "study density." Similarly, an LLM that can perform as well as a larger model but with fewer parameters has a higher capacity density. The researchers looked at many popular LLMs and found something interesting: the capacity density of LLMs is doubling every three months! This means that we're getting much better at creating powerful LLMs without needing to make them ridiculously huge. They call this trend the "Densing Law," and it has some cool implications, like the fact that the cost of running these models is going down rapidly. The authors believe that instead of just focusing on making LLMs bigger, we should aim to make them denser, which will lead to more powerful AI that is also more accessible and environmentally friendly.

https://arxiv.org/pdf/2412.04315