Scaling Theory

My guest is Arvind Narayanan, a Professor of Computer Science at Princeton University, and the director of the Center for Information Technology Policy, also at Princeton. Arvind is renowned for his work on the societal impacts of digital technologies, including his textbook on fairness and machine learning, his online course on cryptocurrencies, his research on data de-anonymization, dark patterns, and more. He has already amassed over 30,000 citations on Google Scholar.

In just a few days, in late September 2024, Arvind will release a book co-authored with Sayash Kapoor titled “AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference.” Having had the privilege of reading an early version, our conversation delves into some of the book’s key arguments. We also explore what Arvind calls AI scaling myths, the reality of artificial general intelligence, how governments can scale effective AI policies, the importance of transparency, the role that antitrust can, and cannot play, the societal impacts of scaling automation, and more. I hope you enjoy our conversation.

Find me on X at @⁠⁠⁠ProfSchrepel⁠⁠⁠⁠⁠⁠. Also, be sure to subscribe.

**

References:

➝ AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference (2024)

➝ AI scaling myths (2024)

➝ AI existential risk probabilities are too unreliable to inform policy (2024)

➝ Foundation Model Transparency Reports (2024)

My guest today is ⁠Sara Hooker⁠, VP of Research at Cohere, where she leads Cohere for AI, a non-profit research lab that seeks to solve complex machine learning problems with researchers from over 100 countries. Sara is the author of numerous research papers, some of which focus specifically on scaling theory in AI. She has been listed as one of AI’s top 13 innovators by Fortune.

In our conversation, we first delve into the scaling laws behind foundation models. We explore what powers the scaling of AI systems and the limits to scaling laws. We then move on to discussing openness in AI, Cohere’s business strategy, the power of ecosystems, the importance of building multilingual LLMs, and the recent change in terms of access to data in the space. I hope you enjoy our conversation.

Find me on X at @⁠⁠ProfSchrepel⁠⁠⁠⁠⁠. Also, be sure to subscribe.

**

References:

➝ Sara Hooker, On the Limitations of Compute Thresholds as a Governance Strategy (2024)

➝ Sara Hooker, The Hardware Lottery (2020)

➝ Sara Hooker, Moving beyond “algorithmic bias is a data problem” (2021)

My guest today is Michael Mauboussin (@mjmauboussin), one of the world’s leading experts in finance. Michael serves as Head of Consilient Research at Counterpoint Global, Morgan Stanley. He has authored three books and regularly appears in the Wall Street Journal, Financial Times, New York Times, and other publications. Since 1993, Michael has been an adjunct professor of finance at Columbia Business School and is also the chairman emeritus of the board of trustees at the Santa Fe Institute.

In our conversation, we delve into the dynamics of markets, discuss all sorts of increasing returns, and explore topics such as Charles Darwin, policymaking, AI and Web3, and the Santa Fe Institute. I hope you enjoy our discussion.

Find me on X at @⁠ProfSchrepel⁠⁠⁠⁠. Also, be sure to subscribe to the Scaling Theory podcast.

**

References:

My guest is Glen Weyl, an influential economist and social technologist known for his interdisciplinary work at the intersection of economics, computer science, sociology, and political science. He is a Principal Researcher at Microsoft Research, a co-author of the books “Radical Markets” and the recently published “Plurality: The Future of Collaborative Technology and Democracy” which he co-authored with Audrey Tang, who has served as the 1st Minister of Digital Affairs of Taiwan. Glen co-founded the RadicalxChange Foundation, the Plural Technology Collaboratory, and the Plurality Institute. He taught economics at the University of Chicago, Princeton, and Yale.

Glen’s work frequently explores the potential of technology and market mechanisms to drive social change and address systemic inequalities. In his new book, Glen lays the foundation for a new societal vision: plurality.

Our conversation is structured into three distinct parts. We begin by situating the concept of plurality in today’s philosophical and technological landscape. We then discuss the technologies that will enable plurality to materialize, and finally, we conclude with a discussion of the challenges ahead. By the end of this discussion, you will know what plurality is, how different it is from the vision of today’s tech CEOs, the role open source AI and blockchain can play to scale plurality, but also the role legal rules and standards are playing, which of the scaling theory concepts (including fractals, fragility, robustness, chaos, etc.) helped Glen and Audrey define their strategy to take over other visions, and more. I hope you enjoy the conversation.

Find me on X at @ProfSchrepel⁠⁠⁠. Also, be sure to subscribe to the Scaling Theory podcast; it helps its growth.

My guest is Yann LeCun, a pioneering French-American computer scientist, known for his groundbreaking work in machine learning, computer vision, and neural networks. Yann is the Silver Professor at the Courant Institute of Mathematical Sciences at New York University and serves as the Vice President and Chief AI Scientist at Meta.

Yann is one of the world’s most influential computer scientists. He has accumulated over 350,000 citations on Google Scholar, he is one of the founding figures in the field of deep learning thanks to its contribution to convolutional neural networks and backpropagation algorithms, and he is a vocal proponent of open source. In recognition of his significant contributions to artificial intelligence, he was awarded the Turing Award in 2018, often referred to as the “Nobel Prize of Computing.”

Our conversation is structured into three distinct parts. We begin by discussing the overarching dynamics in the AI space, then narrow our focus to the firm level, and finally, we conclude with an exploration of the challenges that lie ahead. By the end of this discussion, you will learn whether open source has a chance to make it in AI, the key factors for scaling an AI foundation model, the role ecosystems play in market dynamics, Meta long term strategy in the space, how concentration among chip manufacturers impacts AI companies, the current effect of the European AI Act on AI companies, what Yann would like to see regulators doing, and more. I hope you enjoy the conversation.

J. Doyne Farmer is the Director of the Complexity Economics programme at the Institute for New Economic Thinking at the Oxford Martin School, a Professor at the Mathematical Institute of Oxford University, and an External Professor at the Santa Fe Institute.

In this episode, we explore Doyne’s latest book, “Making Sense of Chaos.” We focus on the relationship between chaos and scaling theory, and more specifically, how chaos can be factored into scaling theory. By the end of this conversation, you will learn why it might be easier to predict the long distant future than predicting tomorrow, how Moore’s Law conflicts with other scaling laws that underpin technological progress, how agent-based modeling can help all scientists and policymakers, how to dominate the world with your theories (...), and even how to trick casinos. I hope you enjoy the conversation.

Find me on X at @⁠⁠ProfSchrepel⁠⁠. Also, be sure to subscribe to the Scaling Theory podcast; it helps its growth.

***

Thomas Wolf is the co-founder and Chief Science Officer of Hugging Face, the company at the center of the open-source AI ecosystem. He has a Ph.D. in statistical & quantum physics.

In this episode, we explore why open-source (“OS”) AI may be preferable to closed-source, whether OS has a real chance to take over the space, the challenges OS developers must overcome to scale their foundational models, the role of data and infrastructure in scaling dynamics, how big tech companies are positioning themselves in the ecosystem and responding to other companies' strategies in a true complexity science fashion, and overall, the power laws that undermine the growth of the ecosystem. I hope you enjoy the conversation. I hope you enjoy the conversation.

Find me on X at @⁠ProfSchrepel⁠. Also, be sure to subscribe to the Scaling Theory podcast; it helps its growth.

***

Geoffrey West is a physicist, former president and distinguished professor of the Santa Fe Institute. His book, “Scale: The Universal Laws of Life, Growth, and Death in Organisms, Cities, and Businesses” (2017), is a masterpiece.

In this episode, we talk about the power laws behind living organisms, cities, businesses, and technologies. By the end of this episode, you will know more about the power law behind the heartbeat of all mammals, the number of patents and crime in big cities compared to small cities, innovation, the way technology scales, and more. I hope you enjoy the conversation.

Find me on X at @ProfSchrepel. Also, be sure to subscribe to the Scaling Theory podcast; it helps its growth.

***

References

In this first episode, Dr. Thibault Schrepel (@⁠ProfSchrepel⁠) introduces “Scaling Theory”, a podcast dedicated to the power laws behind the growth of companies, technologies, legal and living systems.

***

References:

➝ Charles Darwin, On the Origin of Species (1859)

➝ Melanie Mitchell, Complexity: A Guided Tour (2011)

➝ Mitchell Waldrop, Complexity The Emerging Science at the Edge of Order and Chaos (2019)

➝ John H. Miller & Scott Page, Complex Adaptive Systems: An Introduction to Computational Models of Social Life (2007)

➝ W. Brian Arthur, Complexity and the Economy (2014)

➝ Geoffrey West, Scale: The Universal Laws of Life and Death in Organisms, Cities and Companies (2017)