The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Obstacles to the Implementation of Indoor Air Quality Improvements by JesseSmith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Obstacles to the Implementation of Indoor Air Quality Improvements, published by JesseSmith on May 29, 2023 on The Effective Altruism Forum.
    1. Tl;dr
    Many reports indicate that indoor air quality (IAQ) interventions are likely to be effective at reducing respiratory disease transmission. However, to date there’s been very little focus on the workforce that will implement these interventions. I suggest that the US Heating, Ventilation and Air Conditioning (HVAC) and building maintenance workforces have already posed a significant obstacle to these interventions, and broad uptake of IAQ measures will be significantly hindered by them in the future. The impact will vary in predictable ways depending on the nature of the intervention and its implementation. We should favor simple techniques with improved oversight and outsource or crosscheck technically complex work to people outside of the current HVAC workforce. We should also make IAQ conditions and devices as transparent as possible to both experts and building occupants.
    To skip my bio and the technical horrors section, proceed to the recommendations in section 4.
    2. Who am I? Why do I think This? How Certain am I?
    I began working in construction in 1991. I did a formal carpentry apprenticeship in Victoria BC in the mid-90s and moved to the US in ‘99. Around 2008 I started taking greater interest in HVAC because - despite paying top dollar to local subcontractors - our projects had persistent HVAC problems. Despite protestations that they were following exemplary practices, our projects were plagued with high humidity, loud noise, frequent mechanical failure, and room-to-room temperature differences. This led me to first learn all aspects of system design and controls, and culminated in full system installations. Along the way I obtained a NJ Master HVAC license, performed the thermal work of ~2k light-duty energy retrofits, obtained multiple certifications in HVAC and low-energy design, and became a regional expert in building diagnostics. Since 2010 I’ve worked as a contractor or consultant to roughly a dozen major HVAC contractors and hundreds of homeowners.
    I’m reasonably certain that the baseline competence of the HVAC workforce is insufficient to broadly and reliably deploy IAQ interventions and that this is a serious obstacle. My comments are specific to the US. I’ve discussed these problems extensively with friends and acquaintances working at a national level and in other parts of the US and believe them to be common to most of the country. The problems are specific to the light commercial and residential workforce, but not domains that are closely monitored by mechanical engineering teams (e.g. hospitals). Based on some limited experience I suspect these problems are also common to Canada, but I’m less certain about their severity.
    3. Technical Horrors: Why is This so Difficult?
    Within HVAC, many important jobs are currently either not performed or delegated to people who are largely incapable of performing them. Many people convincingly lie about their capacity to perform a job they’re incapable of, report having done things they haven’t, or even make statements at odds with physics.
    Examples include:
    Accurate heat load/loss calculations: These are used to size heating and cooling systems, and in most areas are code mandated for both new and replacement systems. Competent sizing (Manual J for residential) is viewed as highly important by virtually all experts within HVAC. However, despite decades of investment in training and compliance, a lead technical manager of a clean energy program reported to me that >90% of Manual Js reviewed by his program had significant errors made apparent due to internal inconsistency (eg duct load on a hydronic system) or obvious inconsistencies with public information on zi...
    13 min
  • EA - Governments Might Prefer Bringing Resources Back to the Solar System Rather than Space Settlement in Order to Maintain Control, Given that Governing Interstellar Settlements Looks Almost Impossible by Dr. David Mathers
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Governments Might Prefer Bringing Resources Back to the Solar System Rather than Space Settlement in Order to Maintain Control, Given that Governing Interstellar Settlements Looks Almost Impossible, published by Dr. David Mathers on May 29, 2023 on The Effective Altruism Forum.
    Part of my work for Arb Research (/).
    Epistemic Status: I have no scientific background and wrote this after only a couple of days thought, so it is very possible that there is some argument I am unaware of, but which would be obvious to physicists, why a ‘resource-gathering without settlement’ approach to interstellar exploration is not feasible. However, my Arb colleague Vasco Grilo has aerospace engineering expertise, and says he can’t think of any reason why it wouldn’t be feasible in principle. Still, take all this with a large dose of caution.
    Some futurists have considered it likely that, at least absent existential catastrophe in the next few centuries, human beings (or our post-human or machine descendants) will eventually attempt to settle our galaxy. After all, there are vastly more resources in the rest of the Milky Way than in the Solar system. So we could support far more lives and create much more of anything else we care about, if we make use of stuff out there in the wider galaxy. And one very obvious way for us to make use of that stuff is for us to send out spaceships to establish settlements which make use of the energy of the stars they arrive at. Those settlements could in turn seed further settlements in an iterative process. (This would likely require “digital people”/#fnref6 given the distances involved in interstellar travel.)
    However, this is not the only way in which we could try to make use of resources outside the solar system. Another way to do so would be to try and gather resources and bring them back to the Solar system without establishing any permanent settlements of either humans or AIs outside the Solar system itself. I think that a government on Earth (or elsewhere in the solar system) might actually prefer gathering resources in this way to space settlements for the following reason:
    Impossibility of Interstellar Governance (IIG): Because of the huge distances between stars, it is simply not possible for a government in the Solar system to exercise long-term effective governance over any space colonies further away than (at most) the closest handful of stars.
    For a powerful, although not completely conclusive, case for this claim see this Medium post:/@KevinKohlerFM/cosmic-anarchy-and-its-consequences-b1a557b1a2e3
    Given IIG, no government within the Solar system can be the government of a settlement outside it. Therefore, if a government sets up a colony run by agents in another star system, it loses direct control of those resources. Of course, the government can try and exercise more indirect control over what happens by choosing starting colonists with particular values. But it’s unclear the degree of control that will allow for long-term.
    Meanwhile, a government could try and send a mission to other stars which:A) Is not capable of setting-up a new self-sufficient settlement, or can be trusted not to do so. BUT,B) is capable of setting up physical infrastructure to extract the system’s energy and resources and bringing them back to the Solar system. This way, a government situated in the Solar system could maintain direct control over how resources are used. In contrast if they go the space settlement route, the government cannot directly govern the settlement. So it has to rely on the idea that if values of the initial settlers are correct, then the settlement will use its resources in the way the government desires even whilst operating outside the government’s control.
    A purely resource-gathering mission without settlement will be particularly att...
    9 min
  • EA - EA Estonia's Impact in 2022 by RichardAnnilo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Estonia's Impact in 2022, published by RichardAnnilo on May 29, 2023 on The Effective Altruism Forum.
    Background
    This report is about January to December 2022 in EA Estonia, corresponding to our last grant period (funding from the EA Infrastructure Fund for 1 FTE and group expenses).
    Quick facts about Estonia: it has a population of 1.3 million and is placed both geographically and culturally between the Nordics and Eastern Europe. Our language has 14 noun cases, it is the birthplace of 10 unicorns, and we have the best mushroom scientists. Go figure.
    In our national EA group, there are 23–30 people whom I would consider to be “highly engaged” (meaning they have spent more than 100 hours engaging with EA content, have developed career plans and have taken significant steps in pursuit of doing good). You could expect around 30 people to attend our largest community event (Figure 1) and our Slack channel has 50–60 active weekly members (Figure 2).
    Figure 1: Attendees of our largest community event, the EA Estonia Summer Retreat. August 2022.
    Figure 2. EA Estonia Slack statistics from its creation. Weekly active members have been oscillating between 40 and 65 throughout 2022.
    Group strategy
    Here are the main metrics we used to evaluate our impact:
    Awareness: Number of people aware of the term “effective altruism” and EA Estonia.
    Activities:
    Introductory talks
    Direct outreach
    Social media outreach
    First engagement: Number of people who took action because of our outreach.
    Activities:
    Introductory course
    Cause-specific reading groups
    Career planning: Number of people that develop career plans based on EA principles that are well informed and well reasoned.
    Activities:
    Career course
    Taking action: Number of people taking significant action based on EA-informed career plans (e.g. starting full-time jobs, university degrees).
    Activities:
    1-1 career calls
    Peer-mentoring
    Directly sharing opportunities
    Concerns with this model:
    The actual impact comes when people take action within high-impact jobs, which we currently don't measure.
    We don't measure value drift or other kinds of decreased engagement after taking significant next steps.
    This model doesn't prioritize targeting existing Highly Engaged EAs (HEAs) to have a higher impact.
    This also doesn't include a more meta-level goal of keeping people engaged and interested while moving towards an HEA status. We do organize social events for this reason, however the impact of them is not quantified.
    Regarless of these concerns, the main theory of change feels relatively straight-forward: (1) we find young altruistically-minded people who are unclear about their future career plans, then (2) we make them aware of the effective altruism movement and various high-impact career paths, and then (3) we prompt them to develop explicit career plans and encourage them to take action upon them.
    Below I will go into more detail regarding the goals, activities and results of 2022 in two categories: (i) outreach and (ii) growing HEAs. I will end with a short conclusion and key takeaways for next year.
    I Outreach
    Goal: 5,000 new people who know what “effective altruism” means and that there is an active local group in Estonia.
    Actual: 20,776 people reached.
    Activities:
    Liina Salonen started working full-time as the Communictions Specialist in EA Estonia.
    Reached at least 20,000 people on Facebook with the Introductory Course social media campaign
    Student fair tabling.
    At least 155 people reached (played the Giving Game)
    Goal: 10 lecturers mentioning EA Estonia
    Actual: 1 lecturers reached
    Visited a philosophy lecture. Number of students: 20.
    Talked about effective altruism and longtermism. Created a discussion with the lecturer.
    Suggested people sign up for our career course. Nobody responded.
    Wrote to two other philosophy lecturers, but the...
    14 min
  • AF - Wikipedia as an introduction to the alignment problem by SoerenMind
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wikipedia as an introduction to the alignment problem, published by SoerenMind on May 29, 2023 on The AI Alignment Forum.
    AI researchers and others are increasingly looking for an introduction to the alignment problem that is clearly written, credible, and supported by evidence and real examples. The Wikipedia article on AI Alignment has become such an introduction.
    Link:
    Aside from me, it is written by Mantas Mazeika and Gavin Leech (who are great technical writers), other Wikipedia contributors, and copy editor Amber Ace. It also had extensive feedback from this community.
    In the last month, it had ~20k unique readers and was cited by Yoshua Bengio.
    We've tried hard to keep the article accessible for non-technical readers while also making sense to AI researchers.
    I think Wikipedia is a good format to introduce many readers to the alignment problem because it can include videos and illustrations (unlike papers) and it is more credible than blog posts. However, Wikipedia has strict rules and could be changed by anyone.
    Note that we've announced this effort on the Wikipedia talk page and shared public drafts to let other editors give feedback and contribute.
    I you edit the article, please keep in mind Wikipedia's rules, use reliable sources, and consider that we've worked hard to keep it concise because most Wikipedia readers spend <1 minute on the page. For the latter goal, it's best to focus on edits that reduce or don't increase length. To give feedback, feel free to post on the talk page or message me.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Should the EA community be cause-first or member-first? by EdoArad
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should the EA community be cause-first or member-first?, published by EdoArad on May 29, 2023 on The Effective Altruism Forum.
    It's really hard to do community building well. Opinions on strategy and vision vary a lot, and we don't yet know enough about what actually works and how well. Here, I'll suggest one axis of community-building strategy which helped me clarify and compare some contrasting opinions.
    Cause-first
    Will Macaskill's proposed Definition of Effective Altruism is composed of:
    An overarching effort to figure out what are the best opportunities to do good.
    A community of people that work to bring more resources to these opportunities, or work on these directly.
    This suggests a "cause-first" community-building strategy, where the main goal for community builders is to get more manpower into the top cause areas. Communities are measured by the total impact produced directly through the people they engage with. Communities try to find the most promising people, persuade them to work on top causes, and empower them to do so well.
    CEA's definition and strategy seem to be mostly along these lines:
    Effective altruism is a project that aims to find the best ways to help others, and put them into practice.
    It’s both a research field, which aims to identify the world’s most pressing problems and the best solutions to them, and a practical community that aims to use those findings to do good.
    and
    Our mission is to nurture a community of people who are thinking carefully about the world’s biggest problems and taking impactful action to solve them.
    Member-first
    Let's try out a different definition for the EA community, taken from CEA's guiding principles:
    What is the effective altruism community?
    The effective altruism community is a global community of people who care deeply about the world, make helping others a significant part of their lives, and use evidence and reason to figure out how best to do so.
    This, to me, suggests a subtly different vision and strategy for the community. One that is, first of all, focused on these people who live by EA principles. Such a "member-first" strategy could have a supporting infrastructure that is focused on helping the individuals involved to live their lives according to these principles, and an outreach/growth ecosystem that works to make the principles of EA more universal.
    What's the difference?
    I think this dimension has important effects on the value of the community, and that both local and global community-building strategies should be aware of the tradeoffs between the two.
    I'll list some examples and caricatures for the distinction between the two, to give a more intuitive grasp of how these strategies differ, without any clear order:
    Leaning cause-firstLeaning member-firstKeep EA Small and WeirdBig Tent EACurrent EA Handbook (focus on introducing major causes)2015's EA Handbook (focus on core EA principles)80,000 HoursProbably GoodWants more people doing high-quality AI Safety work, regardless of their acceptance of EA principlesWants more people deeply understanding and accepting EA principles, regardless of what they actually work on or donate to.Targeted outreach to students in high ranking universitiesBroad outreach with diverse messagingEncourages people to change occupations to focus on the world's most pressing problemsEncourages people to use the tools and principles of EA to do more good in their current trajectoryRisk of people not finding useful ways to contribute to top causesRisk of not enough people who want to contribute to the world's top causesThe community as a whole leads by example, by taking in-depth prioritization research with the proper seriousnessEach individual is focused more on how to
    implement EA principles in their own lives, taking their personal worldview and situation into account ...
    8 min
  • LW - Gemini will bring the next big timeline update by p.b.
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gemini will bring the next big timeline update, published by p.b. on May 29, 2023 on LessWrong.
    There is a genre of LLM critique that criticises LLMs for being, well, LLMs.
    Yann LeCun for example points to the inability of GPT-4 to visually imagine the rotation of interlocking gears as a fact that shows how far away AGI is, instead of a fact that shows how GPT-4 has not been trained on video data yet.
    There are many models now that "understand" images or videos or even more modalities. However, they are not end-to-end trained on these multiple modalities. Instead they use an intermediary model like CLIP, that translates into the language domain. This is a rather big limitation, because CLIP can only represent concepts in images that are commonly described in image captions.
    Why do I consider this a big limitation? Currently it looks like intelligence emerges from learning to solve a huge number of tiny problems. Language seems to contain a lot of useful tiny problems. Additionally it is the interface to our kind of intelligence, which allows us to assess and use the intelligence extracted from huge amounts of text.
    This means that adding a modality with a CLIP-like embedding and than doing some fine-tuning does not add any intelligence to the system. It only adds eyes or ears or gears.
    Training end-to-end on multi-modal data should allow the model to extract new problem solving circuits from the new modalities. The resulting model would not just have eyes, but visual understanding.
    Deepmind did a mildly convincing proof-of-concept with Gato last year, a small transformer trained on text, images, computer games and robotics. Now it seems they will try to scale Gato to Gemini, leapfrogging GPT-4 in the process.
    GPT-4 itself has image processing capabilities that are not yet available to the general public. But whether these are an add-on or result of integrated image modelling we don't know yet.
    To me it seems very likely, that a world where the current AI boom fizzles is a world where multi-modality does not bring much benefits or we cannot figure out how to do it right or possibly the compute requirements of doing it right is still prohibitive.
    I think Gemini will give us a good chunk of information about whether that is the world we are living in.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Kelly betting vs expectation maximization by MorgneticField
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Kelly betting vs expectation maximization, published by MorgneticField on May 28, 2023 on LessWrong.
    People talk about Kelly betting and expectation maximization as though they're alternate strategies for the same problem. Actually, they're each the best option to pick for different classes of problems. Understanding when to use Kelly betting and when to use expectation maximization is critical.
    Most of the ideas for this came from Ole Peters ergodicity economics writings. Any mistakes are my own.
    The parable of the casino
    Alice and Bob visit a casino together. They each have $100, and they decide it'll be fun to split up, play the first game they each find, and then see who has the most money. They'll then keep doing this until their time in the casino is up in a couple days.
    Alice heads left and finds a game that looks good. It's double or nothing, and there's a 60% chance of winning. That sounds good to Alice. Players buy as many tickets as they want. Each ticket resolves independently from the others at the stated odds, but all resolve at the same time. The tickets are $1 each. How many should Alice buy?
    Bob heads right and finds a different game. It's a similar double or nothing game with a 60% chance of winning. He has to buy a ticket to play, but in Bob's game he's only allowed to buy one ticket. He can pay however much he wants for it, then the double or nothing is against the amount he paid for his ticket. How much should he pay for a ticket?
    Alice's game is optimized by an ensemble average
    Let's estimate the amount of money Alice will win, as a function how many tickets she buys. We don't know how each ticket resolves, but we can say that approximately 60% of the tickets will be winners and 40% will be losers (though we don't know which tickets will be which). This is just calculating the expected value of the bet.
    If she buys x tickets, she'll make 0.6∗x∗2+0.4∗x∗0 dollars. This is a linear function that monotonically increases with x, so Alice should buy as many as she can.
    Since she has $100, she can buy 100 tickets. That means she will probably come away with $120. There will be some variance here. If tickets were cheaper (say only a penny each), then she could lower her variance buy buying more tickets at the lower price.
    Bob's game is optimized by a time-average
    Unlike Alice's game with a result for each ticket, there's only one result to Bob's game. He either doubles his ticket price or gets nothing back.
    One way people tend to approach this is to apply an ensemble average anyway via expectation maximization. If you do this, you end up with basically the same argument that Alice had and try to bet all of your money. There are two problems with this.
    One problem is that Alice and Bob are going to repeat their games as long as they can. Each time they do, they'll have a different amount of money available to bet (since they won or lost on the last round). They want to know who will have the most at the end of it.
    The repeated nature of these games mean that they aren't ergodic. As soon as someone goes bust, they then can't play any more games. If Bob bets the same way Alice does, and goes all in, then he can only get $0 or double out. After one round, he's 60% likely to be solvent. After 10 rounds, he's only 0.610 likely to have any money at all. That's about half a percent, and Bob is likely to spend the last few days of their trip chilling in the bar instead of gaming.
    The second problem with expected value maximization here is that expected value is a terrible statistic for this problem. In Alice's game, her outcomes converge to the expected value. In Bob's game, his outcomes if he expectation maximizes are basically as far from the expected value as they can be.
    This is why Bob should treat his game like a time-average. I highly recommend Ole Peters...
    9 min
  • LW - Morality is Accidental & Self-Congratulatory by ymeskhout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Morality is Accidental & Self-Congratulatory, published by ymeskhout on May 29, 2023 on LessWrong.
    The Selfish Gene remains one of my favorite books of all time and although published in 1976 it remains a compelling and insightful introduction to the brute mechanics of natural selection. Richard Dawkins later acknowledged that his book’s title may give a misleading impression of its thesis, erroneously ascribing conscious motivations or agentic properties to non-sentient strands of DNA. The core argument is dreadfully simple: genes (as opposed to organisms or species) are the primary unit of natural selection, and if you leave the primordial soup brewing for a while, the only genes that can remain are the ones with a higher proclivity towards replication than their neighbors. “Selfish” genes therefore are not cunning strategists or followers of some manifest destiny, but rather simply the accidental consequence of natural selection favoring their propagation. Nothing more.
    Dawkins is responsible for coining the word ‘meme’ in the book to describe how the same principles behind gene replication can apply to ideas replicating. I thought about this when I read WoodFromEden’s post about the origin of patriarchy. Their explanation for why male dominance persisted historically for so long is elegantly tidy:
    Men make war. Or rather, groups of men make war. The groups that were good at making war remained. The groups that were less good at making war perished. That way, human history is a history of successful male military cooperation. Groups with weak male bonding were defeated by groups where men cooperated better.
    Here too, there is no dirigible trajectory mapped out ahead of time. Cultural values which valorize physical male violence and facilitate its coordination at scale will become the dominant paradigm purely as a result of the circumstances’ ruthless logic. Any deviation from this set of values would lead your tribe towards extinction, which accidentally also meant your bards wouldn’t be around to write songs and poems extolling the virtues of sex equality. At least not until there have been an extensive change in circumstance.
    This “security dilemma” may have been borne out of petty squabbles over hunting grounds in the Serengeti but its ramifications persisted throughout history. Military service today may be seen as a low-status and distasteful profession — quite literally grunt work — but it used to be venerated deeply as a path to honor and a cornerstone of civic duty. This philosophy is epitomized by the recurring and central portrayal of military men in stories from a long time ago (Homeric heroes of ancient Greece, Genghis Khan, Jedi knights, etc.), their deeds forming the backbone of societal narratives and cultural mythologies.
    The historian Bret Deveraux analyzed the grand strategy video game Europa Universalis 4 to illustrate the war-hungry reality of the late medieval period:
    Military power requires revenue and manpower (along with staying technologically competitive) and both come from the same source: the land. While a player can develop existing provinces, taking land in war is far cheaper and faster. The game represents this through both developing old land and seizing new land requiring similar resources [but compared to incorporating newly conquered land, development is about 4x as expensive while providing only marginal improvements]. That may seem like the developer has placed their thumb a bit unfairly on the scale, but, as Azar Gat notes in War in Human Civilization (2006) for pre-industrial societies that is a historically correct thumb on the scale. Until the industrial revolution, nearly all of the energy used in production came out of agriculture one way or another; improves in irrigation, tax collection and farming methods might improve...
    10 min
  • LW - TinyStories: Small Language Models That Still Speak Coherent English by Ulisse Mini
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: TinyStories: Small Language Models That Still Speak Coherent English, published by Ulisse Mini on May 28, 2023 on LessWrong.
    Abstract
    Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention). In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4.
    We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities. We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model's output to be very structures, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency.
    We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.
    Implications
    Interpretability
    One part that isn't mentioned in the abstract but is interesting:
    We show that the trained SLMs appear to be substantially more interpretable than larger ones. When models have a small number of neurons and/or a small number of layers, we observe that both attention heads and MLP neurons have a meaningful function: Attention heads produce very clear attention patterns, with a clear separation between local and semantic heads, and MLP neurons typically activated on tokens that have a clear common role in the sentence. We visualize and analyze the attention and activation maps of the models, and show how they relate to the generation process and the story content.
    The difference between highly activating tokens for a neuron is striking, here's the tiny model:
    ...and here's GPT2-XL:
    Capabilities
    Again from the introduction (emphasis mine)
    However, it is currently not clear whether the inability of SLMs to produce coherent text is a result of the intrinsic complexity of natural language, or of the excessive breadth and diversity of the corpora used for training. When we train a model on Wikipedia, for example, we are not only teaching it how to speak English, but also how to encode and retrieve an immense amount of facts and concepts from various domains and disciplines. Could it be that SLMs are overwhelmed by the amount and variety of information they have to process and store, and that this hinders their ability to learn the core mechanisms and principles of language? This raises the question of whether we can design a dataset that preserves the essential elements of natural language, such as grammar, vocabulary, facts, and reasoning, but that is much smaller and more refined in terms of its breadth and diversity.
    If this is true, there could be ways to drastically cut LLM t...
    4 min
  • AF - An LLM-based “exemplary actor” by Roman Leventov
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An LLM-based “exemplary actor”, published by Roman Leventov on May 29, 2023 on The AI Alignment Forum.
    Into and summary
    This post is the second section of "Aligning an H-JEPA agent via training on the outputs of an LLM-based "exemplary actor", posted separately because I think it could warrant a separate discussion, largely independent of the discussion of H-JEPA agent with GFlowNet actors. Here's the summary of this post, copied from the "Overview" section of the main article:
    In section 2, I describe the “exemplary actor”, an LMCA (language model cognitive architecture) that takes a simple, “brute force” approach to alignment: a powerful LLM (think GPT-5/6 level, with a vast, or quasi-unlimited context) is given a list of “approved” textbooks on methodological and scientific disciplines: epistemology, rationality, ethics, physics, etc. Also, the LLM is given tools: narrow AIs (such as for protein folding or for predicting properties of materials, or for formal scientific modelling). Finally, the LLM is given a compute engine such as Wolfram and a knowledge base such as Wikidata or Wolfram Knowledgebase.
    The exemplary actor creates plans or predictions for given situations (described in language and fed to the LLM underlying the exemplary actor as prompts) and iteratively critiques and refines its own plans and predictions while putting different textbooks into the LLM context (first, with the textbook on rationality, then epistemology, then physics, etc., with potentially dozens of different textbooks relevant for a plan or prediction that is being criticised), for many iterations, until convergence.
    In section 2.1, I note that the type of alignment that the exemplary actor’s architecture tries to ensure is called (world) model alignment and that is stronger and also more essential than goal alignment.
    Then, I discuss the properties of the exemplary actor. In section 2.2., I discuss what I see as likely non-issues or straightforwardly addressable issues: the “divergent reasoning nature” of LLMs, the lack of grounded common sense reasoning, and the bias of the quick reactive network (”System 1”), it it is added to the architecture to make it more practically usable in lower-stakes reasoning settings.
    In section 2.3, I discuss the outstanding technical issues and risks of the exemplary actor’s architecture:
    The risk of direct access to the underlying LLM (section 2.3.1).
    The exemplary actor’s reasoning could still be partially directed by “alien” thinking patterns (i.e., the world model) of the underlying LLM even though these influences won’t surface in the explanations of the plan (section 2.3.2).
    Iterated critique and refinement probably won’t make plans strictly conforming to the theories described in the textbooks (section 2.3.3).
    In section 2.3.4, I discuss the alignment tax of the exemplary actor (compared with the baseline of a bare, minimally fine-tuned LLM) and conclude that the main source of alignment tax might happen to be the theory of ethics which may force the exemplary actor to refuse to participate in “games” (i.e., real-world situations and environments) where it doesn’t see ethical ways of “winning”, and thus will consider inaction (or some form of palliative action) the only ethical way forward. This is not a technical problem with the exemplary actor per se, but rather a problem with a higher-level system, i.e., the current economic, social, and political structure of the world. I mention this and other kinds of “higher-level” risks of the plans to build and deploy the exemplary actor (i.e., roughly the plans that OpenAI and Anthropic are betting on, as it seems to me) in section 2.4.
    2. An LLM-based “exemplary actor”
    Let's assume that we have three things:
    First, a very powerful auto-regressive LLM (think GPT-5/6 level) with the ability to effe...
    25 min