The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - PIBBSS (AI alignment) is hiring for a Project Manager by Nora Ammann
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PIBBSS (AI alignment) is hiring for a Project Manager, published by Nora Ammann on September 19, 2022 on The AI Alignment Forum.
    TLDR; PIBBSS is hiring for a full-time Project Manager who will be responsible for running the second iteration of the PIBBSS Summer Research Fellowship.
    To apply, please complete this application form.
    PIBBSS aims to facilitate knowledge transfer from fields studying intelligent behaviour in natural systems to AI safety and alignment.
    The Project Manager will be supported by TJ and Nora (who ran the fellowship in 2022) to help transfer learnings from last year’s fellowship, and work alongside (and manage) 1-3 team members to help execute the program.
    More information about the role here.
    Application form
    We accept and evaluate applications on a rolling basis. Note that we are looking to hire as soon as possible and no later than early October.
    We’re happy to discuss this opportunity with any potential applicants. Feel free to contact me with any questions you might have at: [email protected].
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - The Next EA Global Should Have Safe Air by joshcmorrison
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Next EA Global Should Have Safe Air, published by joshcmorrison on September 19, 2022 on The Effective Altruism Forum.
    One of the coolest EA things I saw during the pandemic was the creation of the microCOVID risk tracker by an EA group house in San Francisco. To me, it was a really inspiring example of the principles of effective altruism in action — using rationality and curiosity to solve a concrete problem to make people’s lives better.
    I was having a dinner party with some friends in last night with a theme of how we could improve indoor air safety, starting with our local community in New York. (Some background here on how my colleagues at 1Day Sooner and i think about the air safety problem). How can we get buildings to clean the air (by filtering it, mixing it with outdoor air, and sterilizing it with ultraviolet light) so that people don’t suffer from pollution and pathogens?
    We were discussing what was feasible to accomplish politically and were struggling because a standard answer to “what air safety interventions are optimal for a space to adopt?” doesn’t yet exist. We agreed that it would be uniquely valuable to recruit early adopters (e.g. tech companies, private schools, universities) to try out solutions and test them for effectiveness in reducing disease. If well-designed, this could generate experimental evidence on effectiveness and create a template for later adopters and governments to implement.
    An obvious place to start would be the EA community and trying to get EA spaces to implement air safety measures (like installing filters and upper-room UV light). There are a number of organizations that could fit the bill, and I’m aware of at least one that is exploring doing this in their own office.
    One suggestion that uniquely resonated with me was the idea that the next EA Global (after EAG DC) should make its air safe. (That is, it should have a respiratory infection risk level it tries to achieve, some surrogate targets it aims to measure, and a set of indoor air interventions that are reasonably likely to achieve the intended risk level).
    I don’t think this will be easy and in fact I think it might be more likely than not that we fail. But part of what is valuable about EA is our commitment to learning from failure and improve over time. Trying to implement air safety interventions will teach us about the existing gaps that need to be filled, which will get us closer for the next EAG (and EAGx) until we get to a point where we’re proud of our community for becoming safer and a better model for achieving good outcomes elsewhere.
    I recognize it already takes a tremendous amount of effort to run EA Global, and I appreciate the work CEA does putting these events on. So my intention is not to create additional burden. But biosecurity is a cause many EAs are passionate about, and air safety is one of the most promising interventions to achieve deterrence-by-denial of engineered respiratory biothreats. I feel like making our own spaces safe from pathogens is a challenge that our community can and should rise to and that doing so will have outsized benefits on our ability to accomplish future policy. If you're interested in helping with this, let me know.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Prize and fast track to alignment research at ALTER by Vanessa
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prize and fast track to alignment research at ALTER, published by Vanessa on September 18, 2022 on The Effective Altruism Forum.
    Cross-posted from the AI Alignment Forum.
    On behalf of ALTER and Superlinear, I am pleased to announce a prize of at least 50,000 USD, to be awarded for the best substantial contribution to the learning-theoretic AI alignment research agenda among those submitted before October 1, 2023. Depending on the quality of submissions, the winner(s) may be offered a position as a researcher in ALTER (similar to this one), to continue work on the agenda, if they so desire.
    Submit here.
    Topics
    The research topics eligible for the prize are:
    Studying the mathematical properties of the algorithmic information-theoretic definition of intelligence.
    Building and analyzing formal models of value learning based on the above.
    Pursuing any of the future research directions listed in the article on infra-Bayesian physicalism.
    Studying infra-Bayesian logic in general, and its applications to infra-Bayesian reinforcement learning in particular.
    Theoretical study of the behavior of RL agents in population games. In particular, understand to what extent infra-Bayesianism helps to avoid the grain-of-truth problem.
    Studying the conjectures relating superrationality to thermodynamic Nash equilibria.
    Studying the theoretical properties of the infra-Bayesian Turing reinforcement learning setting.
    Developing a theory of reinforcement learning with traps, i.e. irreversible state transitions. Possible research directions include studying the computational complexity of Bayes-optimality for finite state policies (in order to avoid the NP-hardness for arbitrary policies) and bootstrapping from a safe baseline policy.
    New topics might be added to this list over the year.
    Requirements
    The format of the submission can be either a LessWrong post/sequence or an arXiv paper.
    The submission is allowed to have one or more authors. In the latter case, the authors will be considered for the prize as a team, and if they win, the prize money will be split between them either equally or according to their own internal agreement. For the submission to be eligible, its authors must not include:
    Anyone employed or supported by ALTER.
    Members of the board of directors of ALTER.
    Members of the panel of the judges.
    First-degree relatives or romantic partners of judges.
    In order to win, the submission must be a substantial contribution to the mathematical theory of one of the topics above. For this, it must include at least one of:
    A novel theorem, relevant to the topic, which is difficult to prove.
    A novel unexpected mathematical definition, relevant to the topic, with an array of natural properties.
    Some examples of known results which would be considered substantial at the time:
    Theorems 1 and 2 in "RL with imperceptible rewards".
    Definition 1.1 in "infra-Bayesian physicalism", with the various theorems proved about it.
    Theorem 1 in "Forecasting using incomplete models".
    Definition 7 in "Basic Inframeasure Theory", with the various theorems proved about it.
    Evaluation
    The evaluation will consist of two phases. In the first phase, I will select 3 finalists. In the second phase, each of the finalists will be evaluated by a panel of judges comprising of:
    Adam Shimi
    Alexander Appel
    Daniel Filan
    Vanessa Kosoy (me)
    Each judge will score the submission on a scale of 0 to 4. These scores will be added to produce a total score between 0 and 16. If no submission achieves a score of 12 or more, the main prize will not be awarded. If at least one submission achieves a score of 12 or more, the submission with the highest score will be the winner. In case of a tie, the money will be split between the front runners.
    The final winner will be announced publicly, but the scores received by various submissions...
    6 min
  • LW - Pretending not to Notice by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pretending not to Notice, published by jefftk on September 19, 2022 on LessWrong.
    Sometimes parents will pretend not to notice kids doing things they shouldn't: why might you do this and when is it a good idea?
    My experience has been that things go much better when parents set clear boundaries. If it's sometimes ok to pick the flowers and other times you get told to stop, there's a lot of trying to figure out what makes these particular flowers or this particular time different from others. Some complexity is unavoidable (you can't pick flowers from the neighbor's garden, you can pick all the dandelions you want) but a parent who's not thinking about it can accidentally make things harder for their kids by being inconsistent (only sometimes stopping them from picking flowers that shouldn't be picked, not making up their mind about which flowers are ok).
    With very little kids, what matters is what happens when they try to pick the flower. If you don't stop them, that's some information about what's ok; if you do, it's some information about what's not. Kids who are a bit older are smarter though: they care about whether you've observed them. Picking a flower without objection while being watched is a signal that the flower is fine to pick, but picking a flower while not being watched doesn't actually tell you very much about what's ok. It does tell you a little: it's probably not so hugely bad to pick flowers that a sensible adult would continue supervising. But mostly they understand to only update their model of the boundaries the adult is enforcing when the adult is paying attention.
    Which means, very occasionally, you might want to pretend to not observe something. If they believed you'd seen it, you'd need to enforce the boundary or they'd think it had changed. But by pretending not to notice you can give them close to "no signal", as if you weren't there at all.
    For example, let's say you have been enforcing a clear "no hitting" rule, and your kid hits an older kid, someone mature enough to stand up for themself. With the boundary you had been enforcing you would need to step in, but, if they both believe you didn't see, they can get practice in resolving their dispute independently ("Don't hit me!") or at least in bringing their concern to you ("Alex hit me!"). This could be part of a process of moving away from a toddler's
    "hitting is not allowed" to a Kindergartener's "hitting is not allowed, but try to work it out on your own first".
    The most blatant forms of this stop working in, maybe, early elementary school, once they get smart enough to either put together patterns of when you don't seem to notice things, or learn what you look like when you're pretending not to notice. You probably can't
    actually give no signal to a sharp-eyed child; I know I can't. And it stops working earlier the more you do it, because they'll have more chances to learn your patterns.
    I don't like that it involves misleading them slightly, but doing it occasionally in ways that help them grow seems positive on balance.
    Comment via: facebook
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - How my team at Lightcone sometimes gets stuff done by jacobjacob
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How my team at Lightcone sometimes gets stuff done, published by jacobjacob on September 19, 2022 on LessWrong.
    Disclaimer: I originally wrote this as a private doc for the Lightcone team. I then showed it to John and he said he would pay me to post it here. That sounded awfully compelling. However, I wanted to note that I’m an early founder who haven't built anything truly great yet. I’m writing this doc because as Lightcone is growing, I have to take a stance on these questions. I need to design our org to handle more people. Still, I haven’t seen the results long-term, and who knows if this is good advice. Don’t overinterpret this.
    Suppose you went up on stage in front of a company you founded, that now had grown to 100, or 1000, 10 000+ people. You were going to give a talk about your company values. You can say things like “We care about moving fast, taking responsibility, and being creative” -- but I expect these words would mostly fall flat. At the end of the day, the path the water takes down the hill is determined by the shape of the territory, not the sound the water makes as it swooshes by. To manage that many people, it seems to me you need clear, concrete instructions. What are those? What are things you could write down on a piece of paper and pass along your chain of command, such that if at the end people go ahead and just implement them, without asking what you meant, they would still preserve some chunk of what makes your org work?
    Here’s my current best guess at how I would do this for Lightcone Infrastructure, the organisation where I spend the majority of my waking hours. I wrote it by thinking about how the team I'm on has actually operated during periods of high output, and then trying to turn that into a set of rules. Others on the team might disagree about which rules matter and where the magic sauce is, but I think this is at least empirically descriptive of how my team spends much of our time.
    0. Blockers are death. Above all else, your job is to unblock anything that prevents you from moving as fast as you can toward your top priority.
    The organisation has one CEO who’s the final decision-maker on all decisions, unless they explicitly delegate a decision. Consensus is slow and blocking. Having a tie-breaker means you can move faster.
    Start each Monday with an all-hands meeting where the CEO sets or clarifies the top priority of the organisation.
    After the Monday meeting, there’s a block of time on everyone’s calendar during which no one is allowed to schedule any meetings in advance. This is the “top priority block”, where everyone’s sole goal is to work on whatever the top priority is, as identified in the all-hands meeting. The point of not scheduling meetings is so that no one ends up blocked and waiting for someone else to come out of a call or meeting (that’s not itself about the top priority). Any person on the team who could unblock someone else’s pursuit of the top priority, is available to do so during this block.
    Have a single day, e.g. Tuesday, that’s the “meeting day”, where people are expected to schedule any miscellaneous, external meetings (e.g. giving someone career advice, or grabbing coffee with a contact). The reason I find this important is that, if you don’t have it, people end up scheduling their meetings at random, uncoordinated times during the week. And this means that there ends up being very few slots where the people who might unblock each other are both free at the same time.
    No remote work. Everyone is in the office. If you need to be unblocked by someone, the fastest way is to just go to their desk and ask them in person.
    People on the same team work in the same room. Sudden questions, comments, or info sharing out loud is encouraged. If people want to focus deeply for a while, they can put on headphones.
    For...
    12 min
  • EA - Levels of donation by vipulnaik
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Levels of donation, published by vipulnaik on September 18, 2022 on The Effective Altruism Forum.
    Via the MagnatesMedia YouTube video on the subject, I came across a
    Reddit post on levels of wealth. The post separates out different levels of wealth and the qualitative differences between them, with the four levels being $10-30 million, $30-100 million, $100 million-$1 billion, and $1 billion+.
    As with all such analyses, the threshold selection is somewhat arbitrary, and a lot of nuance is missed, but I still find the exercise interesting.
    I thought it would be interesting to do a somewhat similar exercise for levels of donation; this seemed like a good prompt in response to which I could express several thoughts that have been floating around in my mind.
    This post looks at various levels of donation as well as the ways one could level up (if deemed desirable).
    The levels of donation
    Level 1: retail donor (< $1,000)
    Lower limit for level 1: retail donor: $0
    Upper limit for level 1: retail donor: $500 to $1,000
    Prototypical donor: Low-income person (typically young, such as a student) donating some money; higher-income person donating 1% of income.
    Split between donees? (no): The one-charity argument is pretty much definitely true here; it's very hard to imagine a case where the marginal value functions of two donees cross within this small an amount of donation.
    Investigate? (no): It probably doesn't make sense to spend extensive resources investigating by yourself or hiring others to investigate. You should either donate directly to the best endpoint donee, or give money to a reallocator (such as the EA Funds, GiveWell Maximum Impact Fund, or GiveWell All Grants Fund) that is sufficiently value-aligned with you. It may not even make sense to write extensively about your decision.
    Engage donee? (no): It probably doesn't make sense to ask a bunch of questions of the donee or expect any kind of personalized response; it's best to rely on information you already have as a donor.
    Optimize for transaction costs? (no): You should probably just use whatever donation method the donee recommends, rather than reaching out to the donee to figure out the best method for your particular case (for instance, if they suggest using PayPal, use
    PayPal, even if you know it has network fees that you'd rather not spend).
    As a data point, GiveWell's page on how to donate sets a threshold for $1,000 for when you should switch from donating by credit card to donating by check.
    Optimize for taxes? (no): Optimizing donation timing and other donation mechanics for tax considerations probably doesn't make sense at this level.
    Level 2 (four-digit donors)
    Lower limit for level 2: $500 to $1,000
    Upper limit for level 2: $5,000 to $10,000
    Prototypical donors:
    Person or couple with median income (by first-world standards) donating a nontrivial percentage of income annually (such as the 10% set by the Giving What We Can Pledge).
    Person with higher income donating a smaller percentage of salary or donating at a cadence more frequency than annual.
    Considerations:
    Split between donees? (conceivable but unlikely): It's conceivable (though unlikely) that the one-charity argument starts breaking down here, if the expected best use of your funds also has sharply diminishing marginal value (e.g., helping an organization get off the ground, or close a small funding gap). So you might find yourself splitting your donation.
    Investigate? (yes, a few hours): If you don't have any obvious endpoint donee or reallocator with whom you are closely value-aligned a priori, it may make sense to spend a few hours investigating the options. It may also make sense to document your thinking.
    Engage donee? (maybe, hope for responsiveness but not proactive effort from donee): It may make sense to ask questions of potential done...
    19 min
  • AF - The Inter-Agent Facet of AI Alignment by Michael Oesterle
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Inter-Agent Facet of AI Alignment, published by Michael Oesterle on September 18, 2022 on The AI Alignment Forum.
    We argue that intent alignment and capability robustness do not capture the full “chain of command” (i.e., “alignment” without any qualifier) from the programmer's intentions to an AI agent’s real-world impact, but stops at the stage where the agent performs its actions. We argue that, especially in complex Multi-Agent Systems (MAS), this is not sufficient to actually achieve aligned outcomes. Therefore, we introduce another facet of AI Alignment, which deals with the connection between an agent’s intention and its actual impact, due to multi-agent interactions (as opposed to capability robustness).
    Motivation
    Independent of whether the first AGIs will be Tool or Agent AIs, they will most likely not live and act in an empty space. Instead, they will interact and communicate with humans and other AGIs in a variety of ways, thereby effectively creating a socio-technical system or an AI-based society.
    As a consequence, the effects that an AGI will have on its environment, i.e., the world, will not necessarily coincide with the effects that the AGI plans to have, since actions in such systems commonly interfere and influence/cancel each other.
    This, in turn, leads to the conclusion that it is not sufficient to build an AGI which satisfies intent alignment and capability robustness. Instead, we need one more step to be controlled: The step from an AGI's robust intention to its societal impact.
    Schema
    In Evan Hubinger’s alignment terminology, alignment is the result of intent alignment (which is, in turn, composed of the well-known concepts of inner and outer alignment, among others) and capability robustness. The implicit assumption is that the AGI's model is equivalent to how its actions will affect the world.
    Inter-Agent Alignment
    Inter-Agent Alignment is aligning the (aligned and robust) intentions of an AGI agent with the actual outcome within the multi-agent system of which the agent is a part. Since the impact of the agent’s actions depends on what everyone else in this system is doing (and not just its own actions), inter-agent alignment is a non-trivial task.
    Our central argument is that multi-agent interactions add a sufficient amount of complexity to an AI agent’s action policy to justify its own type of alignment, separated from the existing alignment model. Adding inter-agent alignment to the alignment tree, we see that it complements intent alignment and capability robustness, resulting in full alignment (i.e., impact alignment).
    Challenging Questions
    Why is inter-agent alignment not just a part of intent alignment?
    Intent alignment is a part of a single AGI without any reference to multi-agent systems. Of course, we could expand its meaning, but since the steps between the programmer's intentions and the AGI's impact can be neatly broken down, we argue that it makes sense to introduce a new term for this additional dimension.
    Naturally, one can always argue that a sufficiently advanced AGI will be able to see through this dilemma and resolve it, but this seems to be a variant of the No-True-Scotsman fallacy: As long as we can imagine an AGI which is advanced enough to interact with other agents, but not able to successfully coordinate their actions and therefore avoid unintended outcomes, inter-agent alignment is a problem that needs to be addressed. Independent learners using state-of-the-art RL algorithms like DQN or PPO provide evidence that these “medium-level” agents do, in fact, exist.
    What does it look like if an AGI is intent aligned and capability robust, but not inter-agent aligned?
    The Tragedy of the Commons is the most prominent example of the fact that the result of selfish optimization of multiple agents is not always optimal fo...
    9 min
  • AF - Summaries: Alignment Fundamentals Curriculum by Leon Lang
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summaries: Alignment Fundamentals Curriculum, published by Leon Lang on September 18, 2022 on The AI Alignment Forum.
    The linked document provides my summaries for most core readings and many further readings of the alignment fundamentals curriculum composed by Richard Ngo, as accessed from July to early September 2022. Additionally, it often contains my preliminary opinions on the texts. Note that I’m not an expert on the topic.
    I have read all texts while simultaneously doing full-time work unrelated to AI alignment, and thus, due to time constraints, many summaries probably contain mistakes, and my opinions would change upon further reflection. Additionally:
    I only streamlined the process after a few weeks
    the summaries of the first weeks are of lower quality, and more of them or my opinions are missing
    Some summaries are also missing since I had a minor repetitive strain issue along the way, and since the curriculum changed while reading through it
    Sometimes, the formatting is not ideal since I originally wrote the summaries on a slack channel and then copy-pasted them to google docs
    Nevertheless, I was told that these summaries are useful, and therefore I’m sharing them with the wider community of people interested in alignment.
    If anyone wants to contribute their own summary, please put a suggestion into the google doc, and I will accept it with an attribution to the (optionally anonymous) author.
    Acknowledgments: I want to thank Albert Garde, Benjamin Kolb, Fritz Dorn, Jens Brandt, and Tom Lieberum for discussions on the curriculum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Inner alignment: what are we pointing at? by Luke H Miles
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inner alignment: what are we pointing at?, published by Luke H Miles on September 18, 2022 on The AI Alignment Forum.
    Proof that a model is an optimizer says very little about the model. I do not know what a research group is studying outer alignment is studying. Inner alignment seems to cover the entire problem at the limit. Whether an optimizer is mesa or not depends on your point of view. These terms seem to be a magnet for confusion and debate. I have to do background reading on someone to even understand what claim they're making. These are all indicators that we are using the wrong terms.
    What are we actually pointing at? What questions do we want answered?
    Do we care if a model is an optimizer? Is it important whether it is creating plans through an explicit search process or a clever collection of heuristics? A poor search algorithm cannot plan much and clever enough heuristics can take you to any goal. What's the important metric?
    Sometimes a model will have great capacity to shape its environment but little inclination. How to divide between capacity and inclination in a way that closely corresponds to agents and models as we observe them? (One could say that capacity and inclination cannot be separated but the right definitions would split them right apart.)
    When you specify what you want the model to do in code, what is the central difficulty? Is there a common risk or error in giving examples and giving loss/reward/value functions that we can name?
    Is there a clear, accepted term for when models do not maintain desired behavior under distribution shift?
    Should we distinguish between trained RL models that optimize and spontaneous agents that emerge in dynamical systems? One might expect the first to almost always happen and the second very rarely. What's the key difference?
    I'll post my answers to these questions in a couple days but I'm curious how other people slice it. Does "inner alignment failure" mean anything or do we need to point more directly?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Cardinals baseball tickets currently a cost-effective way to fundraise by Pat Myron
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cardinals baseball tickets currently a cost-effective way to fundraise, published by Pat Myron on September 17, 2022 on The Effective Altruism Forum.
    Albert Pujols hit his 698th career home run tonight, and he graciously encourages fans not to worry about returning baseballs. Inflation-adjusted, the last 700th career home run baseball sold for over $1M USD Players average ~3 at-bats per game, leading to ~30% chance Pujols hits at least one HR each game he plays lately: 3 HRs on 33 AB last 10 games, 20 HRs on 146 AB since July 1 (54 games)
    Expected revenue from auctioning the ball is now hundreds of thousands per game
    Current cheapest outfield ticket prices before fees by city for the rest of their season:
    St. Louis: $45 (41k / 45k - avg attendance / capacity)
    San Diego: $30 (37k / 43k)
    LA Dodgers: $30 (48k / 56k)
    Milwaukee: $10 (31k / 42k)
    Pittsburgh: $20 (16k / 38k)
    Since the vast majority of seats aren't outfield seats, the expected value for the minority in all the outfield seats still seems like hundreds of thousands per game even after ticket costs, so you should consider them if you don't mind going:
    Other considerations that slightly alter your chances of making a catch:
    Pujols got moved up in the hitting lineup for more ABs
    Home run odds vary slightly by stadium
    Average MLB fan is almost 60 years old
    Go easy on drinks and bring gloves :)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min