The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - 🤔 Coordination explosion before intelligence explosion...? by tailcalled
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 🤔 Coordination explosion before intelligence explosion...?, published by tailcalled on March 5, 2023 on LessWrong.
    Epistemic status: musing that I wanted to throw out there.
    A traditional AI risk worry has been the notion of an intelligence explosion: An AI system will rapidly grow in intelligence and become able to make huge changes using small subtle tricks such as bioengineering or hacking. Since small actions are not that tightly regulated, these huge changes would be made in a relatively unregulated way, probably destroying a lot of things, maybe even the entire world or human civilization.
    Modern AI systems such as LLMs seem to be making rapid progress in turning sensory data into useful information, aggregating information from messy sources, processing information in commonsense ways, and delivering information to people. These abilities do not seem likely to generalize to bioengineering or hacking (which involve generating novel capabilities), but they do seem plausibly useful for some things.
    Two scenarios of interest:
    Coordination implosion: Some people suggest that because modern AI systems are extremely error-prone, they will not be useful, except for stuff like spam, which degrades our coordination abilities. I'm not sure this scenario is realistic because there seem to be a lot of people working on making it work for useful stuff.
    Coordination explosion: By being able to automatically do basic information processing, it seems like we might be able to do better coordination. We are already seeing this with chatbots that work as assistants, sometimes being able to give useful advice based on their mountains of integrated knowledge. But we could imagine going further, e.g. by automatically registering people's experiences and actions, and aggregating this information and routing it to relevant places.
    (For instance, maybe a software company installs AI-based surveillance, and this surveillance notices when developers encounter bugs, and takes note of how they solve the bugs so that it can advise future developers who encounter similar bugs about what to do.)
    This might revolutionize the way we act. Rather than having to create, spread, and collect information, maybe we would end up always having relevant information at hand, ready for our decisions. With a bit of rationing, we might even be able to keep spam down to a workable level.
    I'm not particularly sure this is what things are going to look like. However I think the possibility is useful to keep in mind: There may be an intermediate phase between "full AGI" and now, where we have a sort of transformative artificial intelligence, but not in the sense of leading to an intelligence explosion. There may still be an intelligence explosion afterwards. Or not, if you don't believe in intelligence explosions.
    I foresee privacy to be one counteracting force. These sorts of systems seem like they work better when they invade your privacy more, so people will resist that.
    Small = Involving relatively minor changes in terms of e.g. matter manually moved.
    Subtle = Dependent on getting many "bits" right at a distance.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - EA Infosec: skill up in or make a transition to infosec via this book club by Jason Clinton
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Infosec: skill up in or make a transition to infosec via this book club, published by Jason Clinton on March 5, 2023 on The Effective Altruism Forum.
    Ahoy! Our community has become acutely aware of the need for skilled infosec folks to help out in all cause areas. The market conditions are that information security skilled individuals are in shorter supply than demand. This book club aims to remedy that problem.
    I have been leading the Chrome Infrastructure Security team at Google for 3 years, have 11 years of infosec experience, and 24 years of career experience. My team’s current focus includes APT and insider defense. I built that team with a mix of folks with infosec skills—yes—but the team is also made up of individuals who were strong general software engineers who had an interest in security. I applied this book and a comprehensive, 18 month training program to transition those folks to infosec and that has been successful. Reading this book as a book club is the first 5 months of that program. So, while this book club is not sufficient to make a career transition to infosec, it is a significant first step in doing so.
    The goal of this group and our meetings is to teach infosec practices, engineering, and policies to those who are interested in learning them, and to refresh and fill in gaps in those who are already in the infosec focus area.
    Find the book as a free PDF or via these links. From the book reviews:
    This book is the first to really capture the knowledge of some of the best security and reliability teams in the world, and while very few companies will need to operate at Google’s scale many engineers and operators can benefit from some of the hard-earned lessons on securing wide-flung distributed systems. This book is full of useful insights from cover to cover, and each example and anecdote is heavy with authenticity and the wisdom that comes from experimenting, failing and measuring real outcomes at scale. It is a must for anybody looking to build their systems the correct way from day one.
    This is a dry, information-dense book. But it also contains a comprehensive manual for how to implement what is widely considered the most secure company in the world.
    Audience
    Any software engineer who is curious about becoming security engineering focused or anyone looking to up their existing infosec career path. It is beyond the level of new bachelor’s graduates. However, anyone with 3-ish years of engineering practice on real-world engineering systems should be able to keep up. A person with a CompSci masters degree but no hands-on experience might also be ready to join.
    Openness
    Directed to anyone who considers themselves EA-aligned. Will discuss publicly known exploits and news stories, as they relate to the book contents, and avoid confidential cases from private orgs. Will discuss applicability to various aspects of EA-aligned work across all cause areas.
    Format, length, time and signup
    Meet for 1 hour on Google Meet every 2 weeks where we will discuss 2 chapters. ~11 meetings over 22 weeks.
    The meetings will be facilitated by me.
    The discussion format will be:
    The facilitator will select a theme from the chapters, in order, and then prompt the participants to offer their perspective, ensuring that everyone has ample opportunity to participate, if they choose.
    Discussion on each theme will continue for 5-10 minutes and then proceed to the next theme. Participants should offer any relevant, current news or applicability to cause areas, if time permits.
    The facilitator will ensure that discussion is relevant and move the conversation along to the next topic, being mindful of the time limit.
    Any threads that warrant more discussion than we have time for in the call will be taken to the Slack channel for the book club (see form below for invite) where pa...
    5 min
  • EA - The Role of a Non-Profit Board by Grayden
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Role of a Non-Profit Board, published by Grayden on March 4, 2023 on The Effective Altruism Forum.
    Background
    This is a cross-post from the website of the EA Good Governance Project. It is shared here for the purposes of ensuring it reaches a wider audience and to invite comment.
    The content is largely consensus best-practice adapted for an EA audience. Leveraging this content should help boards be more effective, but governance is complex, subjective and context dependent, so there is never a right answer.
    Introduction
    The responsibilities of a board can fall largely into three categories: governance, advisory and execution. Here, we explain how to work out what falls in which category and key considerations about whether the board should be involved.
    Essential: Governance
    The Board comprises individuals who hold assets “on trust” for the beneficiaries. By default, all power is held by the board until delegated, and the board always remains responsible for ensuring the organization delivers on its objects.
    In practice, this means:
    Appointing the CEO, holding them to account and ensuring their weaknesses are compensated for;
    Taking important strategic decisions, especially those that would bind future CEOs;
    Evaluating organizational performance and testing the case for its existence; and
    Ensuring the board itself has the right composition and is performing strongly.
    Being good at governance doesn’t just mean having the right intentions; it requires strong people & organization skills, subject matter expertise and cognitive diversity.
    When founding a new non-profit, it is often easiest to fill the board with friends. However, if we are to hold ourselves up to the highest standards of rationality, we should seek to strengthen the board quickly.
    Optional: Advisory
    The best leaders know when and where to get advice. This might be technical in areas where they are not strong, such as legal or accounting, or it might be executive coaching to help an individual build their own capabilities, e.g. people management.
    It is common for board members to fill this role. There is significant overlap in the skills required for governance and the skills that an organization might want advice on. For example, it is good for at least one member of the board to have accounting experience and a small organization might not know how to set up a bookkeeping system. Board members also already have the prerequisite knowledge of the organization, its people and its direction. However, there is no need for advisors to be on the Board.
    We recommend empowering the organization’s staff leadership to choose their own advisors. The best mentoring relationships are built informally over time with strong interpersonal fit. If these people are members of the board, that’s fine. If they are not, that’s also fine.
    The board should build itself for the purpose of governance. The executives should build a network of advisors. It is best to keep these things separate.
    Best Avoided: Execution
    In some organizations, Board members get involved in day-to-day execution. This is particularly true of small and start-up organizations that might have organizational gaps. Tasks might include:
    Bookkeeping, financial reporting and budgeting
    Fundraising and donor management
    Line management of staff other than CEO
    Assisting at events
    Wherever practical, this should be avoided. Tasks undertaken by Board members can reduce the Board’s independence and impede governance. The tasks themselves often lack proper accountability.
    If new opportunities, such as a potential new project or employee, come through Board members, they should be handed over to staff members asap. It’s a good idea for Board members to remove themselves from decision-making on such issues, especially if there is a conflict of loyalty or conflict of i...
    4 min
  • AF - Do humans derive values from fictitious imputed coherence? by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Do humans derive values from fictitious imputed coherence?, published by Tsvi Benson-Tilsen on March 5, 2023 on The AI Alignment Forum.
    [Metadata: crossposted from. First completed November 1, 2022. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text may be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]
    Humans are born with some elements of their minds, and without many other elements, some of which they'll acquire as their life unfolds. In particular, the elements that we pretheoretically call "values"--aesthetic preferences, goals, life goals, squad goals, aspirations, needs, wants, yearnings, drives, cravings, principles, morals, ethics, senses of importance, and so on--are for the most part acquired or at least unfolded, rather than being explicitly present in a newborn. How does this happen? What generates these mental elements?
    Hypothesis: a human derives many of zer values by imputing coherent agency to zer past behavior, and then adopting the goals of that fictitious agency as actively influential criteria for future action.
    Thanks to Sam Eisenstat for relevant conversations.
    The FIAT hypothesis
    As a shorthand: "the FIAT hypothesis" = "the Fictitious Imputed Adopted Telos hypothesis". ("Fiat" is Latin for "may it happen" or "may it be made", which has some resonance with the FIAT hypothesis in that they both talk about a free creation of goals.) FIAT goals are goals imputed to some behavior and then adopted as goals.
    Human behavior is determined by many things: built-in behavior-determiners such as the instinctive ability to breath, socially learned behavior and values, convergent instrumental goals, and freely created autopoietic goals such as artistic goals. The FIAT hypothesis says that a major determiner of a human's behavior is the process of adopting goals based on interpreting zer past behavior as agentic.
    Ze can be interpreted as asking the question: if my past behavior were the behavior of a coherent agent trying to do something, what would that something be? Then, whatever the answer was, ze adopts it as a goal--a target of more coherent behavior (more effective, more strategic, more orchestrated, more coordinated, more conscious, better resourced, more reflective, more univocal, more wasteless).
    This hypothesis gives a possible answer to the question: how did evolution build something with some substantial level of agentic coherence, even though evolution can't directly program conscious concepts like "avoiding death" or "saving food" or "inclusive genetic fitness" for use as terms in a utility function for an organism to pursue?
    This process could be continuous, with goals becoming gradually more coherent (and then potentially deprioritized, but usually not de-cohered). This process is iterative, starting with built-in behavior-determiners, then adopting new FIAT goals based on past behavior mainly generated by built-in determiners (and also maybe adopting new goals for other reasons), and then adopting new goals based on past behavior influenced by previously adopted goals, including previous FIAT goals, and so on. FIAT goals also come from not just imputing goals to zer own behavior, but also to the behavior of others, such as parents and leaders. Everything gets enshrined, but everything is open to criticism.
    Note that calling this a hypothesis is maybe presumptuous; it's an idea, but since it's abstract and it's about a complex system, there's a lot of ambiguity between FIAT and other explanations or descriptions of behavior, and it's not necessarily obvious how to make different predictions according to the FIAT hypothesis.
    Something left quite unspecified is how the FIAT process picks different possible interpretations ...
    25 min
  • LW - Does polyamory at a workplace turn nepotism up to eleven? by Viliam
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Does polyamory at a workplace turn nepotism up to eleven?, published by Viliam on March 5, 2023 on LessWrong.
    I am reading debates on the Effective Altruism forum, and they seem to focus on how a low-status person might be afraid to refuse to consent to sex with a high-status person, because the high-status person might then refuse to provide them a grant or a job position.
    True, but this seems to be just the tip of the iceberg.
    Making a high-status polyamorous person angry potentially means not just creating one enemy, but rather a group of enemies -- namely the person's entire polycule. They are naturally likely to trust that person. Even if they have doubts, they still have a selfish incentive not to disrupt their own polycule. And if they often meet and talk in private, the person will be able to tell all of them their version first.
    Imagine the usual monogamous workplace. Imagine rejecting the advances of your manager, and then experiencing retaliation. Imagine contacting HR to complain, and finding out that the HR person handling your complaint is the manager's wife. Would you expect a fair treatment?
    Now imagine a workplace with lots of polyamorous dating. The difference is that if your manager is polyamorous, the chance of meeting a member of their polycule is higher, and you probably wouldn't even know. Or you might give up and try your luck at a different company in this tightly connected small industry... but the new manager, or the job interviewer might be a part of the same polycule, too. Perhaps you have just locked yourself out of the entire industry.
    On the other hand, joining a polycule of high-status people might be the ultimate office politics move. (Unless you had a bad luck, and the polycule turns out to be two-tiered: the high-status members who treat each other seriously, and the "underlings with benefits" that keep coming and going.)
    This is not about "polyphobia".
    I am not saying that monogamy is good and polyamory is bad. The failures of monogamy are numerous and well known, and we have lots of rules created around them. The rules against nepotism were created in the context of monogamy. The rules against sexual harassment were created in the context of (at least officially) monogamy. The archetype of a vindictive rejected man (or a woman) taking a revenge also exists mostly in the context of monogamy. We already know that monogamy can be bad, and we have a set of rules trying to minimize the harm (arguably, not sufficiently, but better than nothing).
    I am just assuming that if members of the same species start dating polyamorously, we should expect some of that to happen there, too. And that perhaps we should adopt similar rules. I expect that most polyamorous people are nice and friendly, just like most monogamous people. I also expect that each group might have a few sexual predators.
    The main difference is that polyamory seems like it could scale up some of the problems. (Larger group size. Possibility to hit on multiple people at the same time without violating a social norm.) The problematic part is "at the workplace"... but polyamory could be a multiplier of the problems.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Selective, Corrective, Structural: Three Ways of Making Social Systems Work by Said Achmiz
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Selective, Corrective, Structural: Three Ways of Making Social Systems Work, published by Said Achmiz on March 5, 2023 on LessWrong.
    (A reference post for a concept that comes up often enough to warrant such a thing.)
    If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary.
    James Madison
    Three kinds of methods
    The one comes to us (wishing to build a social system, and having only people, and no angels, to work with) and says:
    “My system would work perfectly, if only everyone involved would behave in the optimal manner!”
    Granted; unfortunately (or fortunately), not all people can be relied on to behave optimally. How to make the system work despite this?
    There are three sorts of approaches:
    Selective methods—build your system out of only the right sort of people, and exclude the wrong sort.
    Corrective methods—apply such measures as will make the people in your system alter their behavior, to conform to relevant optimality criteria.
    Structural methods—build your system in such a way that it will work if people behave in the ways that they can be expected to behave.
    Examples
    Work
    The challenge: build an organization (or a team within one) that will be able to accomplish various desirable projects.
    Selective: hire people who have the skills/experience/etc. to do the work; don’t hire (or fire, if discovered post-hiring) people who aren’t capable of doing the work.
    Corrective: on-the-job training; social approval/disapproval from coworkers for good/bad work.
    Structural: bonuses and other financial incentives for performance; technological and process improvements that reduce skill requirements.
    World of Warcraft
    The challenge: assemble a raiding guild that will be able to defeat the most challenging boss monsters.
    Selective: accept players who can demonstrate competence in their chosen raid role; exclude those who can’t or won’t perform.
    Corrective: teach inexperienced players to play better; shame lazy or selfish players into putting in effort, and contributing to the guild’s success.
    Structural: assign raid members to roles that best fit their talents and inclinations; design a loot distribution system that incentivizes effort and effective participation.
    Governance
    The challenge: place over society a government, that will rule for the good of all.
    Selective: choose wise and just rulers; prevent the foolish and the wicked from gaining power.
    Corrective: “We demand that public administration be taught by conscientious workers and soldiers and that it be started immediately, i.e. that all the working people, all the poor, should immediately be enlisted in this training.”
    Structural: checks and balances; a nation of laws, not of men.
    Which way is best?
    I have no revelatory answer. Probably it varies from one case to another. And—as the examples show—the approaches aren’t mutually exclusive. All three can be combined, potentially, or any two. Each has its advantages; each, also, its drawbacks. (I will explore some of these in the comments section for this post.)
    The critical thing, I think, is just to be aware that all three types exist.
    Postscript
    This post explains, finally—it only took five years!—what I meant by this comment.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Animal welfare certified meat is not a stepping stone to meat reduction or abolition by Stijn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Animal welfare certified meat is not a stepping stone to meat reduction or abolition, published by Stijn on March 4, 2023 on The Effective Altruism Forum.
    Summary: evidence from a survey suggests that campaigning for farm animal welfare reforms and promoting animal welfare certified meat could in the long run result in a suboptimal state of continued animal suffering and exploitation. Campaigns to reduce or eliminate animal-based meat and promote animal-free meat substitutes are probably more effective in the long run.
    Note: this research is not yet published in academic, peer-reviewed literature.
    The debate: welfarism versus abolitionism
    There is an ongoing debate within the animal advocacy movement, between so-called welfarists or moderates on the one side and abolitionists or radicals on the other side. The welfarist camp aims for welfare improvements of farm animals. Stronger animal welfare laws are needed to reduce animal suffering. The abolitionists on the other hand, want to abolish the property status of animals. This means abolishing the exploitation of animals, eliminating animal farming and adopting an animal-free, vegan diet.
    The abolitionists are worried that the welfarist approach results in complacency, by soothing the conscience of meat eaters. They argue that people who eat meat produced with higher animal welfare standards might believe that eating such animal welfare certified meat is good and no further steps to further reduce farm animal suffering are needed. Those people will not take further steps towards animal welfare because of a believe one already does enough. Complacency could delay reaching the abolitionist’s desired goal, the abolition of the exploitation of animals. Animal welfare regulations are not enough, according to abolitionists, because they do not sufficiently reduce animal suffering. People will continue eating meat that is only slightly better in terms of animal welfare. In the long run, this results in more animal suffering compared to the situation where people adopted animal-free diets sooner.
    In extremis, the welfarist approach could backfire due to people engaging in moral balancing: eating animal welfare certified meat might decrease the likelihood of making animal welfare improving choices again later.
    Welfarists, on the other hand, argue that in the short run demanding abolition is politically or socially unfeasible, that demanding animal welfare improvements is more tractable, and that these welfare reforms can create a momentum for ever increasing public concern of animal welfare, resulting in eventual reduction and abolition of animal farming. According to welfarists, animal welfare reforms are a stepping stone to reduced meat consumption and veganism. Meat consumers will first switch to higher quality, ‘humane’ meat with improved animal welfare standards in production. And after a while, when this switch strengthens their concern for animal welfare and increases their meat expenditures (due to the higher price of animal welfare certified meat), they will reduce their meat consumption and eventually become vegetarian or vegan.
    The stepping-stone model
    Who is right in this debate between abolitionists and welfarists? There is no strong empirical evidence in favor of one side or the other. But recently, economists developed an empirical method that can shed light on this issue: a stepping stone model of harmful social norms (Gulesci e.a., 2021). A social norm is a practice that is dominant in society. In its simplest form, the stepping stone model assumes three stones that represent three social states. The first stone represents the current state where people adopt a harmful social norm or costly practice L (for low value). In the example of food consumption, this state corresponds with the consumption of convention...
    16 min
  • LW - Contra "Strong Coherence" by DragonGod
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra "Strong Coherence", published by DragonGod on March 4, 2023 on LessWrong.
    Polished from my shortform
    See also: Is "Strong Coherence" Anti-Natural?
    Introduction
    Many AI risk failure modes imagine strong coherence/goal directedness (e.g. [expected] utility maximisers).Such strong coherence is not represented in humans (or any other animal), seems unlikely to emerge from deep learning and may be "anti-natural" to general intelligence in our universe.
    I suspect the focus on strongly coherent systems was a mistake that set the field back a bit, and it's not yet fully recovered from that error.I think most of the AI safety work for strongly coherent agents (e.g. decision theory) will end up inapplicable/useless for aligning powerful systems, because powerful systems in the real world are "of an importantly different type".
    Ontological Error?
    I don't think it nails everything, but on a purely ontological level, @Quintin Pope and @TurnTrout's shard theory feels a lot more right to me than e.g. HRAD. HRAD is based on an ontology that seems to me to be mistaken/flawed in important respects.
    The shard theory account of value formation (while lacking) seems much more plausible as an account of how intelligent systems develop values (where values are "contextual influences on decision making") than the immutable terminal goals in strong coherence ontologies. I currently believe that (immutable) terminal goals is just a wrong frame for reasoning about generally intelligent systems in our world (e.g. humans, animals and future powerful AI systems).
    Theoretical Justification and Empirical Investigation Needed
    I'd be interested in more investigation into what environments/objective functions select for coherence and to what degree said selection occurs.And empirical demonstrations of systems that actually become more coherent as they are trained for longer/"scaled up" or otherwise amplified.
    I want advocates of strong coherence to explain why agents operating in rich environments (e.g. animals, humans) or sophisticated ML systems (e.g. foundation models) aren't strongly coherent.And mechanistic interpretability analysis of sophisticated RL agents (e.g. AlphaStar, OpenAI Five [or replications thereof]) to investigate their degree of coherence.
    Conclusions
    Currently, I think strong coherence is unlikely (plausibly "anti-natural") and am unenthusiastic about research agendas and threat models predicated on strong coherence.
    Disclaimer
    The above is all low confidence speculation, and I may well be speaking out of my ass.
    By "strong coherence/goal directedness" I mean something like:
    Informally: a system has immutable terminal goals.
    Semi-formally: a system's decision making is well described as (an approximation) of argmax over actions (or higher level mappings thereof) to maximise the expected value of a single fixed utility function over states.
    You cannot well predict the behaviour/revealed preferences of humans or other animals by the assumption that they have immutable terminal goals or are expected utility maximisers.
    The ontology that intelligent systems in the real world instead have "values" (contextual influences on decision making) seems to explain their observed behaviour (and purported "incoherencies") better.
    Many observed values in humans and other mammals (see) (e.g. fear, play/boredom, friendship/altruism, love, etc.) seem to be values that were instrumental for increasing inclusive genetic fitness (promoting survival, exploration, cooperation and sexual reproduction/survival of progeny respectively). Yet, humans and mammals seem to value these terminally and not because of their instrumental value on inclusive genetic fitness.
    That the instrumentally convergent goals of evolution's fitness criterion manifested as "terminal" values in mammals is IMO strong empiric...
    8 min
  • LW - Why Not Just... Build Weak AI Tools For AI Alignment Research? by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Not Just... Build Weak AI Tools For AI Alignment Research?, published by johnswentworth on March 5, 2023 on LessWrong.
    “Weak” cognitive tools are clearly a thing, and are useful. Google search is a fine example. There are plenty of flavors of “weak AI” which are potentially helpful for alignment research in a similar way to google search.
    In principle, I think there’s room for reasonably-large boosts to alignment research from such tools. Alas, the very large majority of people who I hear intend to build such tools do not have the right skills/background to do so (at least not for the high-value versions of the tools). Worse, I expect that most people who aim to build such tools are trying to avoid the sort of work they would need to do to build the relevant skills/background.
    Analogy: A Startup Founder’s Domain Expertise (Or Lack Thereof)
    Imagine a startup building tools meant to help biologists during their day-to-day work in the wetlab. I expect domain expertise to matter a lot here: I would guess that if none of the founders have ample personal experience doing research work in a wetlab, the chance of this startup building an actually-highly-useful wetlab product drops by about an order of magnitude. Our hypothetical startup might still “succeed” some other way, e.g. by pivoting to something else, or by being good at pitching their shitty product to managers who make purchasing decisions without actually using the product, or by building something very marginally useful and pricing it very cheaply. But their chance of building a wetlab product which actually provides a lot of value is pretty slim.
    One might reply: but couldn’t hypothetical founders without domain experience do things to improve their chances? For instance, they could do a bunch of user studies on biologists working in wetlabs, and they could deploy the whole arsenal of UX study techniques intended to distinguish things-users-say-matter from things-which-actually-matter-to-users.
    . and my response is that I was already assuming our hypothetical founders do that sort of thing. If the founders don’t have much domain experience themselves, and don’t do basic things like lots of user studies, then I’d guess their chance of building an actually-high-value wetlab product drops by two or three orders of magnitude, not just one order of magnitude. At that point it’s entirely plausible that we’d have to go through thousands of times more startups to find one that succeeded at building a high-value product.
    How is this analogous to plans to build AI tools for alignment research?
    So we want to build products (specifically AI products) to boost alignment research. The products need to help solve the hard parts of aligning AI, not just easy things where we can clearly see what’s going on and iterate on it, not just problems which are readily legible or conceptually straightforward. Think problems like e.g. sharp left turn, deception, getting what we measure, or at a deeper level the problem of fully updated deference, the pointers problem, value drift under self-modification, or ontology identification. And the tools need to help align strong AI; the sort of hacky tricks which fall apart under a few bits of optimization pressure are basically irrelevant at that point. (Otherwise the relevant conversation to have is not about how the tools will be useful, but about how whatever thing the tools are building will be useful.)
    The problem for most people who aim to work on AI tools for alignment research is that they have approximately-zero experience working on those sorts of problems. Indeed, as far as I can tell, people usually turn to tool-building as a way to avoid working on the hard problems.
    I expect failure modes here to mostly look like solving the wrong problems, i.e. not actually addressing bottlene...
    12 min
  • AF - Why Not Just... Build Weak AI Tools For AI Alignment Research? by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Not Just... Build Weak AI Tools For AI Alignment Research?, published by johnswentworth on March 5, 2023 on The AI Alignment Forum.
    “Weak” cognitive tools are clearly a thing, and are useful. Google search is a fine example. There are plenty of flavors of “weak AI” which are potentially helpful for alignment research in a similar way to google search.
    In principle, I think there’s room for reasonably-large boosts to alignment research from such tools. Alas, the very large majority of people who I hear intend to build such tools do not have the right skills/background to do so (at least not for the high-value versions of the tools). Worse, I expect that most people who aim to build such tools are trying to avoid the sort of work they would need to do to build the relevant skills/background.
    Analogy: A Startup Founder’s Domain Expertise (Or Lack Thereof)
    Imagine a startup building tools meant to help biologists during their day-to-day work in the wetlab. I expect domain expertise to matter a lot here: I would guess that if none of the founders have ample personal experience doing research work in a wetlab, the chance of this startup building an actually-highly-useful wetlab product drops by about an order of magnitude. Our hypothetical startup might still “succeed” some other way, e.g. by pivoting to something else, or by being good at pitching their shitty product to managers who make purchasing decisions without actually using the product, or by building something very marginally useful and pricing it very cheaply. But their chance of building a wetlab product which actually provides a lot of value is pretty slim.
    One might reply: but couldn’t hypothetical founders without domain experience do things to improve their chances? For instance, they could do a bunch of user studies on biologists working in wetlabs, and they could deploy the whole arsenal of UX study techniques intended to distinguish things-users-say-matter from things-which-actually-matter-to-users.
    . and my response is that I was already assuming our hypothetical founders do that sort of thing. If the founders don’t have much domain experience themselves, and don’t do basic things like lots of user studies, then I’d guess their chance of building an actually-high-value wetlab product drops by two or three orders of magnitude, not just one order of magnitude. At that point it’s entirely plausible that we’d have to go through thousands of times more startups to find one that succeeded at building a high-value product.
    How is this analogous to plans to build AI tools for alignment research?
    So we want to build products (specifically AI products) to boost alignment research. The products need to help solve the hard parts of aligning AI, not just easy things where we can clearly see what’s going on and iterate on it, not just problems which are readily legible or conceptually straightforward. Think problems like e.g. sharp left turn, deception, getting what we measure, or at a deeper level the problem of fully updated deference, the pointers problem, value drift under self-modification, or ontology identification. And the tools need to help align strong AI; the sort of hacky tricks which fall apart under a few bits of optimization pressure are basically irrelevant at that point. (Otherwise the relevant conversation to have is not about how the tools will be useful, but about how whatever thing the tools are building will be useful.)
    The problem for most people who aim to work on AI tools for alignment research is that they have approximately-zero experience working on those sorts of problems. Indeed, as far as I can tell, people usually turn to tool-building as a way to avoid working on the hard problems.
    I expect failure modes here to mostly look like solving the wrong problems, i.e. not actually addres...
    12 min