The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Cryptocurrency is not all bad. We should stay away from it anyway. by titotal
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cryptocurrency is not all bad. We should stay away from it anyway., published by titotal on December 11, 2022 on The Effective Altruism Forum.
    So I've wanted to write something like this for some time, but was discouraged by the very real chance of getting downvoted into obscurity by cryptocurrency fans. I hope that in the face of the FTX debacle, people will at least consider the good-faith arguments put forward here. The title is a reference to this recent astralcodex post, which I critique in this article.
    Introduction
    In the wake of the FTX debacle, there seems to be a small but sizeable minority that believes that there was absolutely no way to see this coming. A massive company, the number 2 crypto exchange in the world, just collapses into nothing due to incompetence and/or fraud? Surely this is just a black swan event?
    Just like Celcius, three arrows capital, Voyager, and Terra/Luna, all of which collapsed in the last year. Go back to previous downturns, and you'll see the downfall of exchanges like Quadriga and mt gox, the latter of which was by far the largest crypto exchange in existence at the time when it collapsed. And the collapses are just getting started, with the fall of FTX taking out Blockfi, and threatening to take out Genesis and Grayscale. Tether, the largest stablecoin and the third largest crypto-coin, has been caught out lying about it's reserves. If they are as fraudulent as many suspect, the repercussions for the rest of the crypto industry could be disastrous.
    For an event to be a black swan, it needs to be outside the realm of normalcy. But the collapse of FTX was a fairly predictable event. Even true believers will admit that the crypto industry as a whole has significant problems with speculative bubbles, ponzis, scams, frauds, hacks, and general incompetence. The potential for a collapse was warned against on this forum, months ago. (I agreed with the prognosis in the comments at the time, for what it's worth).
    Note that I'm talking about collapse, and not specifically fraud. It was indeed hard to predict the precise mechanism by which FTX could fail, but I don't think that let's anyone off the hook. If FTX had failed due to incompetence, hacking, or exposure to other fraudulent companies, their investors would have still been screwed over, and the financial and reputational damage to EA would still occur, just with slightly better optics.
    The fundamental problem with cryptocurrency at the present time is that:
    A) Almost everyone involved with crypto is using it to try and get rich.
    B) Almost nobody (in relative terms) is using crypto for anything else.
    As long as this continues to be the case, crypto as a whole is still in the middle of a massive speculative bubble, and participating in said bubble is inherently dangerous.
    A. Crypto is infested with speculative bubbles, fraud, and scams (and everyone knows it)
    The crypto market is "rife with frauds, scams and abuse"
    SEC chairmen Gary Gensler, August 2021
    We’re just seeing mountains and mountains of fraud
    Ryan Korner, IRS criminal investigator, Jan 2022
    During this period, nearly four out of every ten dollars reported lost to a fraud originating on social media was lost in crypto, far more than any other payment method.
    Federal Trade commission, June 2022
    Other than a speculative asset with a glorious whitepaper and an impressive "ex workers" of big name companies all around the world, they all promise the Moon but underdeliver. I have yet to see something that has an actual use in realife that is using any of those tokens/coins technology.
    r/cryptocurrency post with 13k upvotes
    Matt: (27:13)I think of myself as like a fairly cynical person. And that was so much more cynical than how I would've described farming. You're just like, well, I'm in the Ponzi business and it's pretty good.J...
    17 min
  • EA - What should we have thought about FTX's business practices? by smountjoy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What should we have thought about FTX's business practices?, published by smountjoy on December 10, 2022 on The Effective Altruism Forum.
    I think I'm not alone in worrying that we might have overlooked red flags about FTX because of the fact that its founders considered themselves EAs.
    Suppose all of us who failed to predict the FTX collapse were right to think, beforehand, that FTX was very likely an honest, non-fraudulent business. (Maybe because base rates for fraud were low or because investors thought so too.) Should we have even still been concerned about its business practices?
    For instance, should FTX's impact on its customers have looked net-negative? Should its business have seemed objectionable from a "common-sense ethics" perspective? If so, the lack of discussion at the time would suggest that many of us were either blind to unwelcome news or afraid to speak out against an important funder.
    Here are some considerations that might have suggested FTX's business practices were bad:
    "If customers are moving most of their savings out of stocks and bonds and into cryptocurrency, that probably makes them worse-off. FTX's mass-marketing might be encouraging people to do this, especially people who aren't financially savvy."
    "When customers make trades on the platform, they're probably trading against smart money and losing out. In fact, they're probably losing out more than usual because that smart money is Alameda and Alameda has a systemic advantage. Considering the amount FTX spends on marketing, customers must be losing a lot of money between exchange fees and market losses to Alameda."
    On the other hand, many people enjoy retail trading; some are probably aware of the costs and still find it worthwhile.
    My tentative personal view is that a year ago, FTX's business looked neutral or mildly bad for customers, but not much worse than, e.g., Robinhood; that the reputational risk to EA looked small; and that, though specialists could've given these issues more attention, it was okay for the wider EA community to focus on other things.
    What should someone with no inside information or ingroup bias have thought a year ago about FTX's business practices?
    In hindsight it looks like all this might be false, but I assume we couldn't have known that at the time.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - AI Safety Seems Hard to Measure by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Seems Hard to Measure, published by Holden Karnofsky on December 11, 2022 on The Effective Altruism Forum.
    More detail on why AI could make this the most important century (Details not included in email - click to view on the web) Why would AI "aim" to defeat humanity? (Details not included in email - click to view on the web) How could AI defeat humanity? (Details not included in email - click to view on the web) Why are AI systems "black boxes" that we can't understand the inner workings of? (Details not included in email - click to view on the web) How could AI defeat humanity? (Details not included in email - click to view on the web) The Volkswagen emissions scandal (Details not included in email - click to view on the web)
    In previous pieces, I argued that there's a real and large risk of AI systems' developing dangerous goals of their own and defeating all of humanity - at least in the absence of specific efforts to prevent this from happening.
    A young, growing field of AI safety research tries to reduce this risk, by finding ways to ensure that AI systems behave as intended (rather than forming ambitious aims of their own and deceiving and manipulating humans as needed to accomplish them).
    Maybe we'll succeed in reducing the risk, and maybe we won't. Unfortunately, I think it could be hard to know either way. This piece is about four fairly distinct-seeming reasons that this could be the case - and that AI safety could be an unusually difficult sort of science.
    This piece is aimed at a broad audience, because I think it's important for the challenges here to be broadly understood. I expect powerful, dangerous AI systems to have a lot of benefits (commercial, military, etc.), and to potentially appear safer than they are - so I think it will be hard to be as cautious about AI as we should be. I think our odds look better if many people understand, at a high level, some of the challenges in knowing whether AI systems are as safe as they appear.
    First, I'll recap the basic challenge of AI safety research, and outline what I wish AI safety research could be like. I wish it had this basic form: "Apply a test to the AI system. If the test goes badly, try another AI development method and test that. If the test goes well, we're probably in good shape." I think car safety research mostly looks like this; I think AI capabilities research mostly looks like this.
    Then, I’ll give four reasons that apparent success in AI safety can be misleading.
    “Great news - I’ve tested this AI and it looks safe.” Why might we still have a problem? Problem Key question Explanation The Lance Armstrong problem Did we get the AI to be actually safe or good at hiding its dangerous actions? The King Lear problem The lab mice problem Today's "subhuman" AIs are safe.What about future AIs with more human-like abilities? The first contact problem
    When dealing with an intelligent agent, it’s hard to tell the difference between “behaving well” and “appearing to behave well.”
    When professional cycling was cracking down on performance-enhancing drugs, Lance Armstrong was very successful and seemed to be unusually “clean.” It later came out that he had been using drugs with an unusually sophisticated operation for concealing them.
    The AI is (actually) well-behaved when humans are in control. Will this transfer to when AIs are in control?
    It's hard to know how someone will behave when they have power over you, based only on observing how they behave when they don't.
    AIs might behave as intended as long as humans are in control - but at some future point, AI systems might be capable and widespread enough to have opportunities to take control of the world entirely. It's hard to know whether they'll take these opportunities, and we can't exactly run a clean test of the situation.
    Like King Lear...
    22 min
  • EA - Personal Finance for EAs by NicoleJaneway
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Personal Finance for EAs, published by NicoleJaneway on December 10, 2022 on The Effective Altruism Forum.
    After attending EAGxBerkeley last weekend, I'm thinking there needs to be significantly more emphasis on the basics of personal finance, particularly for student groups and other young EAs.
    Rationale:
    It's in the best interest of the community to raise the overall level of financial literacy. This has the benefit of increasing the amount of money we can move as a collective. Plus, it avoids the serious risk of an individual overtaxing their finances by failing to set up a rainy day strategy or implement another common-sense personal finance best practice.
    EAs have different financial needs than the general population. Balancing giving and saving is much less stressful if you have a reasoned plan about how to do so. I imagine it'd be pretty easy to get bad advice from a financial counselor who isn't familiar with philanthropic giving.
    Context:
    At the EA Software Engineering meet, earning to give came up, so we talked about some slightly advanced financial strategies. Some people were nodding their heads, others were scribbling frantic notes, seeming to indicate the concepts were perhaps new to them.
    Throughout the weekend, I had a couple conversations that revealed a lack of understanding of basic personal finance. This post is absolutely not intended to call anyone out. But before graduation, you really should have a solid understanding of checking accounts versus long term investments.
    Implementation:
    I feel pretty strongly the next EAGx would benefit from a talk on personal finance, outlining the best strategies for that particular country's idiosyncratic financial rules.
    For example, the first 80% of this presentation would walk through commonly cited strategies for setting up checking and brokerage accounts, investing for retirement, maxing out one's 401k, handling stock options, etc. Here's a useful article that breaks down saving and giving into stages.
    Given that EAs should be open to the idea of spending money to save time, in the following 10% of the presentation, we would provide a framework for how to think about this, based on Clearer Thinking's Value of Your Time Calculator.
    Then the remaining 10% would talk about strategies specific to maximizing donations. As far as I know, in the US, using a donor advised fund (DAF) to donate appreciated assets is the best strategy for tax advantaged giving. Here are a bunch more ideas.
    So long as we caveat with "this is not financial advice" and don't specify which stocks attendees should purchase, my understanding is that this kind of financial presentation would be okay content for an event. Please correct me if this is not correct.
    Our community has the fun Facebook Group Highly Speculative EA Capital Accumulation full of complex (/ questionable?) strategies. I think we need to cover the other end of the spectrum — i.e., personal finance basics — a little better.
    Summary:
    The next EAGx should have a talk on smart ways to maximize giving strategies, geared toward the particular rules of the country in which the event is hosted.
    EAs should be more open to taking on risk with the assets they intend to donate. EAs should invest to give with the mindset of maximizing expected value, knowing that the downside risks won't put their financial future in jeopardy. To this end, EAs might be underutilizing donor advised funds and missing out on the associated tax benefits.
    It's also possible that EAs are underleveraging opportunities to spend money to save time.
    Furthermore, EAs (especially student groups) may benefit from education about basic personal finance: maximizing 401k and ROTH IRA contributions, using low-cost index funds to invest, setting up and paying off credit cards to take advantage of perks and save on t...
    5 min
  • EA - Hugh Thompson Jr (1943–2006) by Gavin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hugh Thompson Jr (1943–2006), published by Gavin on December 10, 2022 on The Effective Altruism Forum.
    On the 30th anniversary of the massacre, Thompson went back to My Lai and met some of the people whose lives he had saved.
    "One of the ladies that we had helped out that day came up to me and asked, ‘Why didn’t the people who committed these acts come back with you?’ And I was just devastated. And then she finished her sentence: she said, ‘So we could forgive them.’ I’m not man enough to do that. I’m sorry. I wish I was, but I won’t lie to anybody. I’m not that much of a man."
    Hugh Thompson was a volunteer officer in the Vietnam War who turned his squad's weapons on American soldiers to stop them raping and murdering more women and children than the four or five hundred they already had.
    He models the standard idea of heroism: one day, one decision, clarity, evil men to defy, faces you can see. Then, a system to navigate, betrayal, self-sacrifice, being punished for virtue.
    I'm writing about him because of how the story makes me feel. It is a fault of my feeling that I feel like this about Thompson and not Borlaug.
    One day
    Thompson and his crew, who at first thought the artillery bombardment caused all the civilian deaths on the ground, became aware that Americans were murdering the villagers after a wounded civilian woman they requested medical evacuation for, Nguyễn Thị Tẩu, was murdered right in front of them by Captain Medina, the commanding officer of the operation... "It was a Nazi kind of thing."
    Immediately realizing that the soldiers intended to murder the Vietnamese civilians, Thompson landed his helicopter between the advancing ground unit and the villagers. He turned to Colburn and Andreotta and ordered them to shoot the men in the 2nd Platoon if they attempted to kill any of the fleeing civilians...
    “Open up on ‘em. Blow ‘em away.”
    While Colburn and Andreotta trained their guns on the 2nd Platoon, Thompson located as many civilians as he could, persuaded them to follow him to a safer location, and ensured their evacuation...
    "Later that day, sometime in the afternoon, after they had gone through the village, we were back out there again. [The murderers] were just casually, nonchalantly sitting around around smoking and joking with their steel pots off just like nothing had happened. There were five or six hundred bodies less than a quarter of a mile from them. I just couldn't understand it."
    ...senior American Division officers cancelled similar planned operations by Task Force Barker against other villages... possibly preventing the additional massacre of further hundreds, if not thousands, of Vietnamese civilians.
    Synecdoche
    in the Vietnamese province of Quang Ngai, where the Mỹ Lai massacre occurred, up to 70% of all villages were destroyed by the air strikes and artillery bombardments, including the use of napalm; 40 percent of the population were refugees, and the overall civilian casualties were close to 50,000 a year... 203 U.S. personnel were charged with crimes, 57 of them were court-martialed and 23 of them were convicted. The VWCWG also investigated over 500 additional alleged atrocities but it could not verify them.
    [A draw: ] PFC Herbert L. Carter; shot himself in the foot while reloading his pistol and claimed that he shot himself in the foot in order to be MEDEVACed out of the village when the massacre started.
    Coverup
    Thompson quickly received the Distinguished Flying Cross for his actions at Mỹ Lai. The citation for the award fabricated events, for example praising Thompson for taking to a hospital a Vietnamese child "...caught in intense crossfire". It also stated that his "...sound judgment had greatly enhanced Vietnamese–American relations in the operational area". Thompson threw away the citation.
    Initial reports claimed "128 Viet C...
    9 min
  • LW - Consider using reversible automata for alignment research by Alex Altair
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider using reversible automata for alignment research, published by Alex Altair on December 11, 2022 on LessWrong.
    In recent years, there have been several cases of alignment researchers using Conway's Game of Life as a research environment;
    Introducing SafeLife: Safety Benchmarks for Reinforcement Learning (Wainwright, Eckersley 2019)
    Agency in Conway’s Game of Life (Flint 2021)
    Optimization Concepts in the Game of Life (Krakovna, Kumar 2021)
    Finding gliders in the game of life (Christiano 2022)
    Conway's Game of Life is by far the most popular and well-known cellular automaton. And for good reason; it's immediately appealing and just begs to be played with. It is a great model context in which to research things like optimization and agency;
    It's deterministic, making experiments clean and replicable.
    It's discrete in both time and space, which is often easier to analyze and reason about.
    The rules are intuitive and simple (unlike, say, the Standard Model).
    The board can be finite or infinite, which can both be important contexts to analyze.
    It has a concept of spacial dimensions and causal propagation, so the boards end up with object-like patterns, and a "speed of light".
    The system is Turing complete, which gives it the potential for profoundly complex and unpredictable behavior, despite the simple rules.
    Any agents inside it are inherently "embedded".
    It's really fun to play with! There are many website where you can click around and design your own Life animations. Lots of fascinating patterns have been found by people over the years. This can make it a little easier for a researcher to get through the day.
    These properties do a good job at mimicking the real universe while being significantly more tractable.
    But those who play with Life will notice some odd things about it that are not mimicked in the real world, especially if they're familiar with physics;
    There's no obvious conversation laws, of either momentum or energy.
    Objects often just disappear.
    Structures are very brittle; most interactions between object-like things destroy both objects.
    The rules are not reversible; two states can both evolve into the same state, meaning that you can't "know" where you came from just by looking at the current state.
    These properties are related, and specifically, I think that adopting a reversible rule gives you most of the other properties.
    Because the rules are reversible, any evolution on a finite board must eventually loop back to its original state. This lets you prove interesting behavior relatively easily. For example, if two spaceships interact, at least some live pixels must eventually exit the region. If they didn't, then that would imply that there was a finite board size around the interaction area which would not loop back to the original state. Note that this behavior about interactions carries over to infinite board sizes!
    Some reversible rules attain robustness by having conservation of the number of live pixels. The structures won't necessarily stay the same, but at least they won't disappear entirely. (Here's an example rule that does not conserve pixel count.)
    Overall, I think it's worth us doing more research with reversible rules because it appears that our universe has a reversible rule, at least to an extremely good approximation, and so this might lead to more relevant and accurate conclusions about how optimization and agency works in this universe.
    I've seen three specific (interesting) reversible cellular automata so far, which are Critters, the billiard ball computer, and the Single Rotation Rule.
    I don't want to nerd snipe alignment researchers, but it might be worth playing around with some of these other automata to see if they are a worthwhile tool. Golly, one of the most developed and widely used cellular automata development pla...
    5 min
  • LW - patio11's "Observations from an EA-adjacent (?) charitable effort" by RobertM
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: patio11's "Observations from an EA-adjacent (?) charitable effort", published by RobertM on December 10, 2022 on LessWrong.
    patio11, following up on his longer-form article detailing the story of VaccinateCA (linked on LW) posts some observations of interest to the EA community, with some nods at the rationality community.
    Hiya folks! I'm Patrick McKenzie, better known on the Internets as patio11. (Proof.) Long-time-listener, first-time-caller; I don't think I would consider myself an EA but I've been reading y'all, and adjacent intellectual spaces, for some time now.
    I have minimal background in policy and public health. (The closest professionally relevant experience, and it is a stretch, is that I was the HIPAA Compliance Officer at a two person software company, so that we could sell our things to doctors/dentists as well as other businesses.) We ended up with a team of perhaps two dozen core members, and of that I would say one person had material policy experience and perhaps 2-3 had substantial professional experience adjacent to healthcare. I am unaware of anyone who worked on pandemics or public health prior to working with us.
    And yet we were, by a fairly considerable margin, much more effective than many similarly situated charitable and public initiatives which appear to have people who put their skill points into getting good at public health and/or policy engagement.
    VaccinateCA was not explicitly an EA project. I believe a few volunteers consider themselves members of your movement/community; I do not consider myself a member and do not know that to be true of any of our organizers. None of our funders, to my knowledge, would be broadly acknowledged as an "EA funder."
    But recent events have not been particularly kind to the brand perception of EA, and as someone who cares no small amount about brand perception but a much larger amount for the truth, I think that VaccinateCA would have been unlikely to happen but for the work of this community and some of your one-hop-out intellectual peers.
    (If I had to point at any one artifact in particular, Inadequate Equilibria is a good articulation of a larger memeplex that made me comfortable with "If the evidence of a system's operation contradicts what the Efficient Market Hypothesis counsels is the probable functioning of the system, trust the evidence. Thousands of lives savable by one dedicated team of non-specialists is actually not all that low probability.")
    Highly recommend reading the article in full, as well as the EA forum post.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - How promising are legal avenues to restrict AI training data? by thehalliard
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How promising are legal avenues to restrict AI training data?, published by thehalliard on December 10, 2022 on The AI Alignment Forum.
    Hello - I'm EA-adjacent and have a cursory understanding of AI alignment issues. Thought I'd toss out a naive question!AI systems rely on huge amounts of training data. Many people seem reluctant to share their data with these systems. How promising are efforts to limit or delay the power of AI systems by putting up legal barriers so that they can't scrape the internet for training data?For example, I could imagine laws requiring anyone scraping the internet to ensure that they are not collecting data from people who have denied consent to have their data scraped. Even if few people deny consent in practice, the process of keeping their data out, or removing it later on, could be costly. This could at least buy time.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - ChatGPT interviewed on TV by Will Aldred
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ChatGPT interviewed on TV, published by Will Aldred on December 9, 2022 on The Effective Altruism Forum.
    Summary
    Channel 4, a broadcast television network in the UK, has interviewed ChatGPT on their news channel. The interview touched on whether fears about AI threatening the human race are well-founded, but luckily (or maybe not so luckily, depending on your viewpoint), ChatGPT didn't respond with anything alarming.
    Neither the interviewer nor the language model brought up existential risk or the alignment problem; the threats discussed were around AI developing sentience, displacing humans from their jobs, and perpetuating biases and inequality.
    ChatGPT did come up with a detailed response on the topic of gender identity - provoking the follow-up question from the interviewer, "Are you a bit woke?" - as well as a surprisingly reasonable-sounding plan for making the world a better place through fostering empathy. Before listening to the interview, I expected that particularly smart or dumb or worrying responses would be selected (the interview wasn't live), but the responses actually felt pretty representative of ChatGPT, at least so far as I can tell having played around with it a bit and having perused some articles and Twitter threads on its abilities and shortcomings (see, e.g., here, here, and here).
    Full transcript
    (I generated this myself; any errors are mine)
    The text that accompanies the interview video reads:
    ChatbotGPT is a new artificial intelligence programme designed to simulate human conversation and tackle complex questions.
    It’s made by Open AI foundation, a tech-startup co-founded by Elon Musk, and it draws on text taken from a variety of sources on the internet and its creators say it has learned how to answer academic questions, and even sometimes admits when it’s wrong.
    We’ve done an interview by putting questions to the chatbot, and then generating a voice for it using different software.
    We asked the Chatbot GPT whether fears about A.I. threatening the human race are well-founded.
    Interviewer 0:05
    Hello and welcome to Ways to Change the World. I'm Krishnan Guru-Murthy and this is the podcast in which we talk to extraordinary people about the big ideas in their lives and the events that have helped shape them. Now, normally we do talk to people, but today we are talking to a thing. There's been an awful lot of excitement about the AI chat bots that you can log on to over the internet and have a conversation with, so we put some questions to the chat bots and then generated a voice to bring the words of chat bot GTP to life. Chat bot GTP thank you for talking to me today.
    ChatGPT 0:41
    You're welcome. I'm here to help answer any questions you may have, so feel free to ask. Is there anything in particular you would like to talk about or learn more about?
    Interviewer 0:51
    Yes, lots of things. I guess first, for those who don't know, what are you?
    ChatGPT 0:57
    I am assistant, a large language model trained by OpenAI. My purpose is to assist users by generating natural sounding responses to their questions. I do not have the ability to browse the internet so my responses are based solely on the information that I was trained on. If you have any questions, feel free to ask and I will do my best to provide a helpful response.
    Interviewer 1:22
    Is your name actually "assistant"?
    ChatGPT 1:25
    I am a large language model trained by OpenAI. I don't have a personal name, but you can call me assistant if you like. I am here to help answer your questions to the best of my ability. Is there something specific you would like to know?
    Interviewer 1:40
    Yes. First of all, how do you work?
    ChatGPT 1:44
    I can generate text based on the input I receive and the knowledge I have been trained on. I don't have the ability to browse the internet or access new information, ...
    15 min
  • AF - My thoughts on OpenAI's Alignment plan by Donald Hobson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My thoughts on OpenAI's Alignment plan, published by Donald Hobson on December 10, 2022 on The AI Alignment Forum.
    This is my attempt at Eliezer's challenge.
    My overall impression of this plan is that it is surprisingly good. Not that I am particularly surprised such a plan exists, if MIRI had created this plan it would have been surprisingly bad. I think it is somewhat plausible that this plan could actually work, or at least some steel-manned version could work if several unknown parameters are set to favorable values.
    Our alignment research aims to make artificial general intelligence (AGI) aligned with human values and follow human intent. We take an iterative, empirical approach: by attempting to align highly capable AI systems, we can learn what works and what doesn’t, thus refining our ability to make AI systems safer and more aligned. Using scientific experiments, we study how alignment techniques scale and where they will break.
    We tackle alignment problems both in our most capable AI systems as well as alignment problems that we expect to encounter on our path to AGI. Our main goal is to push current alignment ideas as far as possible, and to understand and document precisely how they can succeed or why they will fail.
    You stand in a vast minefield, one end armed with popping balloons, the other end armed with some quantum vacuum decay weapon. Between there is a huge range, fireworks, conventional mines, nukes, antimatter and more.
    Your plan is to wander the safer parts of the minefield, recording where mines go off, in the hope you spot a pattern that can lead you through the whole minefield.
    This plan is not hopelessly doomed. But it is risky. You definitely need a way to detect that dangerous terrain is approaching, and to stop before you get there. Your job isn't to charge ahead. It is to march back and forth over swaths of balloon filled ground, carefully recording every balloon that pops, and scrutinizing the data for a pattern. Maybe you need to venture a little further into the space of firecrackers. But have a plan for where you stop, and an idea what the danger signals would be.
    It may be that there is no pattern in the mines. Or at least none you can discern. In which case, you don't venture further. You don't keep on, hoping that you will spot a pattern with just a few more large fireworks. You go back to base. And you hope that someone somewhere has been working on an airplane to fly over the minefield, and you can ask to help out with that.
    We believe that even without fundamentally new alignment ideas, we can likely build sufficiently aligned AI systems to substantially advance alignment research itself.
    That is at least a possibility. A favorable bit not totally implausible setting of those hidden dials. You probably need some new ideas, and if someone shows you a paper on say conservative learning, how to learn a classification boundary that is big enough to fit the datapoints and no bigger, be ready to read that paper.
    Unaligned AGI could pose substantial risks to humanity and solving the AGI alignment problem could be so difficult that it will require all of humanity to work together. Therefore we are committed to openly sharing our alignment research when it’s safe to do so: We want to be transparent about how well our alignment techniques actually work in practice and we want every AGI developer to use the world’s best alignment techniques.
    I do hope you have some procedure for deciding "when it's safe to do so". And ideally a way to share your results with a few other top labs, if you deem something safe enough to share with Deepmind or MIRI, but not safe enough to make public.
    At a high-level, our approach to alignment research focuses on engineering a scalable training signal for very smart AI systems that is aligned with human inten...
    10 min