The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Lessons learned from Tyve, an effective giving startup by Clifford
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lessons learned from Tyve, an effective giving startup, published by Clifford on December 18, 2022 on The Effective Altruism Forum.
    This is a Draft Amnesty Day draft. That means it’s not polished, it’s probably not up to my standards, the ideas are not thought out, and I haven’t checked everything. I was explicitly encouraged to post something unfinished! Commenting and feedback guidelines: I’m going with the default — please be nice. But constructive feedback is appreciated; please let me know what you think is wrong. I probably have a lot more thoughts I haven’t written down so feel free to ask me questions in the comments.
    In 2019 I tried building a startup to promote effective giving in workplaces. It didn’t work out as a for-profit but looks somewhat promising as a non-profit and has facilitated £630k in donations (around £160k of this went to Founders Pledge recommended charities). In this post, I share the story and my lessons learned.
    Since writing this draft, I’m excited that a new CEO with vastly more product experience than me has taken on Tyve as a nonprofit.
    HT to Sebastien who I met at EAG 2020 and suggested I write up a post-mortem and to Lizka for encouraging me to post as Draft Amnesty Day 2 years later.
    Main learnings:
    Fall in love with the problem, not the solution
    I’d prefer to do similar projects as nonprofits in the future
    Probably don’t try to solve a lack of effective giving with technology
    Fundraising at workplaces can work decently well but I’m not sure EA/tech helps
    Don’t confuse “this should exist” with “people want this to exist”
    Ignore all the above advice
    Brief history
    I started Tyve (like Tithe with a funky spelling) after working at Founders Pledge in September 2019.
    The idea was to encourage employees to give a % of their income to charities and we’d encourage them to give effectively.
    My startup pitch was as follows:
    millennial employees care about purpose and employers need to find better ways of delivering this
    one way they could do this would be by supporting people in giving to causes they care about
    there’s a nifty system in the UK called payroll giving where the employer can play a helpful role in enabling employees to donate pre-tax to charity through their payroll
    payroll giving appeared to be a potentially untapped market. Only 2% of people in the UK give through payroll and apparently this is 30% in the US
    the existing players all had terrible UX and we were going to fix it and create a modern experience (like challenger banks had done to high street banks)
    we were also going to use the cost-effective estimates of our recommended charities to quantify the good they were doing, making it more satisfying
    I now feel pretty iffy about most of the assumptions and the argument - I’d like to think this is based on what I learnt but I also think I could have realised at the time that some of this argument is pretty weak.
    I was relatively sceptical about donation apps but I thought I might have found a problem (small startups have employees who want to feel like they’re at a “good” company but doing CSR at small startups is an effort). I thought a cheap alternative would be to provide a spruced-up version of payroll giving.
    I built an MVP with my cofounder Ben O (a great software developer who I met at EAG Oxford 2016 with the explicit intention to find cofounders - kudos to that conference).
    I got 5 companies whose founders I knew well signed up and paying for the beta. We were pretty excited as people were giving unusual high amounts (thanks to anchoring people on % and targeting fairly well-paid startup employees) and about 50% was going to charities we’d recommended (either based on GiveWell or Founders Pledge).
    I quit my job and then raised £150k SEIS from well-known angel investors in the UK.
    It was a buzz raising this money ...
    10 min
  • EA - Be less trusting of intuitive arguments about social phenomena by Nathan Barnard
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Be less trusting of intuitive arguments about social phenomena, published by Nathan Barnard on December 18, 2022 on The Effective Altruism Forum.
    note: I think this applies much less or even not all in domains where you’re getting tight feedback on your models and have to take actions based on them which you’re then evaluated on.
    I think there’s a trend in the effective altruist and rationality communities to be quite trusting of arguments about how social phenomena work that have theoretical models that are intuitively appealing and have anecdotal evidence or non-systematic observational evidence to support them. The sorts of things I’m thinking about are:
    The evaporative cooling model of communities
    My friend’s argument that community builders shouldn't spend [edit: most] of their time talking to people they consider less sharp than them to people less sharp than them because it’ll harm their epistemic
    Current EA community is selecting for uncritical people
    Asking people explicitly if they’re altruistic will just select for people who are good lairs (person doing selections for admittance to an EA thing)
    The toxoplasma of rage
    Max Tegmark’s model of nuclear war
    John Wentworth’s post on takeoff speeds
    I think this is a really bad epistemology for thinking about social phenomena.
    Here are some examples of arguments I could make that we know are wrong but seem reasonable based on arguments some people find intuitive and observational evidence:
    Having a minimum wage will increase unemployment rates. Employers hire workers up until the point that the marginal revenue generated by each worker equals the marginal cost of hiring workers. If the wage workers have to be paid goes up then unemployment will go up because marginal productivity is diminishing in the number of workers.
    Increasing interest rates will increase inflation. Firms set their prices as a cost plus a markup and so if their costs increase because the price of loans goes up then firms will increase prices which means that inflation goes up. My friend works as a handyman and he charges £150 for a day of work plus the price of materials. If the price of materials went up he’d charge more
    Letting people emigrate to rich countries from poor countries will increase crime in rich countries. The immigrants who are most likely to leave their home countries are those who have the least social ties and the worst employment outlooks in their home countries. This selects people who are more likely to be criminals because criminals are likely to have bad job opportunities in their home countries and weak ties to their families. If we try and filter out criminals we end up selecting smart criminals who are good at hiding their misdeeds. If you look at areas with high crime rates they often have large foreign immigrant populations. [Edit - most people wouldn't find this selection argument intuitive but I thought it was worth including because of how common selection based arguments are in the EA and rationality communities.
    I'm also not taking aim at arguments that are intuitively obvious rather arguments that those making find intuitively appeal, even if they're counterintuitive in some way. i.e some people think that adverse selection is a common and powerful force even though adverse selection is a counter-intutive concept.]
    Cash transfers increase poverty, or at least are unlikely to reduce more than in-kind transfers or job training. We that people in low-income countries often spend a large fraction of their incomes on tobacco and alcohol products. By giving these people cash they have more money to spend on tobacco and alcohol meaning they’re more likely to suffer from addiction problems that keep them in poverty. We also know that poverty selects people who make poor financial decisions, so giving people cash give...
    7 min
  • AF - Take 11: "Aligning language models" should be weirder. by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 11: "Aligning language models" should be weirder., published by Charlie Steiner on December 18, 2022 on The AI Alignment Forum.
    As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days. I have now procrastinated enough that I probably have enough hot takes.
    People often talk about aligning language models, either to promote it, or to pooh-pooh it. I'm here to do both.
    Sometimes, aligning language models just means trying to get a present-day model not to say bad outputs that would embarrass your organization. There is a cottage industry of papers on arxiv doing slightly different variants of RLHF against bad behavior, measuring slightly different endpoints. These people deserve their light mockery for diluting the keyword "alignment."
    The good meaning of aligning language models is to use "get language models to not say bad things" as a toy problem to teach us new, interesting skills that we can apply to future powerful AI. For example, you could see the recent paper "Discovering Latent Knowledge in Language Models Without Supervision" as using "get language models to not lie" as a toy problem to teach us something new and interesting about interpretability. Aligning language models with an eye towards the future doesn't have to just be interpretability research, either, it can be anything that builds skills that the authors expect will be useful for aligning future AI, like self-reflection as explored in Constitutional AI.
    If you're brainstorming ideas for research aligning language models, I encourage you to think about connections between current language models and future AI that navigates the real world. In particular, connections between potential alignment strategies for future AIs and situations that language models can be studied in.
    Here's an example: Constitutional AI uses a model to give feedback on itself, which is incorporated into RL fine-tuning. But we expect future AI that navigates the real world to not merely be prompted to self-reflect as part of the training process, but to self-reflect during deployment - an AI that is acting in the real world will have to consider actions that affect its own hardware and software. We could study this phenomenon using a language model (or language-model-based-agent) by giving it access to outputs that affect itself in a more direct way than adding to an RL signal, and trying to make progress on getting a language model to behave well under those conditions.
    Doing this sounds weird even to me. That's fine. I want the research area of aligning language models to look a lot weirder.
    Not to say that normal-sounding papers can't be useful. There's a lot of room to improve the human feedback in RLHF by leveraging a richer model of the human, for example, and this could be pretty useful for making current language models not say bad things. But to do a sufficiently good job at this, you probably have to start thinking about incorporating unsupervised loss terms (even if they provide no benefit for current models), and addressing scenarios where the AI is a better predictor than the human, and other weird things.
    Overall, I'm happy with the research on aligning language models that's been done by safety-aware people. But we're in the normal-seeming infancy of a research direction that should look pretty weird.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Sir Gavin and the green sky by Gavin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sir Gavin and the green sky, published by Gavin on December 17, 2022 on The Effective Altruism Forum.
    This is a Draft Amnesty Day draft. That means it’s not polished, it’s probably not up to my standards, the ideas are not thought out, and I haven’t checked everything. I was explicitly encouraged to post something unfinished! Commenting and feedback guidelines: I’m going with the default — please be nice. But constructive feedback is appreciated; please let me know what you think is wrong. Feedback on the structure of the argument is also appreciated.
    Epistemic status: crank reportage.
    It suits me that climate change isn't an x-risk. (The movement has trillions of dollars already, and persistently drains talent, attention, and political capital away from actual x-risks.)
    ... But is it one?
    One palaeontologist, Peter Ward, is semi-famous in the field for suggesting a mechanism by which runaway climate change could kill everything: by turning the ocean into a toxic gas factory.
    an increase in carbon dioxide... warms the oceans enough to change circulation patterns. When this happens, sulfur-eating microbes sometimes thrive. These bacteria produce hydrogen sulfide, which, in sufficient quantities and under certain conditions, outgasses into the air, shreds the ozone layer, and poisons other living things. The warming also causes methane ice under the seas to melt and, well, burp, adding to the nasty mix.
    As usual in Zoomed Out Sciences like epidemiology and climatology, the model stops short at the inevitable massive effort to reverse this process. The modellers prefer to think of humans as the pink fella from these comics: inertly lamenting.
    Ward argues that this (and similar mechanisms) is responsible for all past mass extinctions except the dinosaurs one everyone fixates on.
    This is correct as chemistry (I think), and apriori could happen, and for all I know he's right and it actually has happened before. So I have to give it nonzero probability - and one of the real probabilities, with only a few zeroes in it.
    I can't really evaluate this. There are some hallmarks of crankery in the book - most of the citations of the book are unscientific or pseudoscientific - and (as Halstead and others have long noted) climate is one of the slower and more detectable ways to kill a biosphere.
    But in concert with the usual vague definition of an x-risk (where there's no threshold on the probability), I've been thinking of climate change as a (lesser) xrisk for a while and thought I'd come clean, even though I don't think this changes anyone's decisions.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Why we’re getting the Fidelity Model wrong by Alishaandomeda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why we’re getting the Fidelity Model wrong, published by Alishaandomeda on December 17, 2022 on The Effective Altruism Forum.
    This is a Draft Amnesty Day draft. That means it’s not polished, it’s probably not up to my standards, the ideas are not thought out, and I haven’t checked everything. I was explicitly encouraged to post something unfinished! Commenting and feedback guidelines: I’m going with the default — please be nice. But constructive feedback is appreciated; please let me know what you think is wrong. Feedback on the structure of the argument is also appreciated.
    This is a response to The fidelity model of spreading ideas - EA Forum (effectivealtruism.org), the associated CEA blog post, plus those who have referenced it in good-spirited debate about how EA comms ought to develop in the coming years.
    Epistemic status: As confident as anyone experienced enough in manipulating the whims of social media algorithms for social change can ever be, given that they change semi-regularly. 6+ years of experience using social media to effect social change, including advising politicians on their use of social media, conducting experimental social media for social change projects that have informed national political party strategies, and training upwards of 500+ activists.
    Overview
    This post is an exploration of the Fidelity Model, an approach to the spreading of ideas that is widely referenced within EA spaces.
    While I am broadly supportive of the fidelity model and agree with its central premises, I disagree with one influential interpretation of it: that we shouldn't use social media to communicate key EA ideas.
    This interpretation of the argument is claimed to be central to CEA's social media scepticism, which I argue is misplaced. I am more sympathetic to the mass media scepticism this argument also brings about.
    I explain why having an "official presence" on social media is important using the Fidelity Model. Social media is an important communications model, even if social media itself is low-fidelity.
    The Fidelity Model
    I am taking this from the original post, so as to keep our arguments clear and consistent.
    Definition
    Fidelity The key term in this model is "fidelity." Therefore it will be useful to define this term. By fidelity [the original author has] in mind nothing more than the classic dictionary definition of "adherence to fact or detail" or "accuracy; exactness."
    As an example, imagine I am shooting a movie on an old camera. If the image captured by the camera causes it to seem as though I am wearing a blue shirt when actually my shirt is red, then the image captured by the camera is low fidelity.
    The problem
    With reference to The Telephone Game, the author of the original argument accepts that "EA" ideas and arguments are nuanced and worries that some methods of communication require that the nuance be stripped back in order to be easily communicable.
    When the context gets stripped away, those who receive the ideas leave with something that's similar to effective altruism, but different. Thus, when we hear the EA message repeated back to us, we get sentences like "EA is about earning all the money you can and donating it to GiveWell charities" or "EAs only care about interventions that are supported by randomized controlled trials." To a certain extent we can influence the sentences we get back by being more clever about how we frame our ideas, but it seems unlikely that framing can do all the work.
    How we get to social media scepticism
    Very few people who use social media believe it to be a high-fidelity information exchange space. Platforms like Twitter, with low character counts and a norm towards fast-paced argument threads, are particularly bad at fostering nuanced debate.
    EA ideas are nuanced, and when they aren't covered with that nuance in m...
    11 min
  • EA - A digression about area-specific work in LatAm by Jaime Sevilla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A digression about area-specific work in LatAm, published by Jaime Sevilla on December 17, 2022 on The Effective Altruism Forum.
    Written for draft amnesty day
    Despite the many successes of the Spanish-Speaker community, two Spanish-Speaking organizers I greatly respect have been repeatedly discouraged from pursuing research in their areas of expertise.
    Eg a grantmaker suggested them instead working on the translation of key texts to promote Effective Altruism because it might be easier to balance with their other community-building work, despite having no expertise in translation and there already being a translation project in the community. Their grant request was subsequently put on hold.
    Other members of the international community have echoed the advice in different forms, to the point where the organisers feel upset and disheartened.
    This dynamic challenges the local organisers' expertise and awareness of the situation of their local community. On a meta-level, this (together with my previous impressions of the topic) suggests to me that the community might need to tone down EA branding promotion in support of doing area-specific work.
    They have also been encouraged to move to the main EA hubs to gain legitimacy and experience. While this seems hard to fix, it saddens me that there is so little support for capable people who want to develop their career locally.
    Additionally, I am aware of people outside the Spanish-speaking community who have been encouraged to pursue area-specific projects in LatAm, despite lacking experience in the area or local knowledge. This hinders the intellectual independence of small communities and discourages local efforts. Even though I am glad that this work is being done, this might be a symptom of systematic misprioritization.
    All in all, these are isolated issues and probably better explained by miscommunication and differences in judgement. However, I believe we should have a lower bar for raising critiques and be more transparent about them — this is my targeted contribution to move the culture of EA in that direction.
    There is a deeper conversation to be had about the relationship between the international EA community and EA in LatAm and Low and Middle-Income Countries in general. It is not my place to host that conversation, though I hope that by speaking up and seeing the reaction from the community I can help others feel more confident bringing up the issues that worry them.
    As a takeaway, please do not discourage young professionals from pursuing area-specific work, and seek feedback from local organizers about their community’s needs. I also want to stress that grassroots efforts led by dedicated professionals are disproportionately more likely to succeed.
    Thank you to Agustín Covarrubias, Laura González, Claudette Salinas, Michelle Bruno, Sandra Malagón, Ángela Aristizábal, Pablo Stafforini and Catherine Low for feedback and help editing the post.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - The Rules of Rescue - out now! by Theron
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Rules of Rescue - out now!, published by Theron on December 17, 2022 on The Effective Altruism Forum.
    My book—The Rules of Rescue—is officially out! It’s Open Access, so you can download a PDF for free.
    The book deals with a host of questions that have bothered me for a long time. What costs are we morally required to incur to rescue strangers? Is failing to assist distant people in need morally like letting nearby people drown? When do the needs of the many outweigh the needs of the few?
    I defend a novel picture of the moral reasons and requirements to use time, money, and other resources to help others the most. It’s a non-consequentialist picture according to which there are significant permissions not to spend your life helping, as well as robust constraints against helping when it involves harming, lying, or stealing. Requirements to help are grounded not in the promotion of goodness per se, but in the prevention of serious harm to imperiled individuals, whether near or far.
    I argue that altruistic activities are often constrained by requirements of effectiveness, to direct resources in ways that help more rather than less. I explore effectiveness in the context of what you’re required to do over the course of your whole life. I conclude that even if you’re not morally required to be an effective altruist, you may have to help as much as if you were one.
    Visit the book’s website:/
    Buy a printed copy:
    Buy an ebook (read aloud):
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - How to Convince my Son that Drugs are Bad by concerned dad
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Convince my Son that Drugs are Bad, published by concerned dad on December 17, 2022 on LessWrong.
    Hello.
    My son (16m, henceforth referred to as John) has monologued about this site a few times over the past couple of months, so I figured, based on my brief impression of the community, you might be able to help me with an issue. Given the topical nature here, I am not sure if this is an appropriate type of post to make, however it might be a useful place to make an appeal. Worst case, this gets taken down for incompliance.
    John has always been a little too obsessed with his computer, but things really came to a head when he found this whole subcommunity. For a couple of weeks, I'd regularly notice as he spent hours just sitting in his room scrolling through blog posts and papers and forums. While damaging to health, this doesn't seem unusual among teenagers and I try to let him make his own decisions as I'm sure eventually it will taper out, and that's not my main issue.
    First off: I've noticed some positive changes since he started discussing effective altruism and rationality and such, though I don't know whether to attribute that to this site or just maturing. Thank you all for that. But there are some worrying ideas he seems to have gotten as well, centering around romanticization of drug use, specifically "nootropic"-style with a focus on amphetamines and psychedelics.
    One day he just came up to me and began engaging in a discussion about the merits of doing drugs. I'll give you some approximate quotes so you can understand about how this went:
    "Why the hell is LSD criminalized everywhere? There are NO negative side effects proceeds to compare to prescription drugs"
    "Psychedelics and amphetamines are classes of drugs I plan to do, I've done extensive (3 hr - sigh) research on their side effects and chemical compositions and all seems fine! People take Adderall all the time and I can easily get a script, I've read the entire DSM 5!"
    "What's the difference between you drinking alcohol or coffee and me taking amphetamines and doing LSD? Drugs. are. drugs." "Alcohol isn't synthetic? Well then, what about peyote?"
    "Why not try heroin if the purpose of life is to optimize happiness assuming heroin provides proportionally more even if for a shorter amount of time?" (!)
    "Look, here's this site where someone (a 'gwern' if I recall correctly) did this scientifically! They're fine, I want to do that too!"
    "How will I acquire the drugs? I'll just ... uhhh ... synthesize the LSD myself! Can't be that difficult"
    This discussion re-occurs any time someone brings up recreational or cognitive-enhancing drug use. I'm getting frustrated of explaining how dangerous these ideations are and how a lot of these drugs can permanently damage him, through addiction or brain damage or other negative health effects.
    But he is apparently "a rational being who can make his own decisions". While he's certainly intelligent, he's misguided in this particular direction. How can I persuade him to stop these thoughts?
    Relevant note: our current settlement of the situation is that he's going to wait until he's age of majority then do whatever he wants. This is an outcome I want to avoid, as I fear he will fall into psychosis or addiction. It's possible that these ideas will fade over the next year or so, but I'm looking to accelerate that period.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Effective altruism is worse than traditional philanthropy in the way it excludes the extreme poor in the global south. by Jaime Sevilla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective altruism is worse than traditional philanthropy in the way it excludes the extreme poor in the global south., published by Jaime Sevilla on December 17, 2022 on The Effective Altruism Forum.
    Interesting article from Anthony Kalulu from Uganda.
    It argues that EA recommended charities have very little impact in the average poor in Uganda.
    if you randomly asked one of the people who themselves live in abject poverty, there is no chance that they will mention one of EA’s supported “effective” charities, as having impacted their lives more than the work of traditional global antipoverty agencies. No. That’s out of question.
    Anthony argues too that EA solutions are not persistent
    If you visited a truly impoverished country like Uganda, you will quickly notice that many of the things that effective altruists call “effective” — from mosquito nets, to $100 business grants that are provided to groups of 3 people — are the same short-term, disposable solutions that have not only kept their recipients in abject poverty, but also, they are the very kind of solutions that often disappear the same day their proponents exit.
    And that the solutions implemented do not match the communities needs:
    In my region of Busoga, Uganda’s most impoverished region, we have one [well-funded] international charity which is among those described by the EA movement as being “effective”. That charity is also working with rural poor farmers here, principally on maize.
    But the thing is: every household in our region that depends on maize, lives in chronic extreme poverty, and has lived in chronic poverty for eternity. Neither the effective charity nor the other big antipoverty agencies that came before it, have changed this.
    By contrast, those farmers who are growing crops like sugarcane, no charity or antipoverty agency has ever supported them. But today, every village in our region that you visit, is covered with sugarcane. It is also the same with many other crops (rice, tomatoes, water melon etc) that are at least providing rural farmers with some tangible income.
    The solution advocated by Anthony through the article are grassroots organizations:
    In the name of being “effective”, EA has instead indoctrinated its followers to strictly support a small, select list of charities that have been labelled “most effective” by the movement’s own charity raters like GiveWell, Giving What We Can, The Life You Can Save etc, of which the named charities, right now, are all western.
    But that is the very ingredient that makes traditional philanthropy a sector that keeps the world’s extreme poor on the sidelines. And by consolidating itself as a movement that completely never supports grassroots organizations directly, EA has proved to be more of a blockade to those of us who live in ultra poverty, even more than traditional philanthropy.
    What can we do to help improve this situation?
    Here is a brainstorming of some ideas:
    Seek feedback and listen to local though leaders, and invite them to participate in the global conversation about eradicating poverty
    Conduct third-party surveys of the intended beneficiaries of poverty relief. So far the only instrument of this kind I've seen is GiveDirectly's program.
    If you are a charity working in eg Uganda, try promoting your hiring rounds among your beneficiaries and their surrounding communities
    Lead, together with local EA organizers, incubation programs of grassroot and effective efforts
    Try to identify the best local grassroot efforts to support. This might be tricky because these efforts are usually not scalable, so spending $100k on a study might be pointless when the org only has capacity to absorb $50k. Maybe we could study better the effects of a grassroot support foundation.
    Train and support local EA organizers to run talent identification and nurturin...
    4 min
  • AF - Looking for an alignment tutor by JanBrauner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Looking for an alignment tutor, published by JanBrauner on December 17, 2022 on The AI Alignment Forum.
    Hey, this is me. I’d like to understand AI X-risk better. Is anyone interested in being my “alignment tutor”, for maybe 1 h per week, or 1 h every two weeks? I’m happy to pay.
    Fields I want to understand better:
    Anything related to prosaic AI alignment/existential ML safety
    Failure stories/threat models
    Fields I’m not interested in (right now):
    agent foundation
    decision theory
    other very mathsy stuff that’s not related to ML
    My level of understanding:
    I have a decent knowledge of ML/deep learning (I’m in the last year of my PhD)
    I haven’t done the AGI Safety Fundamentals course, but I just skimmed it, and I think I had independently read essentially all the core readings (which means I probably have also read many things not on the curriculum). I’d say I have a relatively deep understanding of a majority (but not all) of the content in this curriculum.
    Similarly for the AGI Safety Fundamentals 201, excluding the tracks
    Example questions I wrestled with recently, and I might have brought up during the tutoring:
    It seems to me that our currently level of outer alignment tools (RLHF + easy augmentation) is enough to solve the outer alignment problem sufficiently well so that humans don’t end up dead or disempowered (conditional on slow takeoff); and then we can solve further outer alignment problem as the come up, with iteration and regulation. So I basically think that the core of the alignment problem, at the moment, is inner alignment + deceptive alignment. What am I missing? (I read Christiano’s “Another Outer Alignment Failure Story”, but I still have this question.)
    I understand that a reward maximiser would wire-head (take control over the reward provision mechanism), but I don’t see why training an RL agent would necessarily end up in a reward-maximising agent? Turntrout’s Reward is Not the Optimisation Target shed some clarity on this, but I definitely have remaining questions.
    Is the failure mode describe in Ajeya’s Without Specific Countermeasures an inner alignment failure, or an outer alignment failure (I think it’s both).
    You don’t need to have very crisps answers to these to be my tutor, but you should probably have at least some good thoughts.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min