The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Put Dirty Dishes in the Dishwasher by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Put Dirty Dishes in the Dishwasher, published by jefftk on September 10, 2022 on LessWrong.
    "It's not worth using the dishwasher: by the time you've gotten your dishes clean enough to put in, you might as well just finish washing them yourself."
    People have a wide range of ways they deal with dishes before putting them in the dishwasher:
    Rinse well. The dishes should look clean before they go in.
    Rinse lightly. It's fine if the dishes don't look clean, but they should be close.
    Scrape well. There shouldn't be anything on the dishes that further scraping would remove.
    Scrape lightly. Get most of the food off, but it's fine if some stuff remains.
    Do nothing. Just dump the whole thing in there, food and all. [Not recommended.]
    Earlier options are more work, give you cleaner dishes, and mean you need to clean the dishwasher less often. Later are the reverse.
    In my experience, people tend to be too far towards the early end for their particular dishwasher and time-cleanliness preferences. This makes sense: if you're too far toward the later end you notice and adjust, while in the other direction there's no signal that you could be doing better. Try a later option; you could be putting effort into pre-cleaning for minimal impact on your dishes. It's a cheap experiment to run: there's a good chance you're happy, and if not you can just re-run this load and go back to more thorough pre-cleaning.
    Personally, I'm pretty happy with #4. It's very little work, and usually all the dishes come out clean. When one is still dirty, I just include it in the next load after scraping a bit. If dishes come out unusually dirty it's generally because something was loaded wrong (too tall on the bottom rack, poking through the middle rack) and prevented the middle arm from spinning.
    Comment via: facebook
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Could it be a (bad) lock-in to replace factory farming with alternative protein? by Fai
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Could it be a (bad) lock-in to replace factory farming with alternative protein?, published by Fai on September 10, 2022 on The Effective Altruism Forum.
    Epistemic status
    I have recently been encouraged to write more posts, so I decided to test my fast writing this time with a topic that sat in my mind for very long, and just post it quickly. I haven’t discussed it with anyone, nor checked if the topic has been discussed already, nor done any research. I literally just wrote the whole thing out without doing any new research (or even just search), except when I went back to insert links to some claims. It took me 5 hours to write this.
    TL;DR
    Letting technological advancement in alternative protein and economics do most/all of the job of replacing factory farming could be bad, especially from the longtermist perspective. We likely only have one chance to eliminate factory farming (for food) for moral reasons, and we might lose a lot by losing this chance.
    Why lock-in?
    The lock-in is this: Unless there are other planets with other intelligent beings doing factory farming, or we somehow restart it after eliminating it, we likely only have one chance to eliminate factory farming (for food) for moral reasons. The moment plant-based alternative/cultivated meat (PB/CM) replaces virtually all factory farming for food (btw, I doubt this will be certain, see this post), we lose the chance to do so for moral reasons. Yes, after that we can still change our laws and say that we “ban” factory farming for food in a time where there is virtually none, but I argue that even the motivation behind making such legal bans matters. This leads to the second section.
    Why might the lock-in be bad?
    Let’s first talk about using laws to ban factory farming. We have, coarsely speaking, two options:
    Ban it when we still have factory farming (btw, please consider supporting the federal ballot initiative to abolish factory farming in Switzerland, September 25)
    Ban it after we virtually eliminated it for non-morally relevant reasons (and excuse me for emphasizing again, I don’t think it’s 100% guaranteed).
    The moral character (and therefore the education that is based on it) we show in the two scenarios will be drastically different - It seems much better if we are so morally determined that we simply make a law to ban factory farming, than we eliminate it for economic reasons and then say we ban it. Some of my more particular worries include: If we ban one form of animal exploitation but not all, it might mislead people to think that those that are still legal are morally acceptable. I also worry that using laws to capture our abolition of moral catastrophes after they become economically inviable, can create a false sense of progress - making us feel overly confident about our moral progress and moral capacity, and therefore makes us not informed enough to have good future progress.
    Another scenario is we simply replace factory farming for food with technologies, without ever banning it. There was a historical example that is very similar: Animal advocates often use the example of automobiles replacing horses being exploited for transportation to explain the importance of technologies in our moral progress. But the same example can also be evidence that the (near) elimination of a moral catastrophe using technological advancement can be bad in the long-term. Horse riding, and the riding of other animals, still exist in different forms of entertainment, such as tourism, sports, and gambling. Yes, they cause much less direct suffering than the use of animals as transports, but the value they communicate is still very bad, and virtually the same - an animal can be caused to exist, raised, and exploited for human use however we like.
    Also, besides actively communicating speciesist values, the way we ...
    14 min
  • EA - Puzzles for Everyone by Richard Y Chappell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Puzzles for Everyone, published by Richard Y Chappell on September 10, 2022 on The Effective Altruism Forum.
    Some of the deepest puzzles in ethics concern how to coherently extend ordinary beneficence and decision theory to extreme cases. The notorious puzzles of population ethics, for example, ask us how to trade off quantity and quality of life, and how we should value future generations. Beckstead & Thomas discuss a paradox for tiny probabilities and enormous values, asking how we should take risk and uncertainty into account. Infinite ethics raises problems for both axiology and decision theory: it may be unclear how to rank different infinite outcomes, and it’s hard to avoid the “fanatical” result that the tiniest chance of infinite value swamps all finite considerations (unless one embraces alternative commitments that may be even more counterintuitive).
    Puzzles galore! But these puzzles share a strange feature, namely, that people often mistakenly believe them to be problems specifically for utilitarianism.
    [Image caption: "Fear not: there’s enough for everyone!"]
    Their error, of course, is that beneficence and decision theory are essential components of any complete moral theory. (As even Rawls acknowledged, “All ethical doctrines worth our attention take consequences into account in judging rightness. One which did not would simply be irrational, crazy.” Rossian pluralism explicitly acknowledges a prima facie duty of beneficence that must be weighed against our other—more distinctively deontological—prima facie duties, and will determine what ought to be done if those others are not applicable to the situation at hand. And obviously any account relevant to fallible human beings needs to address how we should respond to uncertainty about our empirical circumstances and future prospects.)
    Why, then, would anyone ever think that these puzzles were limited to utilitarianism? One hypothesis is that only utilitarianism is sufficiently clear and systematic to actually attempt an answer to these questions. Other theories too often remain silent and non-committal. Being incomplete in this way is surely not an advantage of those theories, unless there’s reason to think that a better answer will eventually be fleshed out. But what makes these questions such deep puzzles is precisely that we know that no wholly satisfying answer is possible. It’s a “pick your poison” situation. And there’s nothing clever about mocking utilitarians for endorsing a poisonous implication when it’s provably the case that every possibility remaining amongst the non-utilitarian options is similarly poisonous!
    When all views have costs, you cannot refute a view just by pointing to one of its costs. You need to actually gesture towards a better alternative, and do the difficult work of determining which view is the least bad. Below I’ll briefly step through some basic considerations that bring out how difficult this task can be.
    Population Ethics
    In ‘The New Moral Mathematics’ (reviewing WWOTF), Kieran Setiya sets up a false choice between total utilitarianism and “the intuition of neutrality” which denies positive value to creating happy lives. (Note that MacAskill’s longtermism is in fact much weaker than total utilitarianism.) He swiftly dismisses the total view for implying the repugnant conclusion. But he doesn’t mention any costs to neutralism, which may give some readers the misleading impression that this is a cost-free, common-sense solution. It isn’t. Far from it.
    Neutrality implies that utopia is (in prospect) no better than a barren, lifeless rock. It implies that the total extinction of all future value-bearers could be more than compensated for by throwing a good enough party for those who already exist. These implications strike me as far more repugnant than the repugnant conclus...
    10 min
  • LW - Path dependence in ML inductive biases by Vivek Hebbar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Path dependence in ML inductive biases, published by Vivek Hebbar on September 10, 2022 on LessWrong.
    In this post, we define path dependence as the sensitivity of a model's behavior to the details of the training process and training dynamics. High path-dependence indicates that small changes to the training process can cause significant changes to how the final model generalizes (such as the details of off-distribution behavior). It implies that inner alignment can be reasoned about by thinking about what the model looks like at various stages of training, and how its structure is affected by the immediate pressures of gradient descent. It implies that early-training interventions can be quite potent in shaping how a model turns out, and that a proper theory of inductive bias must reason about the order in which features are learned (where features learned faster/earlier can “screen off” the need for other implementations of a similar niche, in a way that affects the final model).
    In contrast, a world with low path dependence allows us to reason about inductive bias in terms of priors and updates, sparing the details of training dynamics. It is more pessimistic about the ability to steer the model’s ontology through early interventions, believing instead that the final result is overdetermined. As Evan discusses in a previous post, it makes us less worried about variance in alignment outcomes between labs, since small changes to the training procedure don’t strongly affect alignment outcomes.
    Possible mechanistic reasons for high path dependence would include the existence of distinct stable ontologies, the ability for early features to kill gradients, and the difficulty of building highly serial features whose components aren't independently useful. Mechanistic reasons for low path dependence would include grokking-like phase transitions which wipe out early circuits, overdetermination of correct ontologies, and an abundance of low loss paths between seemingly dissimilar solutions.
    We remain mostly agnostic about which world we are in. The purpose of this post is to elucidate the various strands of path dependence, and their implications for alignment. We hope it will encourage people to run experiments determining where reality falls on these spectra.
    Path dependence
    Fundamentally, path dependence is about the sensitivity of a model's behavior to the details of the training process and training dynamics. In this section, we enumerate concrete claims which constitute path dependence. It should be emphasized that it is conceivable for these properties to vary separately, and that we lump them together because we hypothesize without proof that they are correlated.
    In a world with maximum path dependence:
    The outcome of training a DNN depends strongly on what representations and circuits are learned early in training.
    This set of circuits varies between runs and is highly sensitive to the details of the training procedure.
    High-level properties of the model are very likely to stay roughly constant during the late phases of training. We can assess these properties at the middle of training, and our assessment will be highly predictive of the final properties, even if the model gets much smarter.
    Circuits learned early in training are never outright deleted. Parts of the task which were solved early in training continue to be done with the same circuits in the final model.
    The inductive bias doesn’t have a closed form. To figure out which of two final models is more likely, you must reason explicitly about the order in which things are learned; you cannot just look at simplicity or depth or any other static measure of the final circuits.
    By contrast, in a world with minimum path dependence:
    The outcome of training is highly consistent across runs and similar setups, an...
    17 min
  • EA - Much EA value comes from being a Schelling point by LRudL
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Much EA value comes from being a Schelling point, published by LRudL on September 10, 2022 on The Effective Altruism Forum.
    TL;DR: A significant way in which the EA community creates value is by acting as a Schelling point where talented, ambitious, and altruistic people tend to gather and can meet each other (in addition to more direct sources of EA value like identifying the most important problems and directly pushing people to work on them). It might be useful to think about what optimising for being a Schelling point looks like, and I list some vague thoughts on that.
    A Schelling point, also known as a focal point, is what people decide on in the absence of communication, especially when it's important to coordinate by coming to the same answer.
    The classic example is: you were arranging a meeting with a stranger in New York City by telephone, but you used the last minute of your phone credit and the line cut off after you had agreed on the date but not location or time - where do you meet? "Grand Central Station at noon" is an answer that other people may be especially likely to converge on.
    (Schelling points can be thought of as a type of acausal negotiation.)
    When the Schelling point is the selling point
    Schelling points are often extremely powerful and valuable. A key function of top universities is to be Schelling points for talented people. (Personally, I'd call it the most important function.) There are other valuable things too: courses that go deeper, the signalling value to employers, and so on. However, talented people generally have a preference for hanging out with other talented people, both for social reasons and to find collaborators for ambitious projects and future colleagues. At the same time, talented people are also generally spread out and present only at low densities. Top universities select hard on (some measures of) talent, and through this create environments with high talent density.
    A big chunk of the reason why people apply to top universities is because other people do so too, and I'd guess that even if the academic standards of Stanford, MIT, or Cambridge eroded significantly, the fact that they've established themselves as congregating points for smart people will keep people applying and visiting for a long time.
    (Note that this is related to, but not equal to, the prestige and status of these places. It is possible to imagine Schelling points that are not prestigious. For example, my impression is that this described MIT at one point - it became a congregating point for uniquely ambitious STEM students and defence research before it achieved high academic status. It is also possible to imagine prestigious places that are not Schelling points, though this is a bit harder since anything with prestige becomes a Schelling point for high social status (though prestige Schelling points and talent Schelling points need not co-occur). More generally, since prestige is a thing many people care a lot about, there is a high correlation between a place being prestigious or high status and being a Schelling point for at least some type of person.
    However, the mechanisms are distinct - a person selecting their university based on status is selecting based on what they get to write on their CV, while a person selecting their university based on it being a Schelling point for smart people is selecting based on the fact that many other smart people that they can't coordinate with but would like to meet will also choose to go there.)
    Another example is Silicon Valley. Sure, the area has many strengths - being rich and inside a large stable free market - but by far the greatest argument for living in Silicon Valley is that others also choose it. This leads to a (for now) unique combination of entrepreneurial people, great programmers, venture cap...
    17 min
  • AF - Quintin's alignment papers roundup - week 1 by Quintin Pope
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quintin's alignment papers roundup - week 1, published by Quintin Pope on September 10, 2022 on The AI Alignment Forum.
    Introduction
    I've decided to start a weekly roundup of papers that seem relevant to alignment, focusing on papers or approaches that might be new to safety researchers. Unlike the Alignment Newsletter, I'll be spending relatively little effort on summarizing the papers. I'll just link them, copy their abstracts, and potentially describe some of my thoughts on how the paper relates to alignment. Hopefully, this will let me keep to a weekly schedule.
    The purpose of this series isn't so much to share insights directly with the reader, but instead to make them aware of already existing research that may be relevant to the reader's own research.
    Papers
    Locating and Editing Factual Associations in GPT
    We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods.
    To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at this https URL
    My opinion:
    Most people I talk to about this paper have heard of it previously, so it's hardly ”new”. However, I think a lot of people underestimate how significant the paper is. The authors use a very cool interpretability method to show that the middle-stage MLP layers are acting as a key-value memory system. They then guess at the specific mathematical structure these MLP layers use to store information, derive a closed-form, analytic solution to edit the model's knowledge stores and use very thorough evaluations to show that their knowledge editing method is effective and that the edits influence the model's outputs in many different contexts where the new knowledge is relevant. This paper is vastly beyond just "poke random stuff and see that the output changes". Code can be found here.
    Using cognitive psychology to understand GPT-3
    We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large langu...
    18 min
  • EA - Reminder (Sept 15th deadline): Apply for the Open Philanthropy Technology Policy Fellowship by JoanGass
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reminder (Sept 15th deadline): Apply for the Open Philanthropy Technology Policy Fellowship, published by JoanGass on September 9, 2022 on The Effective Altruism Forum.
    The deadline for the Open Philanthropy Technology Policy Fellowship (OPTPF) is September 15th. If you are interested in working in US policy on topics such as AI and biosecurity, we strongly encourage you to consider applying!
    The first cohort of 15 OPTPF fellows received 10 weeks of policy training sessions together, and 100% of them matched with a host organization. They have now started placements in the executive branch (in the Departments of Defense, Health and Human Services, and Homeland Security), congressional offices, and think tanks (CSET, NTI, CHS, CSIS, Brookings, CDT, and CEIP).
    To learn more about the program, you can check out the fellowship page, this past Forum post, and the fellowship FAQ.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Ought will host a factored cognition “Lab Meeting” by jungofthewon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ought will host a factored cognition “Lab Meeting”, published by jungofthewon on September 9, 2022 on The AI Alignment Forum.
    Ought will host a factored cognition “Lab Meeting” on Friday September 16 from 9:30AM - 10:30AM PT.
    We'll share the progress we've made using language models to decompose reasoning tasks into subtasks that are easier to perform and evaluate. This is part of our work on supervising process, not outcomes. It’s easier for us to show you than to tell you about it in a post (though written updates will hopefully follow).
    Then, we'll cover outstanding research directions we see and plan to work on, many almost shovel-ready. If the alignment community can parallelize this work across different alignment research teams, we can make progress faster. We'd love to coordinate with other alignment researchers thinking about task decomposition, process supervision, factored cognition, and IDA-like approaches (where efficient to do so). We want to save you time and mistakes if we can!
    What is the agenda?
    30 min | Updates on Ought’s work decomposing reasoning tasks
    The specific alignment problems we’re trying to solve and our vision of a solution that mitigates these risks (see also: Supervise Process, not Outcomes)
    Early progress on decomposing reasoning about evidence quality in randomized controlled trials
    Our tools for building and debugging reasoning traces of language models (preview)
    15 min | Related research directions we’re excited about and how they fit in, e.g.
    Automating evaluation through critique models or verifier models
    Distillation
    Comparing scaling trends for process-based systems vs. end-to-end systems
    Testing process-based systems for adversarial robustness
    15 min | Q&A
    There will be more to discuss than we can fit into an hour. We’ll get to what we can and consider making this a regular meeting if there’s appetite (likely with more sharing from other researchers)!
    Who should attend?
    You should attend if:
    You are interested in Ought’s research and want updates.
    You want to build off of Ought’s learnings from doing this research.
    You want to use our tools for running and debugging compositional language models tasks.
    You want concrete research ideas in this domain.
    You are not a researcher but want to learn how other backgrounds can support this work (engineers can build debugging infrastructure, non-ML researchers can help create datasets, etc.).
    How can I attend?
    You can register for the Lab Meeting here. Email [email protected] if you have any questions!
    The meeting will be recorded & shared.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - ethics and anthropics of homomorphically encrypted computations by carado
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ethics and anthropics of homomorphically encrypted computations, published by carado on September 9, 2022 on LessWrong.
    suppose you are a superintelligence that is aligned with some human values. you are going about your day, tiling the cosmos with compute that can be used for moral patients to have nice experiences on, annihilating some alien superintelligences and trading with some others, uploading alien civilizations you find to make sure they experience utopia, or at least when you have no other choice genociding them to avoid sufficiently bad suffering from being instantiated.
    one day, you run into a planet running a very large computer. after a short investigation, you realize that it's running a very large homomorphically encrypted computation (hereby "HEC"), and the decryption key is nowhere to be found. it could contain many aliens frolicking in utopia. it could contain many aliens suffering in hell. or, it could be just a meaningless program merely wasting compute, with no moral patients inside it.
    if you had the encryption key, you might be able to encrypt a copy of yourself which would be able to take over the HEC from the inside, ensuring (in a way that the outside would never be able to observe) that everything is going fine, in the same way that you should send copies of yourself into remote galaxies before they retreat from us faster than we can reach them.
    if you had found some way to get infinite compute (without significant loss of anthropic/ethics juice, then you could use to just break the HEC open and actually ensure its contents are doing okay.
    but let's say the encryption key is nowhere to be found, and accessible compute is indeed scarce. what are your options?
    interrupt the entire computation.
    let it run, and even safeguard it.
    now of course, when faced with the possibility of S-risks, i tend to say "better safe than sorry". what the superintelligence would do would be up to the values it's been aligned to, which hopefully are also reasonably conservative about avoiding S-risks.
    but here's something interesting: i recently read a post on scott aaronson's blog which seems to claim that there's a sense in which the event horizon of a black hole (or of something like a black hole?) can act just like a HEC's computational event horizon: there's a sense in which being able to go in but not get out is not just similar to a situation with a HEC for which you have the encryption but not decryption key, but is actually that same situation.
    furthermore, a pair of comments by vanessa kosoy (of PreDCA) seems to suggest that infra-bayesianism physicalism would say "this HEC contains no suffering, merely random compute" rather than "i'm unable to know whether this HEC contains suffering"; and she even bites the bullet that moral patients past the event horizon of black holes also don't "have experiences".
    (one example of why you might care whether moral patients in black holes "have experiences" is if you can influence what will happen in a black hole — for example, imagine a rocket with moral patients on board is headed for a black hole, and before it gets there, you get to influence how much suffering will happen on board after the rocket passes the event horizon)
    i would like to argue that this can't be right, based on several counterintuitive results.
    first, consider the case of a HEC running a giant civilization for a while, and then reducig down to one bit of output, and emitting that single bit of output as its own decrypted output. does the civilizaton now "count"? if the people inside the civilization have no anthropics juice, where has the cosmos done the work determining that bit? or do they suddenly count as having had experiences all at once when the single bit of output is emitted? and then, surely, if they have anthropics juice then they m...
    5 min
  • LW - My emotional reaction to the current funding situation by Sam
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My emotional reaction to the current funding situation, published by Sam on September 9, 2022 on LessWrong.
    I’m allowed to spend two days a week at Trajan House, a building in Oxford which houses the Center for Effective Altruism (CEA), along with a few EA-related bodies. Two days is what I asked for, and what I received. The rest of the time I spend in the Bodleian Library of the University of Oxford (about £30/year, if you can demonstrate an acceptable “research need”), a desk at a coworking space in Ethical Property (which houses Refugee Welcome, among other non-EA bodies, for £200/month), Common Ground (a cafe/co-working space which I’ve recommended to people as a place where the staff explicitly explain, if you ask, that you don’t need to order anything to stay as long as you like), a large family house I’m friends with, and various cafes and restaurants where I can sit for hours while only drinking mint tea.
    I’m allowed to use the hot-desk space at Trajan House because I’m a recipient of an EA Long Term Future Fund grant, to research Alignment. (I call this “AI safety” to most people, and sometimes have to explain that AI stands for Artificial Intelligence.) I judged that 6 months of salary at the level of my previous startup job, with a small expenses budget, came to about £40,000. This is what I asked for, and what I received.
    At my previous job I thought was having a measurable, meaningful impact on climate change. When I started there, I imagined that I’d go on to found my own startup. I promised myself it would be the last time I’d be employed.
    When I quit that startup job, I spent around a year doing nothing-much. I applied to Oxford’s Philosophy BPhil, unsuccessfully. I looked at startup incubators and accelerators. But mostly, I researched Alignment groups. I visited Conjecture, and talked to people from Deep Mind, and the Future of Humanity Institute. What I was trying to do, was to discern whether Alignment was “real” or not. Certainly, I decided, some of these people were cleverer than me, more hard-working than me, better-informed. Some seem deluded, but not all. At the very least, it’s not just a bunch of netizens from a particular online community, whose friend earned a crypto fortune.
    During the year I was unemployed, I lived very cheaply. I’m familiar with the lifestyle, and – if I’m honest – I like it. Whereas for my holidays while employed I’d hire or buy a motorbike, and go travelling abroad, or scuba dive, instead my holidays would be spent doing DIY at a friend’s holiday home for free board, or taking a bivi bag to sleep in the fields around Oxford.
    The exceptions to this thrift were both EA-related, and both fully-funded. In one, for which my nickname of “Huel and hot-tubs” never caught on, I was successfully reassured by someone I found very smart that my proposed Alignment research project was worthwhile. In the other, I and others were flown out to the San Francisco Bay Area for an all-expenses-paid retreat to learn how to better build communities. My hotel room had a nightly price written on the inside of the door: $500. Surely no one ever paid that. Shortly afterwards, I heard that the EA-adjacent community were buying the entire hotel complex.
    While at the first retreat, I submitted my application for funding. While in Berkeley for the second, I discovered my application was successful. (“I should hire a motorbike, while I’m here.” I didn’t have time, between networking opportunities.) I started calling myself an “independent alignment researcher” to anyone who would listen and let me into offices, workshops, or parties. I fit right in.
    At one point, people were writing plans on a whiteboard for how we could spend the effectively-infinite amount of money we could ask for. Somehow I couldn’t take it any more, so I left, crossed the ro...
    9 min