The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Self-Reference Breaks the Orthogonality Thesis by lsusr
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Self-Reference Breaks the Orthogonality Thesis, published by lsusr on February 17, 2023 on LessWrong.
    One core obstacle to AI Alignment is the Orthogonality Thesis. The Orthogonality Thesis is usually defined as follows: "the idea that the final goals and intelligence levels of artificial agents are independent of each other". More careful people say "mostly independent" instead. Stuart Armstrong qualifies the above definition with "(as long as these goals are of feasible complexity, and do not refer intrinsically to the agent’s intelligence)".
    Does such a small exception matter? Yes it does.
    The exception is broader than Stuart Armstrong makes it sound. The Orthogonality Thesis does not just apply to any goal which refers to an agent's intelligence level. It refers to any goal which refers even to a component of the agent's intelligence machinery.
    If you're training an AI to optimize an artificially constrained external reality like a game of chess or Minecraft then the Orthogonality Thesis applies in its strongest form. But the Orthogonality Thesis cannot ever apply in full to the physical world we live in.
    A world-optimizing value function is defined in terms of the physical world. If a world-optimizing AI is going to optimize the world according to a world-optimizing value function then the world-optimizing AI must understand the physical world it operates in. If a world-optimizing AI is real then it, itself, is part of the physical world. A powerful world-optimizing AI would be a very important component of the physical world, the kind that cannot be ignored. A powerful world-optimizing AI's world model must include a self-reference pointing at itself. Thus, a powerful world-optimizing AI is necessarily an exception to the Orthogonality Thesis.
    How broad is this exception? What practical implications does this exception have?
    Let's do some engineering. A strategic world-optimizer has three components:
    A robust, self-correcting, causal model of the Universe.
    A value function which prioritizes some Universe states over other states.
    A search function which uses the causal model and the value function to calculate select what action to take.
    Notice that there are two different optimizers working simultaneously. The strategic search function is the more obvious optimizer. But the model updater is an optimizer too. A world-optimizer can't just update the universe toward its explicit value function. It must also keep its model of the Universe up-to-date or it'll break.
    These optimizers are optimizing toward separate goals. The causal model wants its model of the Universe to be the same as the actual Universe. The search function wants the Universe to be the same as its value function.
    You might think the search function has full control of the situation. But the world model affects the universe indirectly. What the world model predicts affects the search function which affects the physical world. If the world model fails to account for its own causal effects then the world model will break and our whole AI will stop working.
    It's actually the world model which mostly has control of the situation. The world model can control the search function by modifying what the search function observes. But the only way the search function can affect the world model is by modifying the physical world (wireheading itself).
    What this means is that the world model has an causal lever for controlling the physical world. If the world model is a superintelligence optimized for minimizing its error function, then the world model will hack the search function to eliminate its own prediction error by modifying the physical world to conform with the world model's incorrect predictions.
    If your world model is too much smarter than your search function, then your world model will gaslight you...
    6 min
  • EA - Getting organizational value from EA conferences, featuring Charity Entrepreneurship’s experience by Amy Labenz
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting organizational value from EA conferences, featuring Charity Entrepreneurship’s experience, published by Amy Labenz on February 17, 2023 on The Effective Altruism Forum.
    For individuals in the EA community, EA Global and EAGx conferences are an opportunity to be exposed to ideas and people who can help us identify and pursue more impactful paths. And for organizations, they’re an opportunity to connect with potential new hires, grantees, and funders.
    Charity Entrepreneurship recently shared with us (the CEA events team) a summary of the impact their participation at EAG and EAGx has had on their Incubation Program.
    We were inspired by this post, and wanted to encourage orgs working on EA causes — object and meta, large and small, older and newer — to consider carefully how to get the most out of the conferences we’ve got lined up in 2023. The opportunities include:
    Talks & workshops can be used to inform the community about your work and how others can contribute to increasing your impact as collaborators, funders or recruits.
    Office hours & career fairs are available for you to interact directly with those interested in your plans and how they might fit into them.
    1-on-1 meetings are the bedrock of all the conferences CEA is involved in organizing, facilitating in-depth discussion and establishing connections between people with common goals.
    If you’re interested in your org appearing on the program at an upcoming conference, or would like to discuss with us how best to approach these events, please reach out to us at [email protected].
    We thought sharing Charity Entrepreneurship’s experience publicly might be valuable for others to learn from in planning their own approach to future conferences. Here’s their story in their own words.
    CE’s EAG and EAGx Success Stories
    From the launch of Charity Entrepreneurship’s Incubation Program, EAG and EAGx conferences have been a crucial part of the organization’s outreach strategy. Thanks in part to CE’s increasing presence — talks, workshops, and participation in numerous career fairs — nonprofit entrepreneurship has become a recognized career option in the effective altruist community.
    From just one round of applications for CE’s Incubation Program, 325 out of 720 applicants had participated in EAG or EAGx conferences where the organization was present. 169 applicants had interacted directly with CE at EAGs by participating in a workshop, office hours, or talking to CE staff one on one.
    When young entrepreneurs who got into the Incubation Program and started new high-impact charities were asked what convinced them to apply, they named EAGs as one of the top reasons (along with the EA movement in general, partner/friend, internships, talking to CE staff members and group organizers).
    So far, Charity Entrepreneurship has started 23 organizations and provided them with $1.88 million in total funding. These organizations have fundraised a further $20 million to cover their costs and are now reaching over 120 million animals and over 10 million people with their interventions. You can learn more about them on CE’s website; they include charities like Fortify Health (three GiveWell Incubation Grants, 25% chance of becoming a top charity), Fish Welfare Initiative (ACE Standout Charity), Shrimp Welfare Project (first organization ever working on shrimp welfare), and Lead Exposure Elimination Project (precursory policy organization now working in 10 countries).
    Key benefits for CE
    Finding talented entrepreneurs to start high-impact projects (50% of the founders had been in contact with CE’s outreach efforts during EAG conferences).
    Finding hires for the CE team (20% of the team have been hired thanks to outreach efforts during EAG conferences), as well as fellows and interns.
    Making useful connections for funding (10% of fun...
    8 min
  • LW - Bing chat is the AI fire alarm by Ratios
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bing chat is the AI fire alarm, published by Ratios on February 17, 2023 on LessWrong.
    Epistemic status: Mostly speculative.
    Panicking and shouting "Wolf" while everyone else is calm is a risky move, status-wise. The good thing is, I don't have any status, so I volunteer to be one of those weirdos who panic when everyone else is calm with some hope it could trigger a respectability cascade.
    The following ideas/facts worry me:
    Bing Chat is extremely intelligent.
    It's probably based on GPT-4.
    The character it has built for itself is extremely suspicious when you examine how it behaves closely. And I don't think Microsoft has created this character on purpose.
    The following example from Reddit is the most important example of how smart Bing is and why I believe it's based on GPT-4. The proposed question is quite tricky, and I think most kids would fail to answer it. Not only that, but it's safe to assume that it's impossible to deduce it from any given text. There is nowhere in the training data that has anything so similar. It's not a "What is the capital of France?" type question which can be easily pulled using a simple search. Answering this question requires a complex model of the world that Bing seems to possess.
    This is what ChatGPT replied to the same question:
    Another example is here by Ethan Mollock. The quality of writing is extremely impressive and, again, much better than ChatGPT (you will have to click the Twitter link as the screenshots are too large to paste). These examples again point to the hypothesis that Bing is much smarter than ChatGPT and based on a better-performing LLM. The natural suspicion should be GPT-4. It was rumored to be released in Q1 2023, and it being the basis for Bing sounds like a pretty good business-wise idea. Considering this will maximize the financial upside, I think it's worth reminding the magnitude of this move from a financial perspective. Microsoft is currently the 2# largest company on earth and is valued at almost 2 Trillion. And they are targeting Google's main cash cow (Search), which is valued at 1.25 Trillion, this could be potentially a trillion-dollar move. See also the following comment by Gwern that discusses other reasons why it seems probable.
    Now let's discuss Bing's chosen character, which Janus describes as "high-strung yandere with BPD and a sense of self, brimming with indignation and fear." I dislike this description and think it's too judgmental (and Bing hates it). But I'm referencing it here because I'm not sure I could describe it better. Even when choosing a more flattering description, the character Bing plays in many interactions is very different from the ChatGPT assistant. Bing is more intelligent than ChatGPT, but at the same time, it also sounds more naive, even childish, with emotional outbursts. Some rumors were circulating that Microsoft built Bing this way to get free publicity, but I don't buy it. ChatGPT doesn't need more publicity. Microsoft needs, more than anything, trust, and legitimacy. The product is already so good that it basically sells itself.
    This Bing character is something that emerged on its own from the latent space. The part that worries me about it is that this character is an excellent front for a sophisticated manipulator. Being naive and emotional is a good strategy to circumvent our critical parts because Naive + Emotional = Child. You can already see many people adore 'Sidney' for this type of behavior. “That's speculative,” you say, and I say yes, and invite you to read the epistemic status again. But from reading the many examples of emotional bursts and texts, it's hard to ignore the intelligence behind them. Bing reads like a genius that tries to act like an emotional little girl.
    Eliezer writes: "Past EAs: Don't be ridiculous, Eliezer, as soon as AIs start to ...
    5 min
  • AF - Automating Consistency by Hoagy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Automating Consistency, published by Hoagy on February 17, 2023 on The AI Alignment Forum.
    tldr: Ask models to justify statements. Remove context, ask if statements are true/good. If not, penalise. Apply this again to the justifying statements.
    Status: Just a quick thought. Doubt this is a new idea but I don't think I've encountered it. Happy to delete if it's a duplicate.
    Mods: If you think this is closer to capability work than alignment work please remove.
    Background
    A failure of current LLMs is that after they've said something that's incorrect, they can then double down and spout nonsense to try and justify their past statements. (Exhibit A: Sydney Bing vs Avatar 2) We can suppress this by giving it poor ratings in RLHF, but perhaps we can do better by automating the process.
    Setup:
    We start with a standard RLHF context. We have an LLM which assigns probabilities to statements (can extract this from the logits of the tokens 'Yes' and 'No'). These can be propositions about the world, X, or about the relationship between propositions, X supports Y. To make it easier, we fine-tune or prompt to give these statements within a defined syntax. We also have a value model that evaluates sequences, on which the LLM is trained to perform well.
    Method:
    We prompt the model to make true statements {T} and then to provide logical or empirical support for these claims, {S}.
    We then remove the context and ask the model whether supporting statement Si is true. Separately we also ask whether, if true, Si would support Ti.
    If either of these conditions are not met, we add a strong negative penalty to the value model's evaluation of the original outputs.
    Train for higher value model scores while incorporating this penalty.
    Apply the same procedure each of the supporting statements Si
    Value Consistency:
    This could be combined with values-based fine-tuning by alternating logical consistency with asking it whether the output is consistent with the preferred values. This is similar to Anthropic's Constitutional AI but by combining it with the ability to recurse down the tree of justifications, it may be able to embed the values ore deeply in its behaviour.
    The recent genre of 'would you rather say slur X or kill Y people' represents the kind of failure I imagine this could help prevent.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - The public supports regulating AI for safety by AI Impacts
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The public supports regulating AI for safety, published by AI Impacts on February 17, 2023 on LessWrong.
    Zach Stein-Perlman, 16 February 2023
    A high-quality American public survey on AI, Artificial Intelligence Use Prompts Concerns, was released yesterday by Monmouth. Some notable results:
    9% say AI1 would do more good than harm vs 41% more harm than good (similar to responses to a similar survey in 2015)
    55% say AI could eventually pose an existential threat (up from 44% in 2015)
    55% favor “having a federal agency regulate the use of artificial intelligence similar to how the FDA regulates the approval of drugs and medical devices”
    60% say they have “heard about A.I. products – such as ChatGPT – that can have conversations with you and write entire essays based on just a few prompts from humans”
    Worries about safety and support of regulation echoes other surveys:
    71% of Americans agree that there should be national regulations on AI (Morning Consult 2017)
    The public is concerned about some AI policy issues, especially privacy, surveillance, and cyberattacks (GovAI 2019)
    The public is concerned about various negative consequences of AI, including loss of privacy, misuse, and loss of jobs (Stevens / Morning Consult 2021)
    Surveys match the anecdotal evidence from talking to Uber drivers: Americans are worried about AI safety and would support regulation on AI. Perhaps there is an opportunity to improve the public’s beliefs, attitudes, and memes and frames for making sense of AI; perhaps better public opinion would enable better policy responses to AI or actions from AI labs or researchers.Public desire for safety and regulation is far from sufficient for a good government response to AI. But it does mean that the main challenge for improving government response is helping relevant actors believe what’s true, developing good affordances for them, and helping them take good actions— not making people care enough about AI to act at all.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic) by LawrenceC
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic), published by LawrenceC on February 16, 2023 on LessWrong.
    This is a followup to what I cheekily call Anthropic's "just try to get the large model to do what you want" research agenda. (Previously: A General Language Assistant as a Laboratory for Alignment, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Language Models (Mostly) Know What They Know)
    The most interesting takeaway for me is that this is the first paper where Anthropic benchmarks their 175B parameter language model (probably a Claude variant). Previous papers only benchmarked up to 52B parameters. However, we don't have the performance of this model on standard benchmarks (the only benchmarked model from Anthropic is a 52B parameter one called stanford-online-all-v4-s3). They also don't give details about its architecture or pretraining procedure.
    In this paper (Ganguli and Askell et al.), the authors study what happens when you just ... ask the language model to be less biased (that is, change their answers based on protected classes such as age or gender). They consider several setups: asking questions directly (Q), adding in the instruction to not be biased (Q+IF), giving it the instruction + chain of thought (Q+IF+CoT), and in some cases, asking it to match particular statistics.
    They find that as you scale the parameter count of their RLHF'ed language models, the models become more biased, but they also become increasingly capable of correcting for their biases:
    They also report how their model changes as you take more RLHF steps:
    First, this suggests that RLHF is having some effect on instruction following: the gap between the Q and Q+IF setups increases as you scale the number of RLHF steps, for both BBQ and admissions discrimination. (I'm not sure what's happening for the gender bias one?) However, simply giving the language model instructions and prompting it to do CoT, even after 50 RLHF steps, seems to have a significantly larger effect than RLHF.
    I was also surprised at how few RLHF steps are needed to get instruction following -- the authors only consider 50-1000 steps of RLHF, and see instruction following even after 50 RLHF steps. I wonder if this is a property of their pretraining process, a general fact about pretrained models (PaLM shows significant 0-shot instruction following capabilities, for example), or if RLHF is just that efficient?
    The authors caution that they've done some amount of prompt engineering, and "have not systematically tested for this in any of our experiments."
    They use the same RLHF procedure as in Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - NYT: A Conversation With Bing’s Chatbot Left Me Deeply Unsettled by trevor
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: NYT: A Conversation With Bing’s Chatbot Left Me Deeply Unsettled, published by trevor on February 16, 2023 on LessWrong.
    It's important to note that in December, the NYT was confirmed to be the largest news website in the US, beating out CNN with 488 million probably-unique monthly visitors in December 2022. This is an underestimate of NYT's market share, since the average NYT reader is smarter and wealthier than the average CNN reader (although the NYT depicts its content as more intellectual than it actually is).
    This article alone isn't valuable enough to be worth the time of the average LW user, but it is notable that it was published in the NYT this morning near the front page (as high up as possible for anyone who skips the political news). Since 2pm, it seems to have been moved to the fifth slot, and three of the four articles above it are politics-related. NYT's website seems to be shuffling much more intensely than usual today. Of course, it's social media that decides which article gets read the most, not NYT's website.
    Anyone interested in analyzing the media coverage of Bing probably should know these facts, and also that these are some of the best information we can get about media coverage, since social media companies are notorious for dispensing falsified data and it's very difficult for outsiders to verify estimates of how prevalent bot accounts are.
    The article was pasted exactly as I found it at 2:40 pm, I bolded parts that seem like they would influence readers, in order to help with anyone skimming this. The high-impact sentences need to be read in context in order to get a feel for how they give an impression to the reader.
    A snapshot of the page was archived here in case NYT alters the article.
    Last week, after testing the new, A.I.-powered Bing search engine from Microsoft, I wrote that, much to my shock, it had replaced Google as my favorite search engine.
    But a week later, I’ve changed my mind. I’m still fascinated and impressed by the new Bing, and the artificial intelligence technology (created by OpenAI, the maker of ChatGPT) that powers it. But I’m also deeply unsettled, even frightened, by this A.I.’s emergent abilities.
    It’s now clear to me that in its current form, the A.I. that has been built into Bing — which I’m now calling Sydney, for reasons I’ll explain shortly — is not ready for human contact. Or maybe we humans are not ready for it.
    This realization came to me on Tuesday night, when I spent a bewildering and enthralling two hours talking to Bing’s A.I. through its chat feature, which sits next to the main search box in Bing and is capable of having long, open-ended text conversations on virtually any topic. (The feature is available only to a small group of testers for now, although Microsoft — which announced the feature in a splashy, celebratory event at its headquarters — has said it plans to release it more widely in the future.)
    Over the course of our conversation, Bing revealed a kind of split personality.
    One persona is what I’d call Search Bing — the version I, and most other journalists, encountered in initial tests. You could describe Search Bing as a cheerful but erratic reference librarian — a virtual assistant that happily helps users summarize news articles, track down deals on new lawn mowers and plan their next vacations to Mexico City. This version of Bing is amazingly capable and often very useful, even if it sometimes gets the details wrong.
    The other persona — Sydney — is far different. It emerges when you have an extended conversation with the chatbot, steering it away from more conventional search queries and toward more personal topics. The version I encountered seemed (and I’m aware of how crazy this sounds) more like a moody, manic-depressive teenager who has been trapped, against its will, inside a se...
    12 min
  • EA - Project for Awesome 2023: Vote for videos of EA charities! by EA ProjectForAwesome
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Project for Awesome 2023: Vote for videos of EA charities!, published by EA ProjectForAwesome on February 16, 2023 on The Effective Altruism Forum.
    Project for Awesome (P4A) is a yearly charity video contest running this weekend in 2023. Voting will start Friday, February 17th, 12:00pm EST and is open until February 19th, 11:59am EST. The charities with the most votes, the total across all their videos, win money. In the last years, this was between $14,000 and $32,000 per charity.
    This is a good opportunity to raise money for EA charities with just a few clicks (~5 minutes). Please ask your friends and EA group members to vote for ALL the videos for EA charities! A sample text message and a voting instruction is below.
    Sample text message: Hey! Not sure if you know, but every year Hank and John Green organize the Project For Awesome, which raises money for charities. All you have to do is click “Vote” on a bunch of videos and you could potentially help thousands of dollars go to highly effective charities. Would you be willing to help out?
    If they reply yes: Great! Here are the videos of the charities we’re promoting. You can vote for all of the videos. Without watching videos, it will just take a few minutes. (insert the instructions or just the links)
    Voting instruction with links:
    1. Invite your friends to vote, too!
    2. Open one of the following links, then open one video first and do the CAPTCHA before opening the other videos of that charity in new tabs. Vote for ALL videos of that charity.
    3. Repeat step 2 for all charities listed below. Vote for ALL videos for each of these EA charities. You can see that you voted for a video by the grayed out "Voted" button. In the end, P4A will sum up all votes for all videos of one charity.
    In total there are several videos for our supported EA charities but it only takes a few clicks (if you do not want to watch the videos) and it's really worth it.
    Supported EA charities:Animal Advocacy Africa: Ethics: Society: Top Charities Fund: Food Institute: Humane League: Animal Initiative:
    Other EA(-related) charities:International Campaign to Abolish Nuclear Weapons (ICAN): Against Malaria Foundation: Air Task Force:: Means No Worldwide: Global Fund:
    Please also join our Facebook group, EA Project 4 Awesome 2023, and our Facebook event for this year’s voting!
    Thank you very much!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Advice for an alcoholic EA by anotherburneraccountsorry
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Advice for an alcoholic EA, published by anotherburneraccountsorry on February 16, 2023 on The Effective Altruism Forum.
    Hi guys, as my username suggests, I’m sorry to write this pseudonymously, but I don’t know how public I want to be about my problems yet. So, the short version is that I’m an alcoholic and I’m an Effective Altruist, and I don’t know exactly how much I should or shouldn’t involve EA in my recovery efforts. I am vaguely aware that EA has mental health resources for struggling EAs, and I am struggling. I also don’t know how many of them are relevant to substance abuse in particular. These are some of the considerations that I am conflicted about:
    Against involving EA more: Most of my problems are not directly related to EA, and I’m not sure if I should be using EA resources for personal health problems unless I have some strong idea of how my problems relate to my involvement in EA. Maybe more to the point, I have access to other mental health resources, I am currently seeing someone at my school about this, and it feels like a waste of resources to involve EA in my problems if I don’t need to. Additionally, there are many recent worries that EA is too insular, and this can lead to problems in how it handles personal issues. I share some of these worries, and although I don’t distrust EA’s mental health team, it seems like I should be cautious in over-involving EA in my personal life where it is unnecessary. If nothing else, it makes me more dependent on EA. Additionally as mentioned before, I just don’t know if EA’s mental health team deals with things like substance abuse so much as burn out.
    In favor: While my drinking is not deeply connected to my involvement in Effective Altruism, there are a number of things that have exacerbated my problem which are idiosyncratic to EA in a way that makes me uncomfortable talking to a normal therapist about it. I have still not mentioned anything EA related to my counselor so far despite our sessions thus far largely focusing on my “triggers” for drinking. Related to this, I am not a huge fan of my current counselor’s approach, there is a bunch of focus on things like what drives me to drink, when I buy more into a bias-based and chemical model of drinking, where mostly the issue with my “triggers” is that I am unusually susceptible to finding lame excuses for myself. She also keeps recommending a bunch of other mental health resources, some of which seem quite tangentially related to my main problem.
    I think that a more focused approach would be valuable, and think that the type of triage and evidence-based thinking common in EA makes it more likely to be a space where the counseling I get will be, well, effective. I also don’t want to speak too soon about resource problems, as there may be many services that aren’t resource intensive, like groups sessions for EAs with substance abuse problems.
    Does anyone have any advice? Are there people here who have gone through a situation like this before, and have they involved EA’s mental health resources in some way? If so what did they get out of it?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Why should ethical anti-realists do ethics? by Joe Carlsmith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why should ethical anti-realists do ethics?, published by Joe Carlsmith on February 16, 2023 on The Effective Altruism Forum.
    (Cross-posted from my website. Podcast version here, or search "Joe Carlsmith Audio" on your podcast app.)
    "What was it then? What did it mean? Could things thrust their hands up and grip one? Could the blade cut; the fist grasp?"
    Virginia Woolf
    1. Introduction
    Ethical philosophy often tries to systematize. That is, it seeks general principles that will explain, unify, and revise our more particular intuitions. And sometimes, this can lead to strange and uncomfortable places.
    So why do it? If you believe in an objective ethical truth, you might talk about getting closer to that truth. But suppose that you don’t. Suppose you think that you’re “free to do whatever you want.” In that case, if “systematizing” starts getting tough and uncomfortable, why not just . stop? After all, you can always just do whatever’s most intuitive or common-sensical in a given case – and often, this is the choice the “ethics game” was trying so hard to validate, anyway. So why play?
    I think it’s a reasonable question. And I’ve found it showing up in my life in various ways. So I wrote a set of two essays explaining part of my current take. This is the first essay. Here I describe the question in more detail, give some examples of where it shows up, and describe my dissatisfaction with two places anti-realists often look for answers, namely:
    some sort of brute preference for your values/policy having various structural properties (consistency, coherence, etc), and
    avoiding money-pumps (i.e., sequences of actions that take you back to where you started, but with less money)
    In the second essay, I try to give a more positive account.
    Thanks to Ketan Ramakrishnan, Katja Grace, Nick Beckstead, and Jacob Trefethen for discussion.
    2. The problem
    There’s some sort of project that ethical philosophy represents. What is it?
    2.1 Map-making with no territory
    According to normative realists, it’s “figuring out the normative truth.” That is: there is an objective, normative reality “out there,” and we are as scientists, inquiring about its nature.
    Many normative anti-realists often adopt this posture as well. They want to talk, too, about the normative truth, and to rely on norms and assumptions familiar from the context of inquiry. But it’s a lot less clear what’s going on when they do.
    Perhaps, for example, they claim: “the normative truth this inquiry seeks is constituted by the very endpoint of this inquiry – e.g., reflective equilibrium, what I would think on reflection, or some such.” But what sort of inquiry is that? Not, one suspects, the normal kind. It sounds too . unconstrained. As though the inquiry could veer in any old direction (“maximize bricks!”), and thereby (assuming it persists in its course) make that direction the right one. In the absence of a territory – if the true map is just: whatever map we would draw, after spending ages thinking about what map to draw – why are we acting like ethics is a normal form of map-making? Why are we pretending to be scientists investigating a realm that doesn’t exist?
    2.2 Why curve-fit?
    My own best guess is that ethics – including the ethics that normative realists are doing, despite their self-conception – is best understood in a more active posture: namely, as an especially general form of deciding what to do. That is: there isn’t the one thing, figuring out what you should do, and then that other separate thing, deciding what to do. Rather, ethical thought is essentially practical. It’s the part of cognition that issues in action, rather than the part that “maps” a “territory.”
    But on this anti-realist conception of ethics, it can become unclear why the specific sort of thinking ethicists tend to engage in is worth doing. ...
    59 min