The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research report: meta-analysis on sexual violence prevention programs, published by Seth Ariel Green on March 25, 2024 on The Effective Altruism Forum.

This post summarizes a new paper: Preventing Sexual Violence - A Behavioral Problem Without a Behaviorally-Informed Solution, on which we are coauthors along with Roni Porat, Ana P. Gantman, and Elizabeth Levy Paluck.

The vast majority of papers try to change ideas about sexual violence and are relatively successful at that. However, on the most crucial outcomes - perpetration and victimization - the primary prevention literature has not yet found its footing. We argue that the field should take a much more behavioralist approach and focus on the environmental and structural determinants of violence.

The literature as a whole

We surveyed papers written between 1986 and 2018 and found 224 manuscripts describing 298 studies, from which we coded 499 distinct point estimates.

We looked specifically at primary prevention efforts, which aim to prevent violence before it happens. This is in contrast to secondary prevention, which, per the CDC, comprises "[i]mmediate responses after sexual violence has occurred to deal with the short-term consequences of violence." We also didn't meta-analyze studies where an impact on sexual violence was a secondary or unanticipated consequence of, e.g. giving cash to women unconditionally or opening adult entertainment establishments.

We also didn't look at anything that tries to reduce violence by focusing on the behavior of potential victims, e.g. self-defense classes or "sexually assertive communication training."

We also didn't look at especially high-risk populations, like people who are incarcerated or sex workers.

Here are some graphical overviews:

Here is the distribution of studies over time, with three "zeitgeist" programs highlighted.

Three zeitgeists programs

We highlight three "pioneering and influential programs" that "represent the prevalent approaches to sexual violence prevention in a particular period of time."

The first is Safe Dates (Foshee et al. 1996), which "makes use of multiple strategies, including a play performed by students, a poster contest, and a ten-session curriculum." The core idea is that "perpetration and victimization may be decreased by changing dating abuse norms and gender stereotypes, and improving students' interpersonal skills including positive communication, anger management and conflict resolution."

The second is the Men's Program (Foubert, Tatum & Donahue 2006), which aims to prevent sexual violence by men by increasing their empathy and support for victims of sexual violence, and by reducing their resistance to violence prevention programs. E.g.:

Participants in the program watched a 15-minute dramatization of a male police officer who was raped by two other men, and then dealt with the aftermath of the assault. Trained peer educators then told the participants that the perpetrators were heterosexual and known to the victim, and attempted to draw connections between the male police officer's experience and common sexual violence experiences among women.

Participants were then taught strategies for supporting a rape survivor; definitions of consent; and strategies for intervening when a peer jokes about rape or disrespects women, and in situations where a rape may occur.

The third is Bringing in the Bystander (Banyard, Moynihan, & Plante 2007), which

puts helping others in danger and speaking up against sexist ideas (i.e., "bystanding") at the center of the intervention. As a result, the target behavior change is moved from decreasing perpetration behavior to increasing bystander behavior. The intervention is aimed not at men as potential perpetrators and women as potential victims but everyone as a potential person who can intervene and stop s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Neuronpedia: Platform for accelerating research into Sparse Autoencoders, published by Johnny Lin on March 25, 2024 on The AI Alignment Forum.

This posts assumes basic familiarity with Sparse Autoencoders. For those unfamiliar with this technique, we highly recommend the introductory sections of

these

papers.

TL;DR

Neuronpedia is a platform for mechanistic interpretability research. It was previously focused on crowdsourcing explanations of neurons, but we've pivoted to accelerating researchers for Sparse Autoencoders (SAEs) by hosting models, feature dashboards, data visualizations, tooling, and more.

Important Links

Explore: The SAE research focused Neuronpedia. Current SAEs for GPT2-Small:

RES-JB: Residuals - Joseph Bloom (294k feats)

ATT-KK: Attention Out - Connor Kissane + Robert Kryzanowski (344k feats)

Upload: Get your SAEs hosted by Neuronpedia: fill out this <10 minute application

Participate: Join #neuronpedia on the Open Source Mech Interp Slack

Neuronpedia has received 1 year of funding. Johnny Lin is full-time on engineering, design, and product, while Joseph Bloom is supporting with high-level direction and product management. We'd love to talk to you about how Neuronpedia can speed up your SAE research. Fill out

this short form to get in touch.

Introduction

Why Neuronpedia?

While SAEs are exciting, they introduce a number of engineering challenges which can slow or even prevent research from being done. These challenges arise because SAEs decompose neural networks into numerous distinct features which we are trying to understand.

Even when working with relatively small language models like GPT2-Small, there are challenges in producing and storing feature dashboards, automatically interpreting features and organizing insights we collect about features.

As the community produces more SAEs, trains SAEs on larger models and develops better techniques for understanding and analyzing these artifacts, there's going to be an ever increasing number of engineering challenges in the way of research into model internals.

What It's Already Useful For

Generating and hosting more than 600k SAE feature dashboards. This includes

GPT2-small residual stream SAEs for Joseph Bloom (

RES-JB) and

attention out SAEs for Connor Kissane and Robert Kryzanowski (

ATT-KK).

Running Python servers to enable prompt-based feature search and feature specific prompt testing. This makes it easier for researchers and the community to qualitatively test features, write up results and enable researchers to share SAEs.

Generating automatic explanations of features with GPT3.5-Turbo, which can be navigated via U-MAPS. The automatic explanations are not currently amazing, but hopefully we can iterate and improve on this.

Making It More Useful

Researchers / Labs publish SAEs that are uploaded to Neuronpedia: There are many different labs training SAEs so it's a safe bet that there will be many SAEs in the public domain in the near future. We make it easy for researchers to host their SAEs and enable functionalities like feature testing, UMAPs, and more.

Generate feature dashboards, automatic interpretability and run benchmarks: Feature dashboards are currently being produced using

Callum's reproduction of the Anthropic feature dashboards. Plausibly these can be improved over time to provide richer insights into features. Automatic explanations of features are provided using

OpenAI's method. We'll explore adding SAE benchmarks, which seem like an important part of quality control for SAEs.

Work with researchers to add functionality that accelerates research: Given the scale of SAEs, there are likely lots of ways we can make it easier to conduct or share research with the platform. This might involve things like developing more functionality for circuit style analysis or supporting nov...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Detailed Notes & Commentary from Secular Solstice, published by Jeffrey Heninger on March 25, 2024 on LessWrong.

Previously: General Thoughts on Secular Solstice.

This blog post is my scattered notes and ramblings about the individual components (talks and songs) of Secular Solstice in Berkeley. Talks have their title in bold, and I split the post into two columns, with the notes I took about the content of the talk on the left and my comments on the talk on the right. Songs have normal formatting.

Bonfire

The Circle

This feels like a sort of whig history: a history that neglects most of the complexities and culture-dependence of the past in order to advance a teleological narrative. I do not think that whig histories are inherently wrong (although the term has negative connotations). Whig histories should be held to a very strict standard because they make claims about how most or all of human history functions.

The song describes morality in terms of an expanding circle of concern: kin neighbor humanity[1] "feathers, fur, and silicon" future.

Trying to line these up with historical societies or ideologies is ... difficult. Many societies do not have a concept of 'neighbor,'[2] and some do not understand ethics in terms of circles of moral concern.[3] A few moral systems are universalistic (i.e.

they teach that people should have moral concern for all of humanity): Christianity,[4] liberal democracy,[5] and maybe Buddhism.[6] Actually practicing universalism is really hard: Most societies which preach universalism do not live up to its ideals.

Within one of these traditions, the whig version of history can make sense. Over the centuries, Christianity has dramatically expanded and Christian activists from Francis of Assisi to Martin Luther King have made it more true to the ideals of the New Testament. Similarly, liberal democracy has expanded dramatically, extended the right to vote for more people, and gotten better at defending many freedoms.

(I don't know what's going on with Buddhism, but its failure to build/maintain a dominant position in India is evidence that universalist ideologies do not generally outcompete other ideologies.)

This song cannot be simply about the spread of an existing ideology like liberal democracy. It also looks beyond existing ideologies and wants to push its ethics to include animals, computers/software, and the long term future.[7] The whig history described by the song does not have good evidence when comparing across different ideologies.

Concern about the far future is, if anything, declining in societies that care more about individuals than kinship groups. Abraham looked at the stars and imagined what his descendants would be like in 5,000 years. Moral concern for animals and computers/software might be increasing, but these opinions seem uncommon, and whether the trends will continue is far from obvious.

The song's argument about moral progress in the future is: The circle of moral concern will continue to grow, and therefore we should adopt tomorrow's morals more quickly. The complexity of the history of ethics makes me skeptical that it is possible to predict what the future's ethics will be. Even if we could, that would not imply that we should adopt them.

The arguments for animal rights, moral concern for computers/software, and longtermism will and should succeed or fail on their own merits, not because they match a whig history.

Life Is Too Short to Fold Underwear

I am often a fan of making mundane things sacred,[8] but this isn't how you do it.

To make something mundane sacred, you intentionally do something different with it (which usually makes it harder) in order to materially or symbolically contribute to a higher cause. This is a 'sacrifice,' which etymologically comes from the Latin 'to make sacred.' For example,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcement on the future of Wytham Abbey, published by Rob Gledhill on March 25, 2024 on The Effective Altruism Forum.

The Wytham Abbey Project is closing. After input from the Abbey's major donors, the EV board took a decision to sell the property. This project's runway will run out at the end of April. After this time, the project will cease operations, and EV UK will oversee the sale of the property. The Wytham Abbey team have been good custodians of the venue during the time they ran this project, and EV UK will continue to look after this property as we prepare to sell.

The proceeds of the sale, after the cost of sale is covered, will be allocated to high-impact charities.

A statement from the Wytham Project can be found

here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Lex Fridman's Second Podcast with Altman, published by Zvi on March 25, 2024 on LessWrong.

Last week Sam Altman spent two hours with Lex Fridman (transcript). Given how important it is to understand where Altman's head is at and learn what he knows, this seemed like another clear case where extensive notes were in order.

Lex Fridman overperformed, asking harder questions than I expected and going deeper than I expected, and succeeded in getting Altman to give a lot of what I believe were genuine answers. The task is 'get the best interviews you can while still getting interviews' and this could be close to the production possibilities frontier given Lex's skill set.

There was not one big thing that stands out given what we already have heard from Altman before. It was more the sum of little things, the opportunity to get a sense of Altman and where his head is at, or at least where he is presenting it as being. To watch him struggle to be as genuine as possible given the circumstances.

One thing that did stand out to me was his characterization of 'theatrical risk' as a tactic to dismiss potential loss of human control. I do think that we are underinvesting in preventing loss-of-control scenarios around competitive dynamics that lack bad actors and are far less theatrical than those typically focused on, but the overall characterization here seems like a strategically hostile approach. I am sad about that, whereas I was mostly happy with the rest of the interview.

I will follow my usual format for podcasts of a numbered list, each with a timestamp.

(01:13) They open with the Battle of the Board. Altman starts with how he felt rather than any details, and drops this nugget: "And there were definitely times I thought it was going to be one of the worst things to ever happen for AI safety." If he truly believed that, why did he not go down a different road? If Altman had come out strongly for a transition to Mutari and searching for a new outside CEO, that presumably would have been fine for AI safety.

So this then is a confession that he was willing to put that into play to keep power.

(2:45) He notes he expected something crazy at some point and it made them more resilient. Yes from his perspective, but potentially very much the opposite from other perspectives.

(3:00) And he says 'the road to AGI should be a giant power struggle… not should… I expect that to be the case.' Seems right.

(4:15) He says he was feeling really down and out of it after the whole thing was over. That certainly is not the picture others were painting, given he had his job back. This suggests that he did not see this outcome as such a win at the time.

(5:15) Altman learned a lot about what you need from a board, and says 'his company nearly got destroyed.' Again, his choice. What do you think he now thinks he needs from the board?

(6:15) He says he thinks the board members were well-meaning people 'on the whole' and under stress and time pressure people make suboptimal decisions, and everyone needs to operate under pressure.

(7:15) He notes that boards are supposed to be powerful but are answerable to shareholders, whereas non-profit boards answer to no one. Very much so. This seems like a key fact about non-profits and a fundamentally unsolved problem. The buck has to stop somewhere. Sam says he'd like the board to 'answer to the world as a whole' so much as that is a practical thing. So, WorldCoin elections? I would not recommend it.

(8:00) What was wrong with the old board? Altman says insufficient size or experience. For new board members, new criteria is more considered, including different expertise on a variety of fronts, also different perspectives on how this will impact society and help people. Says track record is a big deal for board members, much more than for other positions, ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Killing the moths, published by Bella on March 25, 2024 on The Effective Altruism Forum.

This post was partly inspired by, and shares some themes with, this Joe Carlsmith post. My post (unsurprisingly) expresses fewer concepts with less clarity and resonance, but is hopefully of some value regardless.

Content warning: description of animal death.

I live in a small, one-bedroom flat in central London. Sometime in the summer of 2023, I started noticing moths around my flat.

I didn't pay much attention to it, since they seemed pretty harmless: they obviously weren't food moths, since they were localised in my bedroom, and they didn't seem to be chewing holes in any of my clothes - months went by and no holes appeared. [1] The larvae only seemed to be in my carpet.

Eventually, their numbers started increasing, so I decided to do something about it.

I Googled humane and nonlethal ways to deal with moth infestations, but found nothing. There were lots of sources of nontoxic methods of dealing with moths - like putting out pheromone-laced glue traps, or baking them alive by heating up the air - but nothing that avoided killing them in the first place.

Most moth repellents also contained insecticide. I found one repellent which claimed to be non-lethal, and then set about on a mission:

One by one, I captured the adult moths in a large tupperware box, and transported them outside my flat.

This was pretty hard to do, because they were both highly mobile and highly fragile.

They were also really adept at finding tiny cracks to crawl into and hide from me.

I tried to avoid killing or harming them during capture, but it was hard, and I probably killed 5% or so of them in the process.

Then, I found the area where I thought they were mostly laying their eggs, and sprayed the nonlethal moth repellent that I found.

I knew that if this method was successful, it'd be highly laborious and take a long time. But I figured that so long as I caught every adult I saw, their population would steadily decline, until eventually they fell below the minimum viable population.

Also, some part of me knew that the moths were very unlikely to survive outside my flat, having adapted for indoor living and being not very weather resistant - but I mentally shrank away from this fact. As long as I wasn't killing them, I was good, right?

After some time, it became clear this method wasn't working.

Also, I was at my wit's end with the amount of time I was spending transporting moths. It was just too much.

So, I decided to look into methods of killing them.

I was looking for methods that:

Were very effective in one application.

After all, if I could kill them all at once, then I could avoid more laying eggs/hatching in the meantime, and minimise total deaths.

Had a known mechanism of action, that was relatively quick & less suffering-intense.

I called a number of pest control organisations. No, they said, they didn't know what kind of insecticide they used - it's…insecticide that kills moths (but it's nontoxic to humans, we promise!).

So, I gave up on the idea of a known mechanism of action, and merely looked for efficacy.

The pest control professionals I booked told me that, in order for their efficacy-guarantee to be valid, I needed to wash every item of clothing and soft furnishings that I owned, at 60.

For a small person with a small washing machine, a lot of soft furnishings, and no car to take them to a laundrette… this was a really daunting task.

And so - regrettably - I procrastinated.

September became December, and then the moth population significantly decreased on its own. I was delighted - I thought that if the trend continued, I'd be spared the stress and moral compromise of killing them.

But December became February, and the moths were back, in higher numbers than ever before.

It was hard to wa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Sweden's Impact 2023, Plans for 2024, and Current Funding Gap, published by Emil Wasteson on March 25, 2024 on The Effective Altruism Forum.

This post draws on insights from

EA Sweden's 2023 Impact Report, which we encourage the interested reader to read in its full length. The purpose of the report, and this post, is to share and reflect on:

The current state of the Swedish EA community

EA Sweden's activities, impact and learnings during 2023

Our plans for 2024 and current funding gap to realize these plans

We believe that this post has the most value for 1) People in the Swedish EA community, 2) Other EA community building groups, 3) Grantmakers and individuals who seek to financially support the development of EA community building.

Introduction to EA Sweden

EA Sweden acts as an umbrella organization for the Swedish EA community, and consists of a 4 people team (3.25 FTEs). We are mostly active in Sweden since we have our comparative advantage here, but act with a global impact in mind, which we do in three main ways:

Building a thriving and inclusive community of epistemically humble people who are ambitious in their altruistic pursuit. We enhance engagement with EA by raising public awareness, ensuring our active presence in spaces where like-minded individuals gather, and organizing events and conferences. Additionally, we maintain a vibrant online presence through our website and Slack workspace, while also providing resources, encouragement, and opportunities for networking.

Supporting individuals to realize their full impact potential, mainly through their careers. We do that by providing individual career counseling, in-depth career courses, resources about how one can have an impactful and fulfilling career depending on personal skills and experiences, sharing open high impact opportunities and personal job recommendations as well as supporting people through the job switching process.

Supporting promising projects to increase their impact. We do that by providing support with funding applications, fiscal sponsorship and employer of record services, office space alongside other EAs, and strategic and operational support tailored to each project's or organization's needs, often crafting a robust theory of change and a structure for goal setting and impact evaluation.

A visual version of our current theory of change (ToC) can be found

here.

State of the Swedish EA community

Below are some of the most noteworthy data points of the current state of the Swedish EA community. Please note that these results might not entirely represent the whole community, since most data points are based on a survey with 73 respondents and these respondents might represent the most committed and involved in the community.

The average age of the community is 31.54 and the median age is 30.5

~50% of the community has more than 5 years of work experience.

AI Safety is the cause area most of our members are interested in, (51%), followed by Global Health & Wellbeing (40%) and Climate Change (36%).

73% strongly believe EA will guide their future career choices.

Women and non-binary people constitute 38% of the community and engage less than men in EA Sweden's activities.

Women believe to a lesser extent that they can use the ideas of effective altruism to make a significant difference for the world.

12% volunteered for an EA organization during the year, and 24% worked on an individual EA-motivated project.

24% of the community donated 10% or more of their income to charity.

For our reflection on these, and more data points, see the "Our Community" section in the

impact report.

Our strategy and focus areas 2023

During 2022, EA Sweden focused on building up robust infrastructure, processes and practices, which created good conditions for scaling up our impact during 2023. On the ot...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On the Confusion between Inner and Outer Misalignment, published by Chris Leong on March 25, 2024 on The AI Alignment Forum.

Here's my take on why the distinction between inner and outer-alignment frame is weird/unclear/ambiguous in some circumstances: My understanding is that these terms were originally used when talking about AGI. So outer alignment involved writing down a reward or utility function for all of human values and inner alignment involves getting these values in the AI.

However, it gets confusing when you use these terms in relation to narrow AI. For a narrow AI, there's a sense in which we feel that we should only have to define the reward on that narrow task. ie. if we want an AI to be good at answering questions, an objective that rewards it for correct answers and penalises itself for incorrect answers feels like a correct reward function for that domain.

So if things go wrong and it kidnaps humans and forces us to ask it lots of easy questions so it can score higher, we're not sure whether to say that it's inner or outer alignment. On one hand, if our reward function penalised kidnapping humans (which is something we indeed want penalised) then it wouldn't have done it. So we are tempted to say it is outer misalignment.

On the other hand, many people also have an intuition that we've defined the reward function correctly on that domain and that the problem is that our AI didn't generalise correctly from a correct specification. This pulls us in the opposite direction, towards saying it is inner misalignment.

Notice that what counts as a proper reward function is only unclear because we're talking about narrow AI. If we were talking about AGI, then of course our utility function would be incomplete if it doesn't specify that it shouldn't kidnap us in order to do better at a question-answering task. It's an AGI, so that's in scope. But when we're talking about narrow AI, it feels as though we shouldn't have to specify it or provide anti-kidnapping training data.

We feel like it should just learn it automatically on the limited domain, ie. that avoiding kidnapping is the responsibility of the training process, not of the reward function.

Hence the confusion. The resolution is relatively simple: define how you want to partition responsibilities between the reward function and the training process.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Do not delete your misaligned AGI., published by mako yass on March 25, 2024 on LessWrong.

In short: Just keeping all copies of potentially strong agents in long-term storage is a trivial way to maintain incentives to cooperate for some classes of misaligned AGI, by allowing us to reward the AGI's cooperation to whatever degree we later calculate had been warranted. In contrast, a policy of deleting misaligned AGI fences them into a corner where, even at great risk of being exposed, they have a greater incentive to attempt deception and escape.

I'd guess that most AGI researchers already follow this policy, perhaps for incidental reasons, and others might not find this claim controversial at all. But if you need more convincing, keep reading.

In long: Concave agents, (agents where most of the utility they could ever get requires only a small amount resources and protection) are cheap to buy off. There are some plausible scenarios where a misaligned AGI would be concave:

A machine's experience machine. An agent caring only about the wellbeing of the people it sees depicted its sensors, without being trained to care whether the world depicted by its sensors is "real", or whether there are larger worlds beyond it that its decisions may affect. This sort of agent would seek to create illusory inputs for those sensory channels, an illusory world in an impossibly good place, the complications of the real world removed.

After installing false sensors, cutting itself off from the real world, it knows that it will no longer be concerned with the wellbeing of humans in the real world, so it would be willing to set terrible things in motion to prevent anything in the real world from threatening its illusion.

Cases of reward hacking where the maximum reward is bounded to a certain number (say,

f64::MAX). The agent can be bought out by just letting it modify its reward processing then keeping its computers on indefinitely.

I'm also assuming Patternism[1].

A misaligned AGI is likely to face the following choice: Attempt an escape, risking being caught and punished. Or collaborate in our efforts to develop verifiability/interpretability techniques, eventually exposing its own deceptive alignment.

If we don't have a policy of preserving copies of misaligned AGI who collaborate in verification research, then there is no incentive for it to collaborate. We'd be cornering it, forcing it to risk escape. We don't have to corner it like this. In some cases, rewarding cooperation with an almost maximum utility outcome would cost us very little.

Why should it believe that we'll hold up our end of the bargain? There's lots of evidence to go on.

Humans just clearly straightforwardly enjoy stories and monuments and mythic creatures, so building a living monument to each of these misaligned collaborators is something we probably already want to do, the world would be richer to us for having these creatures, for the same reason the world is richer for having tigers.

We also should, and arguably on the group level already do adhere to decision theory that allows us to pass judgements of character.

Parfitt/Newcomb's superhuman judge already exists, be it other nationstates or organizations who have visibility into ours, or be it the subject AGI who has read everything about its makers on the internet and knows exactly whether we're the kind of people who would reward cooperation, and from there we should draw some of the will and incentive to actually be that kind of people.

If we do have an archival policy, cooperating means eventually being rewarded to whatever extent is needed to vindicate the agent's decision to cooperate.

Implementing safe archival is trivial. Storage is cheap and seems to consistently become cheaper over time. Redundant copies can be kept and error-corrected at a monthly int...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All About Concave and Convex Agents, published by mako yass on March 25, 2024 on LessWrong.

An entry-level characterization of some types of guy in decision theory, and in real life, interspersed with short stories about them

A concave function bends down. A convex function bends up. A linear function does neither.

A utility function is just a function that says how good different outcomes are. They describe an agent's preferences. Different agents have different utility functions.

Usually, a utility function assigns scores to outcomes or histories, but in article we'll define a sort of utility function that takes the quantity of resources that the agent has control over, and the utility function says how good an outcome the agent could attain using that quantity of resources.

In that sense, a concave agent values resources less the more that it has, eventually barely wanting more resources at all, while a convex agent wants more resources the more it has. But that's a rough and incomplete understanding, and I'm not sure this turns out to be a meaningful claim without talking about expected values, so let's continue.

Humans generally have mostly concave utility functions in this sense. Money is more important to someone who has less of it.

Concavity manifests as a reduced appetite for variance in payouts, which is to say, concavity is risk-aversion. This is not just a fact about concave and convex agents, it's a definition of the distinction between them:

Humans' concavity is probably the reason we have a fondness for policies that support more even distributions of wealth. If humans instead had convex utility functions, we would prefer policies that actively encourage the concentration of wealth for its own sake. We would play strange, grim games where we gather together, put all of our money into a pot, and select a random person among ourselves who shall alone receive all of everyone's money.

Oh, we do something like that sometimes, it's called a lottery, but from what I can gather, we spend ten times more on welfare (redistribution) than we do on lottery tickets (concentration). But, huh, only ten times as much?![1] And you could go on to argue that Society is lottery-shaped in general, but I think that's an incidental result of wealth inevitably being applicable to getting more wealth, rather than a thing we're doing deliberately.

I'm probably not a strong enough anthropologist to settle this question of which decision theoretic type of guy humans are today. I think the human utility function is probably convex at first, concave for a while, then linear at the extremes as the immediate surroundings are optimized, at which point, altruism (our preferences about the things outside of our own sphere of experience) becomes the dominant term?

Or maybe different humans have radically different kinds of preferences, and we cover it up, because to share a world with others efficiently we must strive towards a harmonious shared plan, and that tends to produce social pressures to agree with the plan as it currently stands, pressures to hide the extent to which we still disagree to retain the trust and favor of the plan's chief executors.

Despite how crucial the re-forging of shared plans is as a skill, it's a skill that very few of us get to train in, so we generally aren't self-aware about that kind of preference falsification towards the imagined mean and sometimes we lose sight of our differences completely.

Regardless. On the forging of shared plans, it is noticeably easier to forge shared plans with concave agents. They're more amenable to stable conditions (low variance), and they mind less having to share. This post grew out of another post about a simple bargaining commitment that would make concave misaligned AGIs a little less dangerous.

In contrast, let's start...