Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What countries are worth funding?, published by Ariel Pontes on December 28, 2022 on The Effective Altruism Forum.

I’m the founder of EA Romania and I applied for funds from the EAIF twice this year and got rejected both times without feedback. In the process of hunting for feedback after my first rejection, I kept hearing questions like “what’s the comparative advantage of Romania?” or “what particular groups or cause areas do you want to focus on?”. I was also told by a fund manager that Romania is “not exactly a hub” and therefore there’s a high opportunity cost in funding community building here unless the project is very creative.

At some point during a recent EAGx, I met a Romanian who had been involved in EA for a while and I told her about my whole experience applying for funds. She seemed very empathetic and encouraging and said we should talk more later. The next day when we met her attitude had shifted completely and she started questioning why I wanted to do community building in Romania, and suggesting that if I want to work in this area maybe I could move somewhere with a more vibrant EA movement, where there are more opportunities. Eventually I confronted her about her change of attitude and she basically started saying that Romania is not ready for EA yet. I said that I felt Romania was being discriminated against and she said I had to understand the EA leadership because when Romania joined the EU a lot of uneducated, low-income immigrants flooded Western countries and now they have a bad impression of Romanians.

I couldn’t help but have the impression that she talked to somebody in the EA leadership and concluded that Romania wouldn’t be funded any time soon.

If some countries are unlikely to be funded in general because they’re low priority, I believe this should be public information, and I also think EAIF should be transparent about the criteria they use to prioritize countries. After attending 4 EAG(x)s, I would say that the perception of the average EA is that there are plenty of funds and if you start a new national group it should be very easy to get funded. This perception seems inaccurate, and if the reality was more transparent this would help founders of new national groups have more realistic expectations.

To be clear, I’m not saying that the only or even the main reason why my applications were rejected was that they were coming from Romania. There are other reasons (as I have discussed here and here). It is of course possible that another person with another project would have been funded. It’s hard to know because, again, EAIF doesn’t give feedback. I think this is a big problem, but that’s a topic for another post. In any case, being in Romania was clearly one of the reasons why the application was rejected, as is illustrated by the stories above and as was confirmed in an exchange of emails I had with CEA. This whole experience has made me realize there is very little transparency regarding group funding and groups in general. There are many questions that I think should be easier to find an answer to, for example:

Which countries are at the top of the priority list to be funded?

Which ones are at the bottom?

How much funding does each country get per year for community building?

How much of this money is for salaries?

How many paid community builders does each country have?

How big is each national group?

How old is each national group?

How many applications come from each country?

What percentage of applications is rejected for each country?

Is this information public but buried in some long PDF report? Or is it not public at all? I know that CEA only funds a fixed list of countries that already have mature communities, but does anybody have any idea what criteria the EAIF uses to prioritize countries? Do you think some countries ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: In Defense of Wrapper-Minds, published by Thane Ruthenis on December 28, 2022 on The AI Alignment Forum.

Recently, there's been a strong push against "wrapper-minds" as a framework. It's argued that there's no specific reason to think that all sufficiently advanced agents would format their goals in terms of expected-utility maximization over future trajectories, and that this view predicts severe problems with e. g. Goodharting that just wouldn't show up in reality.

I think these arguments have merit, and the Shard Theory's model definitely seems to correspond to a real stage in agents' value formation.

But I'd like to offer a fairly prosaic argument in favor of wrapper-minds.

Suppose that we have some agent which is being updated by some greedy optimization process (the SGD, evolution, etc.). On average, updates tend to decrease the magnitude of every subsequent update — with each update, the agent requires less and less correction.

We can say that this process optimizes the agent for good performance according to some reward function R, or that it chisels "effective cognition" into that agent according to some rule.

The wrapper-mind argument states that any "sufficiently strong" agent found by this process would:

Have an explicit representation of R inside itself, which it would explicitly pursue.

Pursue only R, at the expense of everything else in the universe.

I'll defend them separately.

Point 1. It's true that explicit R-optimization is suboptimal for many contexts. Consequentialism is slow, and shallow environment-optimized heuristics often perform just as well while being much faster. Other environments can be just "solved" — an arithmetic calculator doesn't need to be a psychotic universe-eater to do its job correctly. And for more complex environments, we can have shard economies, whose collective goals, taken in sum, would be a strong proxy of R.

But suppose that the agent's training environment is very complex and very diverse indeed. Or, equivalently, that it sometimes jumps between many very different and complex environments, and sometimes ends up in entirely novel, never-before-seen situations. We would still want it to do well at R in all such cases. How can we do so?

Just "solving" environments, as with arithmetic, may be impossible or computationally intractable. Systems of heuristics or shard economies also wouldn't be up to the task — whatever proxy goal they're optimizing, there'd be at least one environment where it decouples from R.

It seems almost tautologically true, here, that the only way to keep an agent pointed at R given this setup is to explicitly point it at R. Nothing else would do!

Thus, our optimization algorithm would necessarily find an R-pursuer, if it optimizes an agent for good performance across a sufficiently diverse (set of) environment(s).

Point 2. But why would that agent be shaped to pursue only R, and so strongly that it'll destroy everything else?

This, more or less, also has to do with environment diversity, plus some instrumental convergence.

As the optimization algorithm is shaping our agent, the agent will be placed in environments where it has preciously few resources, or a low probability of scoring well at R (= high probability of receiving a strong update/correction after this episode ends).

Without knowing when such a circumstance would arise, how can we prepare our agent for this?

We can make it optimize for R strongly, as strongly as it can, in fact. Acquire as much resources as possible, spend them on nothing but R-pursuit, minimize uncertainty of scoring well at R, and so on.

Every goal that isn't R would distract from R-pursuit, and therefore lead to failure at some point, and so our optimization algorithm would eventually update such goals away; with update-strength proportional to how distracting a goa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cause prioritisation: Preventing lake Kivu in Africa eruption which could kill two million., published by turchin on December 28, 2022 on The Effective Altruism Forum.

Epistemic status: a preliminary look at a possible cause area

TL;DR: Lake Kivu could erupt and kill 2 million people around it. But we could prevent this by installing oblique pipes, which will slowly and safely release gases and generate energy.

Large lake Kivu on the border between Rwanda and Democratic Republic of the Congo has a lot of gases dissolved near its bottom and it can erupt as lake Nyos did in 1986 when 1700 people were killed, but Kivu eruption could be 2000 times stronger.

A future overturn and gas release from the deep waters of Lake Kivu would result in catastrophe, dwarfing the historically documented lake overturns at the much smaller Lakes Nyos and Monoun. The lives of the approximately two million people who live in the lake basin area would be threatened.

An experimental vent pipe was installed at Lake Nyos in 2001 to remove gas from the deep water, but such a solution for the much larger Lake Kivu would be considerably more expensive. The approximately 510 million metric tons of carbon dioxide in the lake is a little under 2 percent of the amount released annually by human fossil fuel burning. Therefore, the process of releasing it could potentially have costs beyond simply building and operating the system. Wiki

A report Gas emissions from lake Kivu claims that lake Kivu has erupted in the past with a periodicity of 1000 years and could do it again in 100-200 years.

Lake Kivu has 65 cubic miles of methane (this is around 140 Mt and will produce additional methane concentrations increase of 28 ppb if released in the atmosphere, adding to the current level of around 1900 ppb). The lake also has 260 cubic miles of CO2.

Mitigation

Vertical pipes allow slowly extraction of gases from the lake and the collection of methane for energy use. Commercial extraction already started, but it is slow.

There are risks related to gas extraction via pipes:

Risk 1: what if the pipes will destabilise the lake?

Risk 2: releasing CO2 will contribute to global warming. Methane is even worse as a greenhouse gas, but it could be collected with profit. CO2 actually may be used too for fracking and for chemical and food production.

Concerns about these risks are slowing down current methane extraction. However, if the lake erupts, all methane and CO2 will go into the atmosphere and will be equal to several years of the Earth’s emissions, mostly because of short-term methane’s greenhouse effects. But a part of methane will be combusted in eruption.

The strong methane fire may take the form of an explosion which will contribute to gas release and to the devastation around the lake (total methane energy in the lake is around 1 gigaton TNT). However, explosive methane fire will prevent CO2 accumulation on lower grounds near the shores and CO2 suffocating effects will be smaller because gases will mix with the atmosphere quickly.

The project to reduce gases in the lake is ongoing, but its impact is not clear to me: it may not be enough to stop the increase of the gases’ concentration, which is still increasing, but could be enough to create risks of the eruption is something goes wrong (a pin and balloon effect). Especially because it is made for profit which creates an incentive to take higher risks. A much larger and simultaneously safer project is needed to prevent the eruption of the lake.

As gas concentrations rise in Kivu’s depths, so does the risk. Wüest and colleagues found that from 1974 to 2004 the concentration of CO2 increased by 10%. But the bigger concern at Kivu is the methane concentration, which rose 15-20% during the same period. BBC

Oblique pipes

I think that the pipes should be built in a ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider Financial Independence First, published by River on December 27, 2022 on The Effective Altruism Forum.

Over the course of your working life, you will earn some amount of money, spend some amount of money on yourself (and those close to you), and spend some amount of money and time on strangers. EA is about figuring out how to deploy the time and money we spend on strangers effectively. And as a movement of mostly young people, there is a tendency to start devoting time and money to strangers early in our careers. I want to suggest that this is often a mistake – that many of us should front load the money we spend on ourselves, by achieving financial independence first, by which I mean having enough savings that you could live off of it indefinitely, and then earning to give or doing direct work in an EA cause area.

This is the strategy I am pursuing. I got interested in EA a couple of years ago during the pandemic, about the same time I was moving towards a new career in software engineering. I’ve been a software engineer making low six figures for about a year, and I’ve been putting about half my income towards my current expenses, and half towards financial independence1. After I reach financial independence, in something like five years, depending how the markets go, I expect to either earn to give (which I will be well set up for), or find direct work mitigating existential risks. Either way, learning about the various existential risks and mitigation strategies, and the people and organizations working on them, will prepare me to do that when the time comes.

Some of my reasons for pursing financial independence first are selfish. I’m starting my third career in my mid thirties – I want a feeling of material security, and I’m not going to get it from an employer. I want the lower level of stress and anxiety that I expect I will have when I can pay for my rent and groceries on my own, regardless of my employment. I want the freedom to pick a low paying job, or even an unpaid job, over a higher paying but less meaningful job, without worrying about the impact on my lifestyle. I want the freedom to leave a job if the boss I love gets replaced by a boss I hate, without having to ask whether I can afford it. And it’s ok for an EA to make decisions for selfish reasons like this. EA doesn’t have anything to say about how much money anyone should donate or how much time anyone should spend altruistically. EA only tries to answer the question, after time or money has been committed to strangers, how to deploy it effectively.2

Other reasons for pursing financial independence first are altruistic. If your altruism is of the earning to give variety, then achieving financial independence first allows you to donate the entirety of your post-financial-independence salary. Over the course of your life, that means donating more total dollars. If you believe that lives saved in ten or twenty years are as valuable as lives saved today, and that dollars donated in ten or twenty years will not be too much less effective than dollars donated today, then this means doing more good.

If you hope to work directly in an EA cause area, the accounting is much less obvious, but the conclusion may be the same. The early part of your career, particularly the first few years, is when you have the least skills and the most to learn. I spent quite a bit of time in training at my current job, where I was gaining skills but at a net cost to my employer. So the altruistic cost of working outside EA is at its lowest at the beginning of your career. And the altruistic benefit from building skills faster working outside EA may be worthwhile. Often it is easier and faster to build skills in a large, established organization, compared to a newer smaller startup-like organization. EA organizations...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: StrongMinds should not be a top-rated charity (yet), published by Simon M on December 27, 2022 on The Effective Altruism Forum.

GWWC lists StrongMinds as a “top-rated” charity. Their reason for doing so is because Founders Pledge has determined they are cost-effective in their report into mental health.

I could say here, “and that report was written in 2019 - either they should update the report or remove the top rating” and we could all go home. In fact, most of what I’m about to say does consist of “the data really isn’t that clear yet”.

I think the strongest statement I can make (which I doubt StrongMinds would disagree with) is:

“StrongMinds have made limited effort to be quantitative in their self-evaluation, haven’t continued monitoring impact after intervention, haven’t done the research they once claimed they would. They have not been vetted sufficiently to be considered a top charity, and only one independent group has done the work to look into them.”

My key issues are:

Survey data is notoriously noisy and the data here seems to be especially so

There are reasons to be especially doubtful about the accuracy of the survey data (StrongMinds have twice updated their level of uncertainty in their numbers due to SDB)

One of the main models is (to my eyes) off by a factor of ~2 based on an unrealistic assumption about depression (medium confidence)

StrongMinds haven’t continued to publish new data since their trials very early on

StrongMinds seem to be somewhat deceptive about how they market themselves as “effective” (and EA are playing into that by holding them in such high esteem without scrutiny)

What’s going on with the PHQ-9 scores?

In their last four quarterly reports, StrongMinds have reported PHQ-9 reductions of: -13, -13, -13, -13. In their Phase II report, raw scores dropped by a similar amount:

However, their Phase II analysis reports (emphasis theirs):

As evidenced in Table 5, members in the treatment intervention group, on average, had a 4.5 point reduction in their total PHQ-9 Raw Score over the intervention period, as compared to the control populations. Further, there is also a significant visit effect when controlling for group membership. The PHQ-9 Raw Score decreased on average by 0.86 points for a participant for every two groups she attended. Both of these findings are statistically significant.

Founders Pledge’s cost-effectivenes model uses the 4.5 reduction number in their model. (And further reduces this for reasons we’ll get into later).

Based on Phase I and II surveys, it seems to me that a much more cost-effective intervention would be to go around surveying people. I’m not exactly sure what’s going on with the Phase I / Phase II data, but the best I can tell is in Phase I we had a ~7.5 vs ~5.1 PHQ-9 reduction from “being surveyed” vs “being part of the group” and in Phase II we had ~5.1 vs ~4.5 PHQ-9 reduction from “being surveyed” vs “being part of the group”. For what it’s worth, I don’t believe this is likely the case, I think it’s just a strong sign that the survey mechanism being used is inadequate to determine what is going on.

There are a number of potential reasons we might expect to see such large improvements in the mental health of the control group (as well as the treatment group).

Mean-reversion - StrongMinds happens to sample people at a low ebb and so the progression of time leads their mental health to improve of its own accord

“People in targeted communities often incorrectly believe that StrongMinds will provide them with cash or material goods and may therefore provide misleading responses when being diagnosed.” Potential participants fake their initial scores in order to get into the program (either because they (mistakenly) think there is some material benefit to being in the program or because they think it makes...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Air-gapping evaluation and support, published by Ryan Kidd on December 26, 2022 on LessWrong.

This blog post was written fast to communicate a concept I think is important. I may edit this post for legibility later.

I think evaluation and support mechanisms should be somewhat “air-gapped,” or isolated, in their information-gathering and decision-making processes. The incentives of optimal evaluators (to critique flaws) seem to run counter to the incentives of optimal supporters (to improve flaws). Individuals who might benefit from support may be discouraged from seeking it by fear of harsher evaluation if their private struggles are shared with evaluators. Evaluators who want to provide support may worry about compromising their evaluation ability if they make inconsistent exceptions. To optimally evaluate and support individuals, I believe that it is necessary to establish and declare appropriate information air gaps between different ecosystem roles.

Evaluation mechanisms, such as academic exams, job interviews, grant applications, and the peer review process, aim to critique an individual or their output. To be maximally effective, evaluation mechanisms should be somewhat adversarial to identify flaws and provide useful criticism. It is in the interests of evaluators to have access to all information about a candidate; however, it is not always in the candidate’s best interests to share all information that might affect the evaluation. It is also in the interests of evaluators for candidates to get access to all the support they need to improve.

If an attribute that disadvantages a job candidate (e.g., a disability) is protected by antidiscrimination law, an evaluator may be biased against the attribute either unconsciously or on the basis that it might genuinely reduce performance. Of course, evaluators should be required to ignore or overcome biases against protected attributes, but this “patch” may break or fail to convince candidates to divulge all evaluation-relevant information. Additionally, in the case that a candidate shares sensitive information with an evaluator, they might not have the appropriate resources or experience to provide beneficial support. Thus, an independent support role might benefit the interests of evaluators.

Support mechanisms, such as psychological counseling, legal support, and drug rehabilitation programs, aim to help individuals overcome their personal challenges, often to improve their chances at evaluation. To be maximally effective, support mechanisms should encourage candidates to divulge highly personal and unflattering information. It is in the interests of supporters to guarantee that sensitive information that could affect evaluation is not shared with evaluators (barring information that might prevent harm to others). Generally, the more information a supporter can access, the better support they can provide.

If a candidate has a secret challenge (e.g., a drug problem) that might rightly bias an evaluator (e.g., an employer), they might be motivated not to seek support for this problem if the supporter (e.g., a psychologist or support group) cannot guarantee this information will be kept private. Candidates can be told that evaluators will not punish them for revealing sensitive information, but this policy seems difficult to enforce convincingly. Thus, it is in the interests of supporters to advertise and uphold confidentiality.

A consequentialist who wants to both filter out poor candidates and benefit candidates who could improve from support will have to strike a balance between the competing incentives of evaluation and support. One particularly effective mechanism used in society is establishing and advertising independent, air-gapped evaluators and supporters. I think the EA and AI safety communities could benefi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on my 5-month alignment upskilling grant, published by Jay Bailey on December 27, 2022 on LessWrong.

Five months ago, I received a grant from the Long Term Future Fund to upskill in AI alignment. As of a few days ago, I was invited to Berkeley for two months of full-time alignment research under Owain Evans’s stream in the SERIMATS program. This post is about how I got there.

The post is partially a retrospective for myself, and partially a sketch of the path I took so that others can decide if it’s right for them. This post was written relatively quickly - I’m happy to answer more questions via PM or in the comments.

Summary

I was a software engineer for 3-4 years with little to no ML experience before I was accepted for my grant.

I did a bunch of stuff around fundamental ML maths, understanding RL and transformers, and improving my alignment understanding.

Having tutors, getting feedback on my plan early on, and being able to pivot as I went were all very useful for not getting stuck doing stuff that was no longer useful.

I probably wouldn’t have gotten into SERIMATS without that ability to pivot midway through.

After SERIMATS, I want to finish off the last part of the grant while I find work, then start work as a Research Engineer at an alignment organisation.

If in doubt, put in an application!

My Background

My background is more professional and less academic than most. Until I was 23, I didn’t do much of anything - then I got a Bachelor of Computer Science from a university ranked around 1,000th, with little maths and no intent to study ML at all, let alone alignment. It was known for strong graduate employment though, so I went straight into industry from there. I had 3.5 years of software engineering experience (1.5 at Amazon, 2 as a senior engineer at other jobs) before applying for the LTFF grant. I had no ML experience at the time, besides being halfway through doing the fast.ai course in my spare time.

Not going to lie, seeing how many Top-20 university PhD students I was sharing my cohort with (At least three!) was a tad intimidating - but I made it in the end, so industry experience clearly has a role to play as well.

Grant

The details of the grant are one of the main reasons I wrote this - I’ve been asked for 1:1’s and details on this at least three times in the last six months, and if you get asked something from at least three different people, it might be worth writing it up and sharing it around.

Firstly, the process. Applying for the grant is pretty painless. As long as you have a learning plan already in place, the official guidance is to take 1-2 hours on it. I took a bit longer, polishing it more than required. I later found out my plan was more detailed than it probably had to be. In retrospect, I think my level of detail was good, but I spent too much time editing. AI Safety Support helped me with administration. The main benefit that I got from it was that the tutoring and compute money was tax free (since I didn’t get the money personally, rather I used a card they provided me) and I didn’t have to worry about tax withholding throughout the year.

Secondly, the money. I agonized over how much money to ask for. This took me days. I asked myself how much I really needed, then I asked myself how much I would actually accept gladly with no regrets, then I balked at those numbers, even knowing that most people ask for too little, not too much. I still balk at the numbers, to be honest, but it would have been so much easier to write this if I had other grants to go off. So, in the interest of transparency and hopefully preventing someone else going through the same level of anguish, I’m sharing the full text of my grant request, including money requested (in Australian dollars, but you can always convert it) here. Personal embarrassmen...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Things that can kill you quickly: What everyone should know about first aid, published by jasoncrawford on December 27, 2022 on LessWrong.

There are things that kill you instantly, like a bullet to the head or a fall from twenty stories. First aid can’t help you there. There are also things that kill you relatively slowly, like a bacterial infection. If you have even hours to live, you can get to the emergency room.

But there is a small class of things that will kill you in minutes unless someone comes to the rescue. There isn’t time to get to a hospital, there isn’t even time for help to arrive in an ambulance. There is only time for someone already on the scene to provide emergency treatment that either solves the problem, or stabilizes you until help arrives. Here, first aid can be the difference between life and death.

Not long ago I became a father. Being responsible for the life of someone so helpless and vulnerable spurred me to finally take first aid training, including CPR. Here’s what I learned from that experience, and what I think everyone should know about first aid.

What most of the things that kill you quickly have in common is that oxygen can’t get to your cells. If you are choking, oxygen can’t get in. If your heart stops beating, blood doesn’t flow. If you have a severe wound, you’re losing that blood rapidly. If any link in the respiratory-circulatory chain is broken, your cells are starved for oxygen and you have minutes to live.

The key first aid skills follow from this: CPR manually substitutes for heart and lung action; the Heimlich maneuver expels an object from the airway; a tourniquet stops life-threatening bleeding (on an extremity, at least—if the wound is elsewhere, there is a different technique, known as packing the wound).

The basic skills are remarkably simple. The course that I took was only a few hours of online instruction, followed by about an hour of in-person demonstration and practice with dummy patients. And I went through a lot of the optional material, including things like stroke, fainting, and jellyfish stings. I’m sure I’m nowhere near as good someone with more professional training or experience, but an introductory course is not daunting.

The most important thing I learned is that if you find yourself in an emergency situation, it is better to do almost anything rather than nothing. Again, if someone stops breathing for any reason, they have only minutes to live. They are dead by default, unless someone intervenes. There is very little you can do to them that is worse than cutting off their oxygen.

In fact, it is probably better to attempt CPR or the Heimlich maneuver than to do nothing, even if you have never been trained and are only guessing, or mimicking what you have seen on television. The skills were fairly unsurprising to me and were consistent with what I expected prior to training. This does not mean that you don’t need to bother with the training, and of course if someone trained is on hand then let them take over. But don’t let the bystander effect paralyze you if someone’s life is ever in your hands.

In fact, the American Heart Association promotes a form of CPR called “hands-only,” in which you only do chest compressions, without giving breaths mouth-to-mouth. Their instructions for this are: “push hard and fast in the center of the chest.” That’s about it. if you only know that, you can do better than nothing.

Similarly, if you can find an AED machine (automated external defibrillator), you do not need training to use it. The instructions are literally: open it and follow the prompts. The parts are clearly labeled, and there is a voice recording that walks you through every step of the process.

In the end, the biggest thing I gained was the confidence to act.

I made an Anki flashcard deck for the course a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why The Focus on Expected Utility Maximisers?, published by DragonGod on December 27, 2022 on LessWrong.

Epistemic Status

Unsure, partially noticing my own confusion. Hoping Cunningham's Law can help resolve it.

Confusions About Arguments From Expected Utility Maximisation

Some MIRI people (e.g. Rob Bensinger) still highlight EU maximisers as the paradigm case for existentially dangerous AI systems. I'm confused by this for a few reasons:

Not all consequentialist/goal directed systems are expected utility maximisers

E.g. humans

Some recent developments make me sceptical that VNM expected utility are a natural form of generally intelligent systems

Wentworth's subagents provide a model for inexploitable agents that don't maximise a simple unitary utility function

The main requirement for subagents to be a better model than unitary agents is path dependent preferences or hidden state variables

Alternatively, subagents natively admit partial orders over preferences

If I'm not mistaken, utility functions seem to require a (static) total order over preferences

This might be a very unreasonable ask; it does not seem to describe humans, animals, or even existing sophisticated AI systems

I think the strongest implication of Wentworth's subagents is that expected utility maximisation is not the limit or idealised form of agency

Shard Theory suggests that trained agents (via reinforcement learning) form value "shards"

Values are inherently "contextual influences on decision making"

Hence agents do not have a static total order over preferences (what a utility function implies) as what preferences are active depends on the context

Preferences are dynamic (change over time), and the ordering of them is not necessarily total

This explains many of the observed inconsistencies in human decision making

A multitude of value shards do not admit analysis as a simple unitary utility function

Reward is not the optimisation target

Reinforcement learning does not select for reward maximising agents in general

Reward "upweight certain kinds of actions in certain kinds of situations, and therefore reward chisels cognitive grooves into agents"

I'm thus very sceptical that systems optimised via reinforcement learning to be capable in a wide variety of domains/tasks converge towards maximising a simple expected utility function

I am not aware that humanity actually knows training paradigms that select for expected utility maximisers

Our most capable/economically transformative AI systems are not agents and are definitely not expected utility maximisers

Such systems might converge towards general intelligence under sufficiently strong selection pressure but do not become expected utility maximisers in the limit

The do not become agents in the limit and expected utility maximisation is a particular kind of agency

I am seriously entertaining the hypothesis that expected utility maximisation is anti-natural to selection for general intelligence

I'm not under the impression that systems optimised by stochastic gradient descent to be generally capable optimisers converge towards expected utility maximisers

The generally capable optimisers produced by evolution aren't expected utility maximisers

I'm starting to suspect that "search like" optimisation processes for general intelligence do not in general converge towards expected utility maximisers

I.e. it may end up being the case that the only way to create a generally capable expected utility maximiser is to explicitly design one

And we do not know how to design capable optimisers for rich environments

We can't even design an image classifier

I currently disbelieve the strong orthogonality thesis translated to practice

While it may be in theory feasible to design systems at any intelligence level with any final goal

In practice, we cannot design capab...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I have thousands of copies of HPMOR in Russian. How to use them with the most impact?, published by Samin on December 27, 2022 on The Effective Altruism Forum.

As a result of a crowdfunding campaign a couple of years ago, I printed 21k copies of HPMOR. 11k of those went to the crowdfunding participants. I need ideas for how to use the ones that are left with the most impact.

Over the last few weeks, I've sent 400 copies to winners of IMO, IOI, and other international and Russian olympiads in math, computer science, biology, etc. (and also bought and sent copies of Human Compatible in Russian to about 250 of them who requested it as well). Over the next month, we'll get the media to post about the opportunity for winners of certain olympiads to get free books. I estimate that we can get maybe twice as many (800-1000) impressive students to fill out a form for getting HPMOR and get maybe up to 30k people who follow the same media to read HPMOR online if everything goes well (uncertain estimates, from the previous experience).

The theory of change behind sending the books to winners of olympiads is that people with high potential read HPMOR and share it with friends, get some EA-adjacent mindset and values from it, and then get introduced to EA in emails about 80k Hours (which is being translated into Russian) and other EA and cause-specific content, and start participating in the EA community and get more into specific cause areas that interest them. The anecdotal evidence is that most of the Russian EAs, many of whom now work full-time at EA orgs or as independent researchers, got into EA after reading HPMOR and then the LW sequences.

It's mostly impossible to give the books for donations to EA charities because Russians can't transfer money to EA charities due to sanctions.

Delivering the copies costs around $5-10 in Russia and $40-100 outside Russia.

There are some other ideas, but nothing that lets us spend thousands of books effectively, so we need more.

So, any ideas on how to spend 8-9k of HPMOR copies in Russian?

HPMOR has endorsements from a bunch of Russians important to different audiences. That includes some of the most famous science communicators, computer science professors, literature critics, a person training the Moscow team for math competitions, etc.; currently, there are news media and popular science resources with millions of followers managed by people who've read HPMOR or heard of it in a positive way and who I can talk to. After the war started, some of them were banned, but that doesn't affect the millions of followers they have on, e.g., Telegram.

Initially, we planned first to translate and print 80k and then give HPMOR to people together with copies of 80k, but there's now time pressure due to uncertainty over the future legal status of HPMOR in Russia. Recently, a law came into effect that prohibited all sorts of mentions of LGBTQ in books, and it's not obvious whether HPMOR is at risk, so it's better to spend a lot of books now.

E.g., there's the Yandex School of Data Analysis, one of the top places for students in Russia to get into machine learning; we hope to be able to get them to give hundreds of copies of HPMOR to students who've completed a machine learning course and give us their emails. That might result in more people familiar with the alignment problem in positions where they can help prevent an AI arms race started by Russia.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.