Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link] Why I’m excited about AI-assisted human feedback, published by Jan Leike on April 6, 2022 on The AI Alignment Forum.

This is a link post for

I'm writing a sequence of posts on the approach to alignment I'm currently most excited about. This first post argues for recursive reward modeling and the problem it's meant to address (scaling RLHF to tasks that are hard to evaluate).

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Call For Distillers, published by johnswentworth on April 6, 2022 on The Effective Altruism Forum.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.[Additional note: this post was crossposted with the author's permission, but not by the author. The author may not see or respond to comments on this post. You can see the original post here.]

Many technical alignment researchers are bad-to-mediocre at writing up their ideas and results in a form intelligible to other people. And even for those who are reasonably good at it, writing up a good intuitive explanation still takes a lot of work, and that work lengthens the turn-time on publishing new results. For instance, a couple months ago I wrote a post which formalized the idea of abstractions as redundant information, and argued that it’s equivalent to abstractions as information relevant at a distance. That post came out about two months after I had the rough math worked out, because it took a lot of work to explain it decently - and I don’t even think the end result was all that good an explanation! And I still don’t have a post which explains well why that result is interesting.

I think there’s a lot of potential space in the field for people who are good at figuring out what other researchers’ math is saying intuitively, and why it’s interesting, and then communicating that clearly - i.e. the skill of distillation. This post will briefly sketch out what two kinds of distillation roles might look like, what skills are needed, and talk about how one might get started in such a role.

Two Distiller Roles

The two types of distiller role I’ll sketch are:

“Independent” distiller: someone who works independently, understanding work published by other researchers and producing distillations of that work.

“Adjunct” distiller: someone who works directly with one researcher or a small team, producing regular write-ups of what the person/team is thinking about and why.

These two roles add value in slightly different ways.

An independent distiller’s main value-adds are:

Explaining the motivation and intended applications

Coming up with new examples

Boiling down the “key intuitive story” behind an argument

Showing how the intuitive story fits into the context of the intended applications

I expect the ability to come up with novel examples and boil down the core intuitive story behind a bunch of math are the rate-limiting skills here.

Rob Miles is a good example of an existing independent distiller in the field. He makes YouTube videos intuitively explaining various technical results and arguments. Rob’s work is aimed somewhat more at a popular audience than what I have in mind, but it’s nonetheless been useful for people in the field.

I expect an adjunct distiller’s main value-adds are:

Writing up explanations, examples, and intuitions, similar to the independent distiller

Saving time for the technical researcher/team; allow more specialization

Providing more external visibility/legibility into the research process and motivation

Accelerating the research process directly by coming up with good examples and intuitive explanations

I expect finding a researcher/team to work with is the rate-limiting step to this sort of work.

Mark Xu is a good example of an existing adjunct distiller. He’s worked with both Evan Hubinger and Paul Christiano, and has written up decent distillations of some of their thoughts. I believe Mark did this with the aim of later doing technical research himself, rather than mostly being a distiller. That is a pretty good strategy and I expect it to be a common pathway, though naturally I expect people who aim to specialize in distillation long-term will end up better at distillation.

What Kind Of Skills Are Needed?

I expect the key rate-limiting skills are:

Ability to indep...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Intro to brain-like-AGI safety] 11. Safety ≠ alignment (but they’re close!), published by Steve Byrnes on April 6, 2022 on The AI Alignment Forum.

Part of the “Intro to brain-like-AGI safety” post series.

(If you’re already an AGI safety expert, you can probably skip this short post—I don’t think anything here is new, or too specific to brain-like AGIs.)

11.1 Post summary / Table of contents

In the previous post, I talked about “the alignment problem” for brain-like AGIs. Two points are worth emphasizing: (1) the alignment problem for brain-like AGIs is currently unsolved (just like the alignment problem for any other type of AGI), and (2) solving it would be a giant leap towards AGI safety.

That said, “solving AGI alignment” is not exactly the same as “solving AGI safety”. This post is about how the two may come apart, at least in principle.

As a reminder, here’s the terminology:

“AGI alignment” (previous post) means that an AGI is trying to do things that the AGI designer had intended for it to be trying to do. This notion only makes sense for algorithms that are “trying” to do something in the first place. What does “trying” mean in general? Hoo boy, that’s a whole can of worms. Is a sorting algorithm “trying” to sort numbers? Or is it merely sorting them?? I don’t want to go there. For this series, it’s easy. The “brain-like AGIs” that I’m talking about can definitely “try” to do things, in exactly the same common-sense way that a human can “try” to get out of debt.

“AGI safety” (Post #1) is about what the AGI actually does, not what it’s trying to do. AGI safety means that the AGI’s actual behavior does not lead to a “catastrophic accident”, as judged by the AGI’s own designers.

Thus, these are two different things. And my goal in this post is to describe how they may come apart:

Section 11.2 is “alignment without safety”. A possible story would be: “I wanted my AGI to mop the floor, and my AGI did in fact try to mop my floor, but, well, it’s a bit clumsy, and it seems to have accidentally vaporized the entire universe into pure nothingness.”

Section 11.3 is “safety without alignment”. A possible story would be: “I don’t really know what my AGI is trying to do, but it is constrained, such that it can’t do anything catastrophically dangerous even if it wanted to.” I’ll go through four special cases of safety-without-alignment: “boxing”, “data curation”, “impact limits”, and “non-agentic AI”.

To skip to the final answer, my takeaway is that, although it is not technically correct to say “AGI alignment is necessary and sufficient for AGI safety”, it’s damn close to correct, at least in the brain-like AGIs we’re talking about in this series.

11.2 Alignment without safety?

This is the case where an AGI is aligned (i.e., trying to do things that its designers had intended for it to try to do), but still causes catastrophic accidents. How?

One example: maybe, as designers, we didn’t think carefully about what we had intended for the AGI to do. John Wentworth gives a hypothetical example here: humans ask the AGI for a nuclear fusion power plant design, but they neglect to ask the follow-up question of whether the same design makes it much easier to make nuclear weapons.

Another example: maybe the AGI is trying to do what we had intended for it to try to do, but it screws up. For example, maybe we ask the AGI to build a new better successor AGI, that is still well-behaved and aligned. But the AGI messes up. It makes a successor AGI with the wrong motivations, and the successor gets out of control and kills everyone.

I don’t have much to say in general about alignment-without-safety. But I guess I’m modestly optimistic that, if we solve the alignment problem, then we can muddle our way through to safety. After all, if we solve the alignment problem, then we’ll be able to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PaLM in "Extrapolating GPT-N performance", published by Lukas Finnveden on April 6, 2022 on The AI Alignment Forum.

A bit more than a year ago, I wrote Extrapolating GPT-N performance, trying to predict how fast scaled-up models would improve on a few benchmarks. Google Research just released a paper reporting benchmark performance of PaLM: a 540B parameter model trained on 780B tokens. This post contains an updated version of one of the old graphs, where I've added PaLM's performance.

You can read the original post for the full details, but as a quick explainer of how to read the graph:

Each dot represents a particular model’s performance on a particular benchmark (taken from the GPT-3 paper). Color represents benchmark; y-position represents benchmark performance (normalized between random and my guess of maximum possible performance); and the x-position represents loss on GPT-3’s validation set.

The x-axis is also annotated with the required size+data that you’d need to achieve that loss (if you trained to convergence) according to the original scaling laws paper.

(After the point at which OpenAI’s scaling-laws predicts that you’d only have to train on each data point once, it is also annotated with the amount of FLOP you’d need to train on each data point once.)

The crosses represent Google’s new language model, PaLM. Since they do not report loss, I infer what position it should have from the size and amount of data it was trained on. (The relationship between parameters and data is very similar to what OpenAI's scaling laws recommended.)

The sigmoid lines are only fit to the GPT-3 dots, not the PaLM crosses.

Some reflections:

SuperGLUE is above trend (and happens to appear on the Cloze & completion trendline — this is totally accidental). ANLI sees impressive gains, though nothing too surprising given ~sigmoidal scaling.

Common sense reasoning + Reading tasks are right on trend.

Cloze & completion, Winograd, and Q&A are below trend.

The average is amusingly right-on-trend, though I wouldn’t put a lot of weight on that, given that the weighting of the different benchmarks is totally arbitrary.

(The current set-up gives equal weight to everything — despite e.g. SuperGLUE being a much more robust benchmark than Winograd.)

And a few caveats:

The GPT-3 paper was published 2 years ago. I would’ve expected some algorithmic progress by now — and the PaLM authors claim to have made some improvements. Accounting for that, this looks more like it’s below-trend.

The graph relies a lot on the original scaling laws paper. This is pretty shaky, given that the Chinchilla paper now says that the old scaling laws are sub-optimal.

The graph also relies on a number of other hunches, like what counts as maximum performance for each benchmark. And using sigmoids in particular was never that well-motivated.

Since GPT-3 was developed, people have created much harder benchmarks, like MMLU and Big-bench. I expect these to be more informative than the ones in the graph above, since there’s a limit on how much information you can get from benchmarks that are already almost solved.

On the graph, it looks like the difference between GPT-3 (the rightmost dots) and PaLM is a lot bigger than the difference between GPT-3 and the previous dot. However, the log-distance in compute is actually bigger between the latter than between the former. The reason for this discrepancy is that GPT-3 slightly underperformed the scaling laws, and therefore appears relatively more towards the left than you would have expected from the compute invested in it.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't die with dignity; instead play to your outs, published by landfish on April 6, 2022 on LessWrong.

In MIRI announces new "Death With Dignity" strategy, Eliezer makes a number of claims. As I understood them, they are:

Humanity is very likely doomed, because AI alignment is extremely hard, and no one has a strategy that could plausibly work (or is on track to find one)

The right thing to do is maximize the log odds of humanity’s survival

The way to do this to adopt a mindset Eliezer calls “dying with dignity”

Assuming 1) and 2) are true, I’m not convinced the right mindset to adopt is “dying with dignity”. I don’t like it as a framing of the problem. Personally, it sounds demotivating and depressing, and not the thing that will help me maximize the log odds of success. I recognize that not everyone has the same mental setup as me, and some people might find the dying with dignity framing helpful. If it works for you, great, but I think there is a better problem framing that will work for a wider variety of mental setups, and I call it playing to your outs.

This problem framing arose out of a discussion about the dying with dignity post with Eli Rose and Seraphina Nix. Eli pointed out that in Magic the Gathering, there is a useful strategy to adopt when you’re likely to lose, called playing to your outs. Basically, there are times when you’re losing and it’s tempting to try to minimize your losses and make what gains you can. But if you’re sufficiently behind, this will almost always fail. Instead you need to think ahead to what cards you can draw, or what cards your opponent might fail to draw, that will afford you a chance to win. Then plan for one of those scenarios happening, even if it’s quite unlikely.If 1) is true, then we are losing badly in our game of AI existential risk. It’s hard to stay motivated when things are dire, and I claim our stance should be like that of a good Magic player. Playing carefully, choosing strategies that have a shot at winning, keeping our eye on the prize.

Being ready to win if the opportunity presents itself. I don’t think this is a different strategy than the one Eliezer is proposing; I think the strategy is identical. But I think there’s an important mindset difference, even if it’s just in the name, and that mindset difference matters.I am not saying to condition upon some particular unlikely clever scheme working. The actual world is not a Magic game; it’s much more complex. So you can’t just wait for a particular card to come up in your deck. Playing to your outs might mean spending most of your time looking for outs, e.g. searching for alignment strategies that might work, trying to find extremely talented individuals who can do good research, or searching for AI governance strategies that might give us more time and reduce arms race dynamics. You have to identify a real potential out before you can play to it.

I think Eliezer proposes the dying with dignity framing in order to 1) stay motivated while 2) mitigating common epistemological errors when facing extremely bad odds. He writes:

When Earth's prospects are that far underwater in the basement of the logistic success curve, it may be hard to feel motivated about continuing to fight, since doubling our chances of survival will only take them from 0% to 0%.

That's why I would suggest reframing the problem - especially on an emotional level - to helping humanity die with dignity, or rather, since even this goal is realistically unattainable at this point, die with slightly more dignity than would otherwise be counterfactually obtained.

Obviously this is somewhat tongue in cheek, and possibly overstated for April 1st, but if it’s overstated I think it’s only slightly overstated given what else Eliezer has said about probability of doom. However, I think he’s serious in ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Vultures Are Circling, published by CitizenTen on April 6, 2022 on The Effective Altruism Forum.

It might not have shown up on your radar but the funding situation for EA is currently insane. Like bananas, jumping off the wall, insane. Especially with regards to young people. I personally know of 16 year olds getting more money than the median American salary, and of 21 year olds getting six to seven figure grants. And this isn’t to knock either of those things. There’s really well thought out reasons why this makes sense. And generally I’d even advocate for more of this crazy risk taking. Normal institutions are extremely risk averse and it’s nice to see EA buck the trend.

But here’s the thing. The message is out. There’s easy money to be had. And the vultures are coming. On many internet circles, there’s been a worrying tone. “You should apply for [insert EA grant], all I had to do was pretend to care about x, and I got $$!” Or, “I’m not even an EA, but I can pretend, as getting a 10k grant is a good instrumental goal towards [insert-poor-life-goals-here]” Or, “Did you hear that a 16 year old got x amount of money? That’s ridiculous! I thought EA’s were supposed to be effective!” Or, “All you have to do is mouth the words community building and you get thrown bags of money.” Basically, the sharp increase in rewards has led the number of people who are optimizing for the wrong thing to go up. Hello Goodhart. Instead of the intrinsically motivated EA, we’re beginning to get the resume padders, the career optimizers, and the type of person that cheats on the entry test for preschool in the hopes of getting their child into a better college.

I’ve already heard of discord servers springing up centered around gaming the admission process for grants. And it’s not without reason. The Atlas Fellowship is offering a 50k, no strings attached scholarship. If you want people to throw out any hesitation around cheating the system, having a carrot that’s larger than most adult’s yearly income will do that. TLDR: People are going to begin to optimize really hard around showing [EA grants] what they are thought they want to see. This will lead to just less impactful grants for helping people, and generally less chance of right handed tail successes.

So what to do? I’d like to note that some of the knee jerk reactions when hearing of the problem are examples of things not to do.

Tightening up and becoming more stringent on what is funded. This is a failure mode. The rationale for giving out high risk grants stands and hasn’t changed. So decreasing the riskiness of the grants just means we backslide into becoming like any other risk averse institution.

Increasing purity tests. Are you an EA, or are you an EA. Making people jump through more hoops to prove their alignment with core EA values is a terrible idea. Not only would you get evaporate cooling, you’d get a worse community. As a community that values good epidemics, having a purity test on whether or not this person agrees with the EA consensus on [insert topic here] is a death blow to the current very good MO.

Fund less young people. Giving 16 year olds huge chunks of money with no oversight is a bad PR story waiting to happen. So people will argue that we should stop doing that. First, I'd like to point to the first bullet point. Second, Alexander Hamilton ran a business when he was 16, sailed on a ship from the Caribbean when he was 17 and founded the country that most reading this post live in when he was 20. So not funding young people means this type of talent and potential is wasted. Let's not do that.

Finally, I’d like to note that this problem has yet to become an actual problem. It's just a whisper of what's to (maybe) come. It still happens to be the case that the intrinsically motivated EA’s far, far out number ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Atlantic article on effective Ukraine aid, published by iporophiry on April 5, 2022 on The Effective Altruism Forum.

This is an article from the Atlantic from a few days ago. The writer takes on an EA lens, and seems to understand it well. Includes an interview with Pablo Melchor, co-founder and president of Ayuda Efectiva (in Spain). I'd recommend it, and it seems like a good thing to send to non-EA friends.Some excerpts:

When I asked Pablo Melchor, the president of an effective-altruism nonprofit in Spain called Ayuda Efectiva, if he’d altered his usual giving in response to the war, he said he hadn’t. “This has nothing to do with how much I care (a lot!) but rather with how much I think my donation could achieve,” he wrote to me in an email. “I know the Ukrainian crisis is going to receive a huge amount of resources and any additional amounts will make a much greater difference in now forgotten tragedies,” such as the hundreds of thousands of children who die from malaria each year.

This is not to pit one tragedy against another, but just to note that the donations a cause receives are strongly dictated by media coverage. An effective altruist might also try to identify less publicized effects of the war itself. For instance, Chris Szulc, a member of Effective Altruism Poland, told me that he was interested in finding ways to provide Ukrainian refugees with mental-health services. Still other donors might take a bigger-picture perspective and fund initiatives that aim to reduce the risk of nuclear war and of large-scale conflicts.

But say you want to do something to try to help Ukrainians in this moment. What’s the best thing to do? For starters, donate your money and time, not physical stuff. “It’s better to send money to trusted organizations, because they can buy blankets or coats or whatever is needed at much cheaper prices,” Melinda Haring, the deputy director of the Atlantic Council’s Eurasia Center, told me. “They can get a brand-new coat for less than it costs for you to ship the coat to Ukraine.”

And

(For additional ideas, you could check out suggestions from Effective Altruism Poland and from Vox’s Kelsey Piper, who highlights, among other organizations, independent Russian media outlets.)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Recollection of How This All Got Started, published by G Gordon Worley III on April 6, 2022 on LessWrong.

I've told this story to various folks one-on-one. They usually want to know something like "how did you get into AI safety" or "how did you get into EA". And although I expect to keep telling it one-off, I'll write it down for those of you I'll never get to meet.

Why should you read it? Because my story is partially the story of how all this got started: LessWrong, AI safety, EA, and so on. I'm not saying it's the whole story, I'm just saying I've been hanging around what I think of as this community for over 20 years now, so my story is one facet of how we got here.

My story starts in the late 90s. I was hanging around this mailserv called "extropians". How I ended up there I don't recall, but my best guess is I wandered over directly or indirectly from something I found on Slashdot. I'm pretty sure nanobots or cryonics were involved.

This guy Eli-something-or-other wrote some posts about how most people are unable to think coherently about future tech that's too many "shock levels" above what they already know about. He started a mailing list that split off from extropians to talk coherently about the most shocking stuff: the Shock Level 4 stuff.

Thus the community began to come into existence with the creation of the SL4 mailserv.

We talked about all kinds of wild ideas on there, but the big one was AGI. An important topic in those days was figuring out if AGI was default safe or dangerous. Eliezer said default dangerous and made lots of arguments that this was the case. I was eventually convinced. Some remain unconvinced to this day.

Big names I remember from this time include Hal Finney, Christine Peterson, Max Moore, Eric Drexler, Ben Goertzel, and Robin Hanson. I'm not sure who was on the list and who wasn't. I also remember some other names but we were not among the big players.

The list started to die down after a couple years, but around this time we started hanging out on IRC. It was a lot of fun, but a huge time suck. This helped bring the community more together in real time, but everyone was still spread out. Somewhere along the way the Singularity Institute started.

Around this time Eliezer started to get really into heuristics and biases and Bayes' Theorem, claiming it was the secret of the universe or something. After I studied a bunch of information theory and thermodynamics I basically believed it, although I still prefer to think in the cybernetic terms I picked up from my engineering education. We also all got interested in quantum physics and evolutionary psychology and some other stuff.

Eliezer was really on about building Friendly AI and had been since about the start of the SL4 mailing list. What that meant got clearer over time. What also got clearer is that we were all too stupid to help even though many of us were convinced AGI was going to be dangerous (I remember a particular exchange where Eliezer got so frustrated at my inability to do some basic Bayesian reasoning that he ended up writing a whole guide to Bayes' Theorem). Part of the problem seemed to be not that we lacked intelligence, but that we just didn't know how to think very good. Our epistemics were garbage and it wasn't very clear why.

Eliezer went off into the wilderness and stopped hanging out on IRC. The channel kind of died down and went out with a whimper. I got busy doing other stuff but tried to keep track of what was happening. The community felt like it was in a lull.

Then Overcoming Bias started! Eliezer and Robin posted lots of great stuff! There was an AI foom debate. Sequences of posts were posted. It was fun!

Then at some point LessWrong started. Things really picked up. New people emerged in the community. I found myself busy and sidelined at this time,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Explaining the Twitter Postrat Scene, published by Jacob Falkovich on April 5, 2022 on LessWrong.

You are reading this post on LessWrong. If you're like most of my rationalist friends, you believe that LessWrong's culture of decoupled and straightforward communication and focus on timeless issues of rationality and world-optimization has created a community of like-minded people that you enjoy interacting with and learn a lot from. Once in a while, I tell my LessWrong friends that they should get on Twitter, that the culture of memes and half-joking deeply contextualized and obscure banter on the latest breaking news and culture war topics has also created a wonderful community of like-minded people you can learn a lot from. My rationalist friends are horrified at the suggestion.

They're almost as horrified as people who've tweeted for years about sex and astrology and pineal glands are to discover that half their mutuals are actually LessWrongers.

So, what the hell is going on? Is rationalist twitter a thing and if so, why? Are postrats just rats who post? And am I writing this off the cuff with little editing because that faithfully communicates my Twitter style or because it's Goodheart week? Read on to find out!

Twitter Counterculture

The problems with Twitter are obvious on the surface:

It mostly consists of partisans of competing culture war tribes fighting over polarizing object-level topics with no desire or ability to go meta.

No matter how clearly you express yourself, people will misunderstand it in the least charitable way.

There's no way to enforce barriers to entry, any rando can comment on anything.

But well-known journalists and other bluechecks are even worse, using Twitter either to parrot the same coordinated narrative or to viciously settle personal scores.

The nuggets of vital breaking info and true deep insight are buried under a torrent of bullshit.

Anonymity + audience + zero personal stake in relationships = people being total fuckwads.

But also Twitter is the place where Elon Musk challenges Vladimir Putin to hand to hand combat for the future of Ukraine. People don't want to miss out on that.

To get the best parts of Twitter without the worst, you'd want to join as part of a community with norms that defuse and ameliorate Twitter's worst tendencies. A community where:

Object-level pontificating is looked-down upon, and partisanship is outright scorned.

Illegibility to outsiders is prized, so that only those who have followed you for a while and share a context will understand what you mean while everyone else couldn't even parse it enough to get mad or start a mob against you.

Everyone is encouraged not only to post takes but to socialize, play silly games, flirt, and make friends.

Bluechecks and celebrities are treated with suspicion. The highest status members are (often anonymous) posters who grew their audience on Twitter alone.

The community retweets are a filter surfacing the best nuggets from the rest of Twitter.

The only rule strongly enforced is being cordial in replies and otherwise abiding by Twitter etiquette, violators are blocked.

Members in good standing come hang out in a 400+ attendee in-person summer camp.

This is postrat Twitter, aka TCOT/TPOT (this corner/part of Twitter), aka — my favorite name for the multiple levels of irony and meaning — simply "the ingroup". The ingroup doesn't have a unifying theme or explicit purpose. Most importantly, it doesn't have an outgroup. A couple of sympathetic journalists tried to understand what the ingroup is and only managed to conclude that we’re really weird and that “this is the exact opposite of media Twitter, which is defined exclusively by who you hate”.

The Twitter ingroup is based on who you read and who you're friends on Twitter with, with the fuzziness and porousness of bou...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ideal governance (for companies, countries and more), published by HoldenKarnofsky on April 5, 2022 on LessWrong.

I'm interested in the topic of ideal governance: what kind of governance system should you set up, if you're starting from scratch and can do it however you want?

Here "you" could be a company, a nonprofit, an informal association, or a country. And "governance system" means a Constitution, charter, and/or bylaws answering questions like: "Who has the authority to make decisions (Congress, board of directors, etc.), and how are they selected, and what rules do they have to follow, and what's the process for changing those rules?"

I think this is a very different topic from something like "How does the US's Presidential system compare to the Parliamentary systems common in Europe?" The idea is not to look at today's most common systems and compare them, but rather to generate options for setting up systems radically different from what's common today.

I don't currently know of much literature on this topic (aside from the literature on social choice theory and especially voting methods, which covers only part of the topic). This post describes the general topic and why I care, partly in the hopes that people can point me to any literature I've missed. Whether or not I end up finding any, I'm likely to write more on this topic in the future.

Outline of the rest of the piece:

I'll outline some common governance structures for countries and major organizations today, and highlight how much room there is to try different things that don't seem to be in wide use today. More

I'll discuss why I care about this question. I have a few very different reasons:

A short-term, tangible need: over the last several years, I've spoken with several (more than 3) organizations that feel no traditional corporate governance structure is satisfactory, because the stakes of their business are too great and society-wide for shareholder control to make sense, yet they are too early-stage and niche (and in need of nimbleness) to be structured like a traditional government. An example would be an artificial intelligence company that could end up with a normal commercial product, or could end up bringing about the most important century of all time for humanity. I wish I could point them to someone who was like: "I've read all of, and written much of, the literature on what your options are. I can walk you through the pros and cons and help you pick a governance system that balances them for your needs."

A small probability of a big future win. The world today has lots of governments, but they seem to mostly follow a very small number of basic governance templates. At some point, there will be new states with new Constitutions - maybe via space settlements, maybe via collapse of existing states, etc. - but I expect these moments to be few and far between. A significant literature and set of experts on "ideal governance" could lead to a radically different kind of state government, potentially with radically different policies that the rest of the world could learn from.

A weird, out-of-left-field application. Some of my interest in this topic actually comes via my interest in moral uncertainty: the question of what it's ethical to do when one is struggling between more than one theory of ethics, with radically different implications. This is hard to explain, but I try below.

I'll describe a bit more what I think literature on this question could look like (and what already exists that I know of), partly to guide readers who might be able to help me find more.

Common governance structures today

All of these are simplified; I'm trying to illustrate the basic idea of what questions "ideal governance" is asking.

A standard (e.g., public) corporation works like this: it has shareholders, ...