Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: For alignment, we should simultaneously use multiple theories of cognition and value, published by Roman Leventov on April 24, 2023 on The AI Alignment Forum.

This post is a follow-up to "A multi-disciplinary view on AI safety research". I elaborate on some arguments behind this view.

TL;DR: please skim section headings and bolded sentences in the text.

Computationally tractable mathematical models of alignment are bound to be biased and blind to certain aspects of human values

No single mathematical model of human values that has orders of magnitude fewer degrees of freedom than an actual human will adequately capture the complexity of value because humans are complex systems and, therefore, cannot be reduced to a much simpler model.

If the model is sufficiently complex to robustly capture human values, such as whole-brain emulation, then the ethical concerns and S-risks of actually using these models for alignment appear because the model itself may suffer.

Many mathematical theories of human cognition or frameworks for computing (inferring) human values are considered as the basis for alignment, as well as process theories of alignment that implicitly rely on a particular mathematical theory even if it doesn’t infer (humans’ or AIs’) values explicitly, such as the shard theory (RL-based), or the latest Beren Millidge’s computational anatomy of human values, or cooperative inverse reinforcement learning, or Bayesian models and approaches, or various linguistic process theories of alignment that I expect to become very hot this year due to the astonishing success of LLMs. However, since all these theories are collapsing the complexity of humans (or else they are equivalent to full human simulations), they are all bound to be incomplete.

Moreover, all these theories are bound to be biased (this is a form of inductive bias if you wish), that is, to be relatively blind to specific kinds of human values, or to specific aspects of human nature that we can see as somehow related (or producing) “values”.

In other words, human values are not only complex in the sense that they are very elaborate. Crucially, human values are also not capturable within a single mathematical framework or ontology for describing them.

From “solving the alignment problem” to engineering the alignment process

The main implication of the above thesis is that we should abandon the frame that we should “solve” the alignment problem and seek a smart theory that will “crack” this problem.

I feel that a fair amount of unproductive debates and unproductive alignment research resource allocation stems from this illusion. People often debate whether this or that theory “can or cannot succeed” (in “solving” alignment, it is implied), or try to find the “best” theory and invest their effort into improving that theory because it’s the “best bet”.

Instead, we should adopt a portfolio approach. Theory A captures 90% of “value complexity” to be aligned, then theory B largely overlaps with theory A, but together, they capture 95% of value complexity, then adding theory C to the mix raises it to 97%, etc. (Of course, these “percent” are fictitious and cannot be actually computed.)

This is an engineering approach of adding extra assurances to the alignment process until all stakeholders of the system agree that the quality (the quality of being sufficiently aligned, or alignable to humans, in this case) is assured well enough for production deployment of the system.

When we consider this, it becomes clear that marshalling all effort behind improving a single theory is not optimal, vaguely speaking, due to the law of diminishing returns (also, as noted above, a good fraction of the alignment research community’s brain power goes into actually “finding” that best theory, on both individual and collective levels).

Su...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Brain is Not Close to Thermodynamic Limits on Computation, published by DaemonicSigil on April 24, 2023 on LessWrong.

Introduction

This post is written as a response to jacob_cannel's recent post Contra Yudkowsky on AI Doom. He writes:

EY correctly recognizes that thermodynamic efficiency is a key metric for computation/intelligence, and he confidently, brazenly claims (as of late 2021), that the brain is about 6 OOM from thermodynamic efficiency limits

EY is just completely out of his depth here: he doesn't seem to understand how the Landauer limit actually works, doesn't seem to understand that synapses are analog MACs which minimally require OOMs more energy than simple binary switches, doesn't seem to understand that interconnect dominates energy usage regardless, etc.

Most of Jacob's analysis for brain efficiency is contained in this post: Brain Efficiency: Much More than You Wanted to Know. I believe this analysis is flawed with respect to the thermodynamic energy efficiency of the brain. That's the scope of this post: I will respond to Jacob's claims about thermodynamic limits on brain energy efficiency. Other constraints are out of scope, as is a discussion of the rest of the analysis in Brain Efficiency.

The Landauer limit

Just to review quickly, the Landauer limit says that erasing 1 bit of information has an energy cost of kTlog2. This energy must be dissipated as heat into the environment. Here k is Boltzmann's constant, while T is the temperature of the environment. At room temperature, this is about 0.02 eV.

Erasing a bit is something that you have to do quite often in many types of computations, and the more bit erasures your computation needs, the more energy it costs to do that computation. (To give a general sense of how many erasures are needed to do a given amount of computation: If we add n-bit numbers a and b to get a+bmod2n, and then throw away the original values of a and b, that costs n bit erasures. I.e. the energy cost is nkTlog2.)

Extra reliability costs?

Brain Efficiency claims that the energy dissipation required to erase a bit becomes many times larger when we try to erase the bit reliably.

The key transition error probability α is constrained by the bit energy: α=e−EbkBT Here's a range of bit energies and corresponding minimal room temp switch error rates (in electronvolts):

α=0.49, Eb=0.02eV

α=0.01, Eb=0.1eV

α=10−25, Eb=1eV

This adds a factor of about 50 to the energy cost of erasing a bit, so this would be quite significant if true. To back up this claim, Jacob cites this paper by Michael P. Frank. The relevant equation is pulled from section 2. However, in that entire section, Frank is temporarily assuming that the energy used to represent the bit internally is entirely dissipated when it comes time for the bit to be erased. Dissipating that entire energy is not required by the laws of physics, however. Frank himself explicitly mentions this in the paper (see section 3): The energy used to represent the bit can be partially recovered when erasing it. Only kTlog2 must actually be dissipated when erasing a bit, even if we ask for very high reliability.

(I originally became suspicious of Jacob's numbers here based on a direct calculation. Details in this comment for those interested.)

Analog signals?

Quoting Brain Efficiency:

Analog operations are implemented by a large number of quantal/binary carrier units; with the binary precision equivalent to the signal to noise ratio where the noise follows a binomial distribution.

Because of this analog representation, Jacob estimates about 6000 eV required to do the equivalent of an 8 bit multiplication. However, the laws of physics don't require us to do our floating point operations in analog. "are implemented" does not imply "have to be implemented". Digital multiplication of two 8 bit ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: No, the EMH does not imply that markets have long AGI timelines, published by Jakob on April 24, 2023 on The Effective Altruism Forum.

This post is a response to AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years from January 2023, by @basil.halperin , @J. Zachary Mazlish and @tmychow. In contrast to what they argue, I believe that interest rates are not a reliable instrument for assessing market beliefs about AI timelines - at least not the transformative AI described in the former forum post.

The reason for this is that savvy investors cannot expect to get rich by betting on short AI timelines. Such a bet simply ties up your capital until it loses its value (either because you're dead, or because you're so rich that it hardly matters). Therefore, a savvy investor with short timelines will simply increase their own consumption. No individual can increase their own consumption enough to affect global capital markets - rather, something like tens of millions of people would need to increase their consumption for interest rates to be affected. Interest rates can therefore remain low even if TAI is near, unless all of these people get worried enough about AI to change their savings rate.

Replaying the argument from AGI and the EMH

My understanding of the argument in the former post goes like this:

AI is defined as transformative if it either causes an existential risk, or explosive growth (which, presumably, would be broad-based enough that the typical investor would expect to partake in it)

If we knew that transformative AI was near, people would not need to save as much as they do today - since they would expect to either be dead or very, very rich in the near future

If people save less, capital supply goes down, and interest rates go up. Therefore, if we knew transformative AI was near, interest rates should be high.

Even if we allow for uncertainty around the timing of transformative AI, a significant probability of near-term transformative AI should increase interest rates, since the equilibrium condition is that the expected utility of consumption today and in the future should be equal (reflecting the full distribution of outcomes).

Since interest rates aren't high, if you assume market efficiency, this is evidence against near-term transformative AI.

My high-level response

I will start by granting the definition of transformative AI as either an existential risk or a driver of explosive (and broadly shared) growth. This means that I accept the premise that the marginal value of additional money post-TAI is much, much lower than the marginal value of money today, for any relevant investor.

Second, I consider the dynamics of how prices in markets can change over time. This is something which the original post glosses over a bit, which is forgivable, since the main social value of markets is that they can work like a black box information aggregation mechanism where you don't need to think too carefully about the gears. However, in this case, this is a crucial reason why their argument seems to fail.

Let's consider two possible ways the price of an asset can change. Either, some information becomes available to all players in the market, and they uniformly update their assessment of the value of the asset, and adjust their market positions accordingly. Alternatively, some investor gains private information which indicates the asset is mispriced, and they take a big, directional bet based on said information, unilaterally moving the price. These two situations are extremes on a spectrum, and in most cases, price changes will reflect a situation somewhere in between these extremes.

My argument is that this matters in the special context of interest rates. After all, interest rates reflect the aggregate capital supply in the world....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Yudkowsky on AI Doom, published by jacob cannell on April 24, 2023 on LessWrong.

Eliezer Yudkowsky predicts doom from AI: that humanity faces likely extinction in the near future (years or decades) from a rogue unaligned superintelligent AI system. Moreover he predicts that this is the default outcome, and AI alignment is so incredibly difficult that even he failed to solve it.

EY is an entertaining and skilled writer, but do not confuse rhetorical writing talent for depth and breadth of technical knowledge. I do not have EY's talents there, or Scott Alexander's poetic powers of prose. My skill points instead have gone near exclusively towards extensive study of neuroscience, deep learning, and graphics/GPU programming. More than most, I actually have the depth and breadth of technical knowledge necessary to evaluate these claims in detail.

I have evaluated this model in detail and found it substantially incorrect and in fact brazenly naively overconfident.

Intro

Even though the central prediction of the doom model is necessarily un-observable for anthropic reasons, alternative models (such as my own, or moravec's, or hanson's) have already made substantially better predictions, such that EY's doom model has low posterior probability.

EY has espoused this doom model for over a decade, and hasn't updated it much from what I can tell. Here is the classic doom model as I understand it, starting first with key background assumptions:

The brain inefficiency assumption: The human brain is inefficient in multiple dimensions/ways/metrics that translate into intelligence per dollar; inefficient as a hardware platform in key metrics such as thermodynamic efficiency.

The mind inefficiency or human incompetence assumption: In terms of software he describes the brain as an inefficient complex "kludgy mess of spaghetti-code". He derived these insights from the influential evolved modularity hypothesis as popularized in ev pysch by Tooby and Cosmides. He boo-hooed neural networks, and in fact actively bet against them in actions by hiring researchers trained in abstract math/philosophy, ignoring neuroscience and early DL, etc.

The more room at the bottom assumption: Naturally dovetailing with points 1 and 2, EY confidently predicts there is enormous room for further hardware improvement, especially through strong drexlerian nanotech.

The alien mindspace assumption: EY claims human mindspace is an incredibly narrow twisty complex target to hit, whereas the space of AI mindspace is vast, and AI designs will be something like random rolls from this vast alien mindspace resulting in an incredibly low probability of hitting the narrow human target.

Doom naturally follows from these assumptions: Sometime in the near future some team discovers the hidden keys of intelligence and creates a human-level AGI which then rewrites its own source code, initiating a self improvement recursion which quickly results in the AGI developing strong nanotech and killing all humans within a matter of days or even hours.

If assumptions 1 and 2 don't hold (relative to 3) then there is little to no room for recursive self improvement. If assumption 4 is completely wrong then the default outcome is not doom regardless.

Every one of his key assumptions is mostly wrong, as I and others predicted well in advance. EY seems to have been systematically overconfident as an early futurist, and then perhaps updated later to avoid specific predictions, but without much updating his mental models (specifically his nanotech-woo model, as we will see).

Brain Hardware Efficiency

EY correctly recognizes that thermodynamic efficiency is a key metric for computation/intelligence, and he confidently, brazenly claims (as of late 2021), that the brain is about 6 OOM from thermodynamic efficiency limits:

Which brings me to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting Started With Naturalism, published by LoganStrohl on April 23, 2023 on LessWrong.

How Does a Naturalist Study Begin?

There are many different ways to start. Whichever way(s) you choose, what’s important is that 1) you cultivate a lot of original seeing, and 2) you develop a clear sense of your own curiosity. Any starting point that sets you on that track is a correct starting point for naturalism.

What do I mean by “a clear sense of your own curiosity”?

When dealing with challenges, questions, and observations, there are many ways to regard the world that are useful for different purposes: As something you can control, as a community of which you are a part, as raw material for building things, even as an adversary to overcome. The naturalist stance deemphasizes most ways relating to the world, and regards the world instead as something that may reveal itself to you as you open up to it.

The type of curiosity I’m talking about is a yearning for contact with the territory (or with some part of the territory). When I am curious in this way, and maintain that curiosity, it is not burdensome to engage in patient and direct observation. Rather, I find the practice of naturalism inevitable.

What I have to offer as you start off is not so much a specific recommendation as an incomplete menu of options, all of which attempt to cultivate curiosity and original seeing. In my experience, some people can skip this introductory part altogether, while others need to spend a week or more learning how to deliberately pay attention to anything at all before they even begin to consider their chosen object of study. Even I start my studies in different places at different times. There are many good ways to begin, and I hope you will take the time to find (or make) your own.

Starting Place 1: Skip this whole essay. Go straight to “Locating Fulcrum Experiences”.

I don’t often run into situations where this is really the right move, but it does happen. I’m talking first about the “skip ahead” option here because I hope it’ll make the goal of the other options clearer.

I once talked to a person who had found the idea of “comparative advantage” really useful, and wanted to figure out how it might generalize beyond the domain of economics. “I want to be able to use the concept as a lens,” they told me, “and I want to learn to recognize instances where it applies, more-or-less reflexively.” I could tell that they should probably skip straight to “Locating Fulcrum Experiences” for these reasons:

They already knew quite specifically what it was they wanted to study. They came in with, “I want to generalize comparative advantage,” and not with, “I have a vague sense that something somehow important might be going on with whatever thing people are pointing toward when they say ‘comparative advantage’.”

They seemed primarily curious, primarily driven to learn and to understand; they did not seem desperate to make a problem go away immediately. Problem solving is good, of course, but the naturalist approach usually requires a curiosity-dominant orientation to work.

There was no obvious-to-me sign that they might be deeply confused about comparative advantage, or about cross-domain application, such that they might need a complete overhaul of their conceptualization.

If you know quite precisely what you want to study, you feel more immediately driven by curiosity than by problem-solving, and you’re pretty sure you are not fundamentally confused, then I think there’s a good chance you should skip all of this and go straight to Locating Fulcrum Experience. [I'll put a link here as soon as the next essay's published]

If that’s not you, here are some other possible starting places.

Starting Place 2: Try Catching the Spark (All Of It, Or Just Part Of It)

Catching the Spark is an orientation proce...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Commission for Stopping Further Improvements: A letter of note from Isambard K. Brunel, published by jasoncrawford on April 21, 2023 on LessWrong.

A letter of note from Isambard K. Brunel, civil engineer

On May 24, 1847, a bridge over the Dee River in Chester, England, collapsed. A passenger train plunged into the river; five people were killed and nine seriously injured.

The subsequent investigation blamed the bridge’s cast iron girders. Cast iron, like concrete but unlike wrought iron or steel, is strong in compression but weak in tension, and it is brittle, meaning that it breaks all at once, rather than deforming. The wrought iron trusses evidently were not enough to strengthen the girder.

In response to the disaster, a Royal Commission on the Application of Iron to Railway Structures was created in August of that year, “to inquire into the conditions to be observed by engineers in the application of iron in structures exposed to violent concussions and vibration”—that is, to set up standards and requirements, or as they were known in France at the time, règles de l’art.

In their investigation, the Commission solicited the opinion of one of the most eminent engineers of the age, Isambard Kingdom Brunel. But his response was, presumably, not what they expected.

Brunel begins his letter by saying that he is sorry they asked for his opinion, because of “my doubts of the advantage of such an enquiry, and my fears of its being, on the contrary, productive of much mischief, both to science and to the profession.” (Brunel’s son, writing his biography, says that he called them “The Commission for Stopping Further Improvements in Bridge Building.“) But since they did ask, he felt it necessary to state his full and honest views.

While he was happy to give his engineering opinion to the commission, he warned that

. the attempt to collect and re-issue as facts, with the stamp of authority, all that may be offered gratuitously to a Commission in the shape of evidence or opinions, to stamp with the same mark of value statements and facts, hasty opinions and well-considered and matured convictions, the good and the bad, the metal and the dross . this, I believe, always has rendered, and always will render, such collections of miscalled evidence injurious instead of advantageous to science.

He argued that there was no way the Commission could get better information than an engineer could on his own, but that in addition they would receive a lot of useless opinions, which they would feel compelled to publish anyway.

He went on to explain why he believed that rulemaking by such bodies would stop progress in the field:

If the Commission is to enquire into the conditions “to be observed,” it is to be presumed that they will give the result of their enquiries; or, in other words, that they will lay down, or at least suggest, “rules” and “conditions to be (hereafter) observed” in the construction of bridges, or, in other words, embarrass and shackle the progress of improvement tomorrow by recording and registering as law the prejudices or errors of today.

Nothing, I believe, has tended more to distinguish advantageously the profession of engineering in England and in America, nothing has conduced more to the great advance made in our profession and to our pre-eminence in the real practical application of the science, than the absence of all règles de l’art—a term which I fear is now going to be translated into English by the words “conditions to be observed.” No man, however bold or however high he may stand in his profession, can resist the benumbing effect of rules laid down by authority. Occupied as leading men are, they could not afford the time, or trouble, or responsibility of constantly fighting against them—they would be compelled to abandon all idea of improving upon them...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why do we care about agency for alignment?, published by Chris Leong on April 23, 2023 on The AI Alignment Forum.

Many people believe that understanding "agency" is crucial for alignment, but as far as I know, there isn't a canonical list of reasons why we care about agency. Please describe any reasons why we might care about the concept of agency for understanding alignment below. If you have multiple reasons, please list them in separate answers below.

Please also try to be specific as possible about what our goal is in the scenario. For example, instead of just saying, "We want to know what an agent is so that we can determine whether or not a given AI is a dangerous agent" it might be better to say something along the lines of "We have an AI which may or may not have goals aligned with us and we want to know how dangerous it would be if it weren't aligned. We want to use interpretability tools to determine the extent to which it will pursue instrumental incentives and this would be easier if we knew exactly what we were looking for. In particular, we can set up a dichotomy between agents which have only learned to pursue instrumental incentives based upon a few heuristics it learned during training and an agent which maximises its goals during deployment. We would like the ability to figure out where a particular agent is on this dichotomy".

In a few days, I'll add any use cases I'm aware of myself that either haven't been covered or that I don't think have been adequately explained by different answers.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Could a superintelligence deduce general relativity from a falling apple? An investigation, published by titotal on April 23, 2023 on LessWrong.

Introduction:

In the article/short story “That Alien Message”, Yudkowsky writes the following passage, as part of a general point about how powerful super-intelligences could be:

Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

As a computational physicist, this passage really stuck out to me. I think I can prove that this passage is wrong, or at least misleading. In this post I will cover a wide range of arguments as to why I don't think it holds up.

Before continuing, I want to state my interpretation of the passage, which is what I’ll be arguing against.

1. Upon seeing three frames of a falling apple and with no other information, a superintelligence would assign a high probability to Newtonian mechanics, including Newtonian gravity. So if it was ranking potential laws of physics by likelihood, “Objects are attracted to each other by their masses in the form F=Gmm/r2” would be near the top of the list.

2. Upon seeing only one frame of a falling apple and one frame of a single blade of grass, a superintelligence would assign a decently high likelihood to the theory of general relativity as put forward by Einstein.

This is not the only interpretation of the passage. It could just be saying that an AI would “invent” general relativity in terms of inventing the equations just by idly playing around, like mathematicians playing with 57 dimensional geometries or whatever. However, the phrase “under active consideration”, and “dominant hypothesis” imply that these aren’t just inventions, these are deductions. A monkey on a typewriter could write down the equations for Newtonian gravity, but that doesn’t mean they deduced it. What matters is whether the knowledge gained from the frames could be used to achieve things.

I’m not interested in playing semantics or trying to read minds here. My interpretation above seems to be what the commenters took the passage to mean. I’m mainly using the passage as a starting off point for discussion about the limits of first principles computations.

Here is a frame of an apple, and a frame of a field of grass.

I encourage you to ponder these images in detail. Try and think for yourself the most plausible method for a superintelligence to deduce general relativity from one apple image and one grass image.

I’m going to assume that the AI can somehow figure out the encoding of the images and actually “see” images like the ones above. This is by no means guaranteed, see the comments of this post for a debate on the matter.

What can be deduced:

It’s true that the laws of physics are not that complicated to write down. Doing the math properly would not be an issue for anything we would consider a “superintelligence”.

It’s also true that going from 0 images of the world to 1 image imparts a gigantic amount of information. Before seeing the image, the number of potential worlds it could be in is infinite and unconstrained, apart from cogito ergo sum. After seeing 1 image, all potential worlds that are incompatible with the image can be ruled out, and many more worlds can be deemed unlikely. This still leaves behind an infinite number of potential plausible worlds, but there are significantly more constraints on those worlds.

With 2 image, the constra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: David Edmonds's biography of Parfit is out, published by Pablo on April 23, 2023 on The Effective Altruism Forum.

Parfit: A Philosopher and His Mission to Save Morality, by David Edmonds, was published a few days ago (Amazon | Audible | Goodreads | Oxford).

The book is worth reading in full, but here are some passages that focus specifically on Parfit's involvement with effective altruism:

In his retirement, an organization with which he felt a natural affinity approached him for support. The Oxford-based charity Giving What We Can (GWWC) was set up by Toby Ord, an Australian-born philosopher, and another young philosopher, Will MacAskill. The charity's direct inspiration came from Peter Singer. Singer had devised a thought experiment that was simple but devastatingly effective in making people reflect on their behavior and their attitude to those in need. [...]

This thought experiment has generated a significant secondary literature. Certainly, the GWWC initiators found it compelling. Their mission was to persuade people to give more of their income away and to donate to effective organizations that could make a real difference. [...]

There were spinoff organizations from GWWC such as 80,000 Hours. The number refers to the rough number of hours we might have in our career, and 80,000 Hours was set up to research how people can most effectively devote their time rather than their money to tackling the world's most pressing problems. In 2012, the Centre for Effective Altruism was established to incorporate both GWWC and 80,000 Hours.

Since its launch, the effective altruism movement has grown slowly but steadily. Most of the early backers were idealistic young postgraduates, many of them philosophers. If Singer was the intellectual father of the movement, Parfit was its grandfather. It became an in-joke among some members that anyone who came to work for GWWC had to possess a copy of Reasons and Persons. Some owned two copies: one for home, one for the office. But it took Parfit until 2014 to sign the GWWC pledge. And he agreed to do so only after wrangling over the wording.

Initially, those who joined the GWWC campaign were required to make a public pledge to donate at least 10% of their income to charities that worked to relieve poverty. Parfit had several issues with this. For reasons the organizers never understood, he said that the participants had to make a promise rather than a pledge. He may have believed that a promise entailed a deeper level of commitment. Nor was he keen on the name "Giving What We Can". 10% of a person's income is certainly a generous sum, and in line with what adherents to some world religions are expected to give away. Nevertheless, Parfit pointed out, it was obvious that people could donate more. [...]

Parfit also caviled at the word 'giving'. He believed this implied we are morally entitled to what we hand over, and morally entitled to our wealth and high incomes. This he rejected. Well-off people in the developed world were merely lucky that they were born into rich societies: they did not deserve their fortune.

Linguistic quibbles aside, the issue that Parfit felt most strongly about was the movement's sole focus, initially, on poverty and development. While it was indeed pressing to relieve the suffering of people living today, Parfit argued, there should be an option that at least some of the money donated be earmarked for the problems of tomorrow. The human population has risen to a billion, and faces existential risks such as meteors, nuclear war, bioterrorism, pandemics, and climate change. Parfit claimed that between (A) peace, (B) a war which kills 7.5 billion people and (C) a war which killed everyone, the difference between (B) and (C) was much greater than the difference between (A) and (B). [...]

Given how grim human exist...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Endo-, Dia-, Para-, and Ecto-systemic novelty, published by Tsvi Benson-Tilsen on April 23, 2023 on The AI Alignment Forum.

[Metadata: crossposted from. First completed January 10, 2023. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text might be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]

Novelty can be coarsely described as one of: fitting within a preexisting system; constituting a shift of the system; creating a new parallel subsystem; or standing unintegrated outside the system.

Thanks to Sam Eisenstat for related conversations.

Novelty is understanding (structure, elements) that a mind acquires (finds, understands, makes its own, integrates, becomes, makes available for use to itself or its elements, incorporates into its thinking). A novel element (that is, structure that wasn't already there in the mind fully explicitly) can relate to the mind in a few ways, described here mainly by analogy and example. A clearer understanding of novelty than given here might clarify the forces acting in and on a mind when it is acquiring novelty, such as "value drives".

Definitions

"System" ("together-standing") is used here to emphasize the network of relations between elements of a mind.

These terms aren't supposed to be categories, but more like overlapping regions in the space of possibilities for how novelty relates to the preexisting mind.

Endosystemic novelty (or "basis-aligned" or "in-ontology") is novelty that is integrated into the mind by fitting alongside and connecting to other elements, in ways analogous to how preexisting elements fit in with each other. Endosystemic novelty is "within the system"; it's within the language, ontology, style of thinking, conceptual scheme, or modus operandi of the preexisting mind.

Diasystemic novelty (or "cross-cutting" or "basis-skew" or "ontological shift") is novelty that is constituted as a novel structure of the mind by many shifts in many of the preexisting elements or relations, adding up to something coherent or characteristically patterned. Diasystemic novelty is "throughout the system"; it's skew to the system, cross-cutting the preexisting schemes; it touches (maybe subtly) many elements, many relations, or certain elements that shape much of the mind's activity, hence altering the overall dynamics or character of the system.

Parasystemic novelty is novelty that is only loosely integrated into the whole mind, while being more tightly integrated within a subsystem of the mind. Parasystemic novelty is "alongside the system"; it's neither basis-aligned (since it's outside preexisting tightly integrated systems) nor cross-cutting (as it doesn't touch most of the system, or require most of the system for its constitution).

Ectosystemic novelty is novelty that is merely juxtaposed or appended to the mind, without being really integrated. Ectosystemic novelty is "on or outside the system"; it's external, only loosely related to the mind, as by a narrow interface or by an external aggregration mechanism. It differs from parasystemic novelty by being even less integrated, and by not nucleating or expanding a tightly integrated subsystem.

Analogies

Analogy: If a language is like a mind, then a new word would be endosystemic novelty; a sound shift or (more properly) a grammatical innovation would be diasystemic (cross-cutting) novelty; specialized languages (such as scientific jargon), and dialect formation, would be parasystemic novelty; and an encounter with a foreign language would be ectosystemic novelty. Pidgins, being unstable and noncanonical, witness the ectosystemic nature: the foreign languages don't integrate. Creoles, however, could be dubbed "systemopoetic novelty"--like parasystemic novel...