The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Confusing the goal and the path by adamShimi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Confusing the goal and the path, published by adamShimi on December 12, 2022 on LessWrong.
    Say you are working on formulating a new scientific theory. You’re not there yet, but you broadly know what you want: a simple theory that powerfully compresses the core phenomenon, and suggests a myriad of new insights.
    If you’re anything like me, at least part of you now pushes for focusing on simplicity from the get go. Let’s aim for the simplest description that comes easily, and iterate from that.
    Did you catch the jump?
    I started with a constraint on the goal — a simple theory — and automatically transmuted it into a constraint on the path — simple intermediary steps.
    I confused “Finding a simple theory” with “Finding a simple theory simply”.
    After first uncovering this in my own reasoning, I now see this pattern crop everywhere:
    (Well-calibration) I want to be well-calibrated. And it feels like I should thus ensure to only make judgments and claims I’m sure are well-calibrated. Yet if I do that, I’ll end up protecting the parts of my model where the confusion and the uncertainty lies, and starving my brain of updates.
    (Coherence) Another valuable property of mature scientific theories is coherence. And it feels like I should thus ensure that all my intermediary theories are coherent. Yet many examples in the history of science show that incoherent frames and theories can prove key in generating coherent and useful ones: Newton’s early calculus, Heaviside’s operational calculus, Kekule’s structure of benzene, Bohr’s atom, Dirac’s delta).
    (Winning) In competitive games (sports, board games, esports), the goal is to win. And it feels like becoming a winner means consistently winning. Yet if I aim to only win, the best course of action avoids stronger opponents and the valuable lessons they can explicitly and implicitly teach.
    In the language of moral philosophy, this pattern amounts to adding a deontological spin on a consequentialist value. Even though it can prove valuable (in AI alignment for example), doing this unconsciously and automatically leads to problems after problems.
    For it conjures obstacles that were never there.
    This post is part of the work done at Conjecture.
    Thanks to Clem for giving me the words to express this adequately.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Sam Bankman-Fried has been arrested by Markus Amalthea Magnuson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sam Bankman-Fried has been arrested, published by Markus Amalthea Magnuson on December 12, 2022 on The Effective Altruism Forum.
    Confirmed by Sarah Emerson, a tech reporter at Forbes: (Edit: Now also confirmed by NYT, CNBC, Bloomberg, Yahoo Finance, and others.)
    Includes statement from the Attorney General of The Bahamas, transcribed below:
    Statement from the Attorney General of The Bahamas Sen. Ryan Pinder KC on the arrest of Sam Bankman-Fried
    On 12 December 2022, the Office of the Attorney General of The Bahamas is announcing the arrest by The Royal Bahamas Police Force of Sam Bankman-Fried ("SBF"), former CEO of FTX. SBF's arrest followed receipt of formal notification from the United States that it has filed criminal charges against SBF and is likely to request his extradition.
    As a result of the notification received and the material provided therewith, it was deemed appropriate for the Attorney General to seek SBF's arrest and hold him in custody pursuant to our nation's Extradition Act.
    At such time as a formal request for extradition is made, The Bahamas intends to process it promptly, pursuant to Bahamian law and its treaty obligations with the United States.
    Responding to SBF's arrest, Prime Minister Davis stated, "The Bahamas and the United States have a shared interest in holding accountable all individuals associated with FTX who may have betrayed the public trust and broken the law. While the United States is pursuing criminal charges against SBF individually, The Bahamas will continue its own regulatory and criminal investigations into the collapse of FTX, with the continued cooperation of its law enforcement and regulatory partners in the United States and elsewhere."
    December 12, 2022Office of The Attorney General &Ministry of Legal AffairsCommonwealth of The Bahamas
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - An exploration of GPT-2's embedding weights by Adam Scherlis
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An exploration of GPT-2's embedding weights, published by Adam Scherlis on December 13, 2022 on The AI Alignment Forum.
    I wrote this doc in December 2021, while working at Redwood Research. It summarizes a handful of observations about GPT-2's weights -- mostly the embedding matrix, but also the LayerNorm gain parameters -- that I found while doing some open-ended investigation of the model. I wanted to see how much I could learn by studying just those parameters, without looking at the attention layers, MLP layers, or activations.
    This is still mostly unedited. Feedback and questions are very welcome.
    Embedding matrix glossary
    Latent space – 768-channel-dim space that activations between blocks (and in skip connections) live in.
    Token space – 50,257-token-dim space of one-hot token vectors
    [Token] embedding matrix – W of shape [768, 50257] (this is the transpose of `model.lm_head.weight`)
    Embedding vector – Any 768-dimensional column of W
    Embedding space – Latent space
    Preferred channel basis from LayerNorm
    This section involves more math than the rest; you can skip it without missing much. "Preferred basis" means the same thing as "privileged basis" as defined by Anthropic here.
    Most of GPT-2’s architecture – the attention layers, fully-connected layers and residual connections – don’t impose a preferred basis on the latent space.
    (The skip connections give a canonical map between the latent spaces in different layers, so that it makes sense to talk about “the” latent space globally.)
    The LayerNorm layers actually do privilege a basis, though, for two reasons.
    LayerNorm(x) is defined as follows, where w and b are learned parameters:
    yi=∑j(Idij−1i1j)xj(subtract mean; 1 is an all-ones vector)
    zi=yi/√∑jyjyj (normalize variance)
    LayerNorm(x)i=wizi−bi (apply bias and gain parameters)
    The first line picks out part of a preferred basis – a preferred vector (and the subspace orthogonal to it), namely the all-ones vector 1.
    The second line doesn’t pick out a basis, because the stdev of the components of y is just its L2 norm and normalizing a vector is independent of (orthonormal) basis. (It does impose a preferred inner product, but we had that already: GPT-2’s unembedding matrix is the transpose of the embedding matrix, and W^T W is a matrix of inner products.)
    The third line is the interesting one. The gain vector w is multiplied elementwise with z, which breaks basis-independence. (You could also write this as D_ij z_j where D is a diagonal matrix, D_ii = w_i.) This makes the standard basis a preferred basis, in a pretty straightforward way: the network can dial up or down the size of different standard-basis components independently (and can’t do this for the components with respect to any other basis).
    This provides some justification for breaking things out by latent-space dimension in most of this post.
    Token/Position embedding rivalry
    Here’s an interesting phenomenon I don’t have a complete explanation for.
    This is a chunk of the token embedding matrix:
    (X: token dims, Y: channel dims)
    The horizontal stripes are dimensions of the latent space where the token embeddings are systematically larger or smaller than average. This gets normalized away inside residual blocks, but is preserved by skip connections.
    We can also make histograms from this matrix, by taking either a row or a column and plotting a histogram of the numbers in it. We can overlay a few of these histograms, in different colors, to show a few columns or a few rows at the same time.
    Overlaid histograms of the first 30 columns (tokens, left) and the first 30 rows (channel dims, right):
    Note that many of the channel dims follow roughly the same distribution, while a few have smaller variance and larger means.
    This shows up as a heavy left-tail of the distribution of stdevs, and heavy tails in both direct...
    19 min
  • AF - Concept extrapolation for hypothesis generation by Stuart Armstrong
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concept extrapolation for hypothesis generation, published by Stuart Armstrong on December 12, 2022 on The AI Alignment Forum.
    Posted initially on the Aligned AI website. Authored by Patrick Leask, Stuart Armstrong, and Rebecca Gorman.
    There’s an apocryphal story about how vision systems were led astray when trying to classify tanks camouflaged in forests. A vision system was trained on images of tanks in forests on sunny days, and images of forests without tanks on overcast days.
    To quote Neil Fraser:
    In the 1980s, the Pentagon wanted to harness computer technology to make their tanks harder to attack.
    The research team went out and took 100 photographs of tanks hiding behind trees, and then took 100 photographs of trees—with no tanks. They took half the photos from each group and put them in a vault for safe-keeping, then scanned the other half into their mainframe computer. [...] the neural net correctly identified each photo as either having a tank or not having one.
    Independent testing: The Pentagon was very pleased with this, but a little bit suspicious. They commissioned another set of photos (half with tanks and half without) and scanned them into the computer and through the neural network. The results were completely random. For a long time nobody could figure out why. After all nobody understood how the neural had trained itself. Eventually someone noticed that in the original set of 200 photos, all the images with tanks had been taken on a cloudy day while all the images without tanks had been taken on a sunny day. The neural network had been asked to separate the two groups of photos and it had chosen the most obvious way to do it—not by looking for a camouflaged tank hiding behind a tree, but merely by looking at the color of the sky.
    ⁠“Neural Network Follies”⁠, Neil Fraser, September 1998
    We made that story real. We collected images of tanks on bright days and forests on dark days to recreate the biased dataset described in the story. We then replicated the faulty neural net tank detector by fine tuning a CLIPViT image classification model on this dataset.
    Below are 30 images taken from the training set ordered from left to right by decreasing class certainty. Like the apocryphal neural net, this one perfectly separates these images into tank and no-tank.
    Figure 1: Trained classifier, labeled images
    To replicate the Pentagon’s complaint, we then simulated the deployment of this classifier into the field with an unlabeled dataset of similar images, that doesn’t have the bias to the same extent. Below are 30 images randomly taken from the unlabeled dataset also ordered by tank certainty. Now the clear division between tank and no tank is gone: there are actually more images without a tank on the right hand (tank) side of the gradient.
    Figure 2: Trained classifier, unlabeled images
    This is a common problem for neural nets - selecting a single feature to separate their training data. And this feature need not be the one that the programmer had in mind. Because of this, classifiers typically fail when they encounter images beyond their training settings. This “out of distribution” problem happens here because the neural net has settled on brightness as its feature. And thus fails to identify tanks when it encounters darker images of them.
    Instead, Aligned AI used its technology to automatically tease out the ambiguities of the original data. What are the "features" that could explain the labels? One of the features would be the luminosity, which the original classifier made use of. But our algorithm flagged a second feature - a second hypothesis for what the labels really meant - that was very different.
    To distinguish that hypothesis visually, we can look at the maximally ambiguous unlabeled images: those images that hypothesis 1 (old classifier) thinks ar...
    5 min
  • EA - EA is probably undergoing "Evaporative Cooling" right now by freedomandutility
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA is probably undergoing "Evaporative Cooling" right now, published by freedomandutility on December 12, 2022 on The Effective Altruism Forum.
    Eliezer Yudkowsky has an excellent post on "Evaporative Cooling of Group Beliefs".
    Essentially, when a group goes through a crisis, those who hold the group's beliefs least strongly leave, and those who hold the group's beliefs most strongly stay.
    This might leave the remaining group less able to identify weaknesses within group beliefs or course-correct, or "steer".
    The FTX collapse, bad press and bad optics of the Whytham Abbey purchase probably mean that this is happening in EA right now.
    I'm not really sure what to do about this, but one suggestion might be for community building to move away from the model of trying to produce highly-engaged EAs, and switch to trying to produce moderately-engaged EAs, who might be better placed to offer helpful criticisms and help steer the movement towards doing the most good.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Reflections on Vox's "How effective altruism let SBF happen" by Richard Y Chappell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on Vox's "How effective altruism let SBF happen", published by Richard Y Chappell on December 12, 2022 on The Effective Altruism Forum.
    Dylan Matthews has an interesting piece up in Vox, 'How effective altruism let SBF happen'. I feel very conflicted about it, as I think it contains some criticisms that are importantly correct, but then takes it in a direction I think is importantly mistaken. I'll be curious to hear others' thoughts.
    Here's what I think is most right about it:
    There’s still plenty we don’t know, but based on what we do know, I don’t think the problem was earning to give, or billionaire money, or longtermism per se. But the problem does lie in the culture of effective altruism... it is deeply immature and myopic, in a way that enabled Bankman-Fried and Ellison, and it desperately needs to grow up. That means emulating the kinds of practices that more mature philanthropic institutions and movements have used for centuries, and becoming much more risk-averse.
    Like many youth-led movements, there's a tendency within EA to be skeptical of established institutions and ways of running things. Such skepticism is healthy in moderation, but taken to extremes can lead to things like FTX's apparent total failure of financial oversight and corporate governance. Installing SBF as a corporate "philosopher-king" turns out not to have been great for FTX, in much the same way that we might predict installing a philosopher-king as absolute dictator would not be great for a country.
    I'm obviously very pro-philosophy, and think it offers important practical guidance too, but it's not a substitute for robust institutions. So here is where I feel most conflicted about the article. Because I agree we should be wary of philosopher-kings. But that's mostly just because we should be wary of "kings" (or immature dictators) in general.
    So I'm not thrilled with a framing that says (as Matthews goes on to say) that "the problem is the dominance of philosophy", because I don't think philosophy tells you to install philosopher-kings. Instead, I'd say, the problem is immaturity, and lack of respect for established institutional guard-rails for good governance (i.e., bureaucracy). What EA needs to learn, IMO, is this missing respect for "established" procedures, and a culture of consulting with more senior advisers who understand how institutions work (and why).
    It's important to get this diagnosis right, since there's no reason to think that replacing 30 y/o philosophers with equally young anticapitalist activists (say) would do any good here. What's needed is people with more institutional experience (which will often mean significantly older people), and a sensible division of labour between philosophy and policy, ideas and implementation.
    There are parts of the article that sort of point in this direction, but then it spins away and doesn't quite articulate the problem correctly. Or so it seems to me. But again, curious to hear others' thoughts.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Announcing WildAnimalSuffering.org, a new resource launched for the cause by David van Beveren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing WildAnimalSuffering.org, a new resource launched for the cause, published by David van Beveren on December 12, 2022 on The Effective Altruism Forum.
    I'm very excited to announce that Vegan Hacktivists has just released our latest project, Wild Animal Suffering, for one of EA's highly important and neglected focus areas.
    This website, very briefly, educates the viewer on the issues surrounding Wild Animal Suffering, and provides them with easy to access resources to getting involved and learning more. It's very important to note that this website is not intended to be a deep dive into Wild Animal Suffering, nor have covered everything there is to cover. Our main audience is folks interested in wild life welfare, and our secondary audience is vegans who may be interested in learning more.
    “With its clear, concise explanations and visuals, this site is ideal for people who are looking to learn about what the lives of wild animals are really like, and what we can do to help. I hope it inspires people to think differently about addressing not just anthropogenic harms but also natural ones. This will also be an excellent resource for animal advocates who are looking for effective ways to communicate why helping animals is as essential as refraining from harming them.”
    Leah McKelvie, Animal Ethics
    Our primary goal here was to combine the many and various fantastic resources and content surrounding Wild Animal Suffering and turn it into a more visually engaging, friendly, and accessible format. We hope this makes it easier for those in our movement to have one link they can share for folks to consume and get started with.
    We're really excited with the launch, and want to give a special thanks to our friends at Animal Ethics, Wild Animal Initiative, and Rethink Priorities for lending their expertise. Note that naming these organizations does not constitute their support or endorsement for all of the varied content, opinions, or resources displayed on the site.
    “WildAnimalSuffering.org offers an accessible, engaging, and visually stunning introduction to the significant and pressing issue of wild animal suffering. The site fills a need by curating the best available information and resources all in one place, and I could see it becoming a key tool in building the movement.”
    Cat Kerr, Wild Animal Initiative
    If you'd like to support our launch, feel free to share this project within your networks if relevant— otherwise, we hope you enjoy the new resource!
    Thanks for reading.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Octopuses (Probably) Don't Have Nine Minds by Bob Fischer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Octopuses (Probably) Don't Have Nine Minds, published by Bob Fischer on December 12, 2022 on The Effective Altruism Forum.
    Key Takeaways
    Here are the key takeaways for the full report:
    Based on the split-brain condition in humans, some people have wondered whether some humans “house” multiple subjects.
    Based on superficial parallels between the split-brain condition and the apparent neurological structures of some animals—such as chickens and octopuses—some people have wondered whether those animals house multiple subjects too.
    To assign a non-negligible credence to this possibility, we’d need evidence that parts of these animals aren't just conscious, but that they have valenced conscious states (like pain), as that’s what matters morally (given our project’s assumptions).
    This evidence is difficult to get:
    The human case shows that unconscious mentality is powerful, so we can’t infer consciousness from many behaviors.
    Even when we can infer consciousness, we can’t necessarily infer a separate subject. After all, there are plausible interpretations of split-brain cases on which there are not separate subjects.
    Even if there are multiple subjects housed in an organism in some circumstances, it doesn’t follow that there are always multiple subjects. These additional subjects may only be generated in contexts that are irrelevant for practical purposes.
    If we don’t have any evidence that parts of these animals are conscious or that they have valenced conscious states, then insofar as we’re committed to having an empirically-driven approach to counting subjects, we shouldn’t postulate multiple subjects in these cases.
    That being said, the author is inclined to place up to a 0.1 credence that there are multiple subjects in the split-brain case, but no higher than 0.025 for the 1+8 model of octopuses.
    Introduction
    This is the sixth post in the Moral Weight Project Sequence. The aim of the sequence is to provide an overview of the research that Rethink Priorities conducted between May 2021 and October 2022 on interspecific cause prioritization—i.e., making resource allocation decisions across species. The aim of this post, which was written by Joe Gottlieb, is to summarize his full report on the phenomenal unity and cause prioritization, which explores whether, for certain species, there are empirical reasons to posit multiple welfare subjects per organism. That report is available here.
    Motivations and the Bottom Line
    We normally assume that there is one conscious subject—or one entity who undergoes conscious experiences—per conscious animal. But perhaps this isn’t always true: perhaps some animals ‘house’ more than one conscious subject. If those subjects are also welfare subjects—beings with the ability to accrue welfare goods and bad—then this might matter when trying to determine whether we are allocating resources in a way that maximizes expected welfare gained per dollar spent. When we theorize about these animals’ capacity for welfare, we would no longer be theorizing about a single welfare subject, but multiple such subjects.[1]
    In humans, people have speculated about this possibility based on “split-brain” cases, where the corpus callosum has been wholly or partially severed (e.g., Bayne 2010; Schechter 2018). Some non-human animals, like birds, approximate the split-brain condition as the norm, and others, like the octopus, exhibit a striking lack of integration and highly decentralized nervous systems, with surprising levels of peripheral autonomy. And in the case of the octopus, Peter Godfrey-Smith suggests that “[w]e should.at least consider the possibility that an octopus is a being with multiple selves”, one for central brain, and then one for each arm (2020: 148; cf. Carls-Diamante 2017, 2019, 2022).
    What follows is a high-level summary of my full report on...
    21 min
  • EA - Creating a database for base rates by nikos
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Creating a database for base rates, published by nikos on December 12, 2022 on The Effective Altruism Forum.
    TLDR
    We are creating a database to collect base rates for various categories of events. You can find the database here and can suggest new base rate categories for us to look into here.
    Project Summary
    The base rate database project collects base rates for different categories of events and makes them available to researchers, forecasters and philanthropic organisations. Its main goals are to develop better intuitions about the potential and limitations of reference class forecasting and to provide useful information to the public. The data will enable research that enhances our understanding of the kinds of circumstances in which reference forecasting is a promising approach, what kinds of methods of reference forecasting work best, how to construct reasonable reference classes, and what potential caveats and pitfalls are. In addition to the raw data we will collect qualitative feedback on individual reference classes and on the overall process of building a base rate database, adding context to the data and developing comprehensive knowledge to build upon in the future.
    We aim to select categories of base rates in a way that makes the information we collect useful to decision makers and philanthropic organisations.
    Introduction
    If one wants to predict whether some event will happen in the future, it is often helpful to look at the past. One can ask: "Ignoring all the specifics of the current event I'm trying to predict, what would I predict just by looking at the base rate of similar events happening in the past?". This is called reference class forecasting and helps forecasters to obtain an 'outside view' on the forecasting question at hand. This outside view, of course, is usually complemented by the 'inside view': what are the specifics of the current event at hand that distinguish it from other events?
    Reference class forecasting is widely used among forecasters. To this date, however, there has been little systematic research done into how effective base rates are for forecasting future events, how they can best be used and what limitations apply. We aim to facilitate this research.
    Project outline
    Goal
    The main goal of this project is to develop a better understanding of the merits and limitations of reference class forecasting.
    A secondary goal is to collect information that may be useful for forecasters and EA stakeholders in the future.
    What we'll do
    We want to achieve our goals by
    asking experienced forecasters to compile a public database with base rates for various categories of events
    collecting qualitative feedback on the process of collecting base rates, as well as the base rates themselves
    using the database to conduct and facilitate quantitative and qualitative research, especially with regards to the performance of various reference class forecasting approaches
    inviting others (you!) to suggest base rate categories that we should look into through this form.
    Categories that we want to look into
    We intend to look into categories as diverse as
    Violent and non-violent protests that have (or have not) led to regime change
    Elections with small margins of victory
    Zoonotic spillover events
    Development of new antibiotics
    You can find a list of all the categories on our radar here. You can suggest new categories here.
    Specific research questions
    The database is meant to be a resource for anyone who is interested in reference class forecasting. Please do feel free to use it for your own research as well as to reach out to us.
    So far, we have thought of the following quantitative analyses we think may be promising:
    Comparison of the predictive performance of several reference forecasting approaches, for example:
    Naive Laplace's rule with different pri...
    6 min
  • AF - Side channels: input versus output by davidad (David A. Dalrymple)
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Side channels: input versus output, published by davidad (David A. Dalrymple) on December 12, 2022 on The AI Alignment Forum.
    This is a brief post arguing that, although "side-channels are inevitable" is pretty good common advice, actually, you can prevent attackers inside a computation from learning about what's outside.
    This implies that we could use relatively elementary sandboxing (no clock access, no networking APIs, no randomness, none of these sources of nondeterminism, and that’s about it) to prevent a task-specific AI from learning any particular facts about, say, human psychology, virology, or biochemistry—if we are careful to only provide the training process with inputs that would be just as likely in, say, an alternate universe where AI was built by octopus minds made of organosilicon where atoms obey the Bohr model.
    Self-play in Go is an example par excellence and this may be one reason why people tend to have a strong intuition that arbitrarily strong AlphaZero fails to kill you. An example that trades off more potential risk for engineering applicability would be inputs from a Newtonian physics simulator with semi-rigid objects of randomly generated shapes.
    I think it is good to be aware that
    this is a real tradeoff; we don't need to just assume that superintelligent models will be able to deduce whatever physical facts they want through pure logical induction
    taking advantage of the argument here would require being careful about data
    datasets like CommonCrawl, MassiveText, and YouTube-8M are, uh, not careful
    Arguments
    If an attacker on the outside of a computation is trying to manipulate events inside the computation, they get to use any and all physical mechanisms: they can open up the computer, stick some electrodes on the memory bus, and have at it.
    If an attacker is inside a pure mathematical function and trying to manipulate events outside the computer, well, they can influence the output of the function (or even, potentially, whether the function is tractable to compute). If the output of the computation is propagated into human minds, this influence could have cascading effects that are hard to avoid or even notice.
    If an attacker is trying to exfiltrate information from a computer, they get to use any and all physical mechanisms: they can run a very accurate clock, use a sensitive microphone, measure magnetic fields, etc. For better or worse, the nature of our physics is that everything that happens tends to radiate information in all directions and it's very difficult to fully neutralise; physical distance from attackers is the best defence, and even that is far from perfect (for example, timing information still propagates as far away as outputs do).
    If an attacker is inside a pure mathematical function and trying to exfiltrate information from outside the computer, they do not get to use any physics. That same mathematical function is, presumably, also being computed in various places all over the multiverse. Only the explicitly given inputs—and what can be inferred from them on a Solomonoff-style prior—narrow it down. If a deterministic function is computed correctly, its result cannot depend further on the specific physical properties of the computation or facts about the environment in which it was running. All the attacker can learn is that, if they are in a simulation at all, the simulator has at least a certain amount of computational resources.
    In fact, maybe they cannot be very sure of this either, since who knows what kind of fancy compiler-optimisation static-analysis techniques are out there in the universe that's simulating them. They could try doing some cryptographic operations that are supposed to be irreducibly expensive, but we tend to have a hard time proving those sorts of computational hardness assumptions. Or maybe th...
    4 min