Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Perform Tractable Research While Avoiding Capabilities Externalities [Pragmatic AI Safety #4], published by Dan Hendrycks on May 30, 2022 on The AI Alignment Forum.

This is the fourth post in a sequence of posts that describe our models for Pragmatic AI Safety.

We argued in our last post that the overall AI safety community ought to pursue multiple well-reasoned research directions at once. In this post, we will describe two essential properties of the kinds of research that we believe are most important.

First, we want research to be able to tractably produce tail impact. We will discuss how tail impact is created in general, as well as the fact that certain kinds of asymptotic reasoning exclude valuable lines of research and bias towards many forms of less tractable research.

Second, we want research to avoid creating capabilities externalities: the danger that some safety approaches produce by way of the fact that they may speed up AGI timelines. It may at first appear that capabilities are the price we must pay for more tractable research, but we argue here and in the next post that these are easily avoidable in over a dozen lines of research.

Strategies for Tail Impact

It’s not immediately obvious how to have an impact. In the second post in this sequence, we argued that research ability and impact is tail distributed, so most of the value will come from the small amount of research in the tails. In addition, trends such as scaling laws may make it appear that there isn’t a way to “make a dent” in AI’s development. It is natural to fear that the research collective will wash out individual impact. In this section, we will discuss high-level strategies for producing large or decisive changes and describe how they can be applied to AI safety.

Processes that generate long tails and step changes

Any researcher attempting to make serious progress will try to maximize their probability of being in the tail of research ability. It’s therefore useful to understand some general mechanisms that tend to lead to tail impacts. The mechanisms below are not the only ones: others include thresholds (e.g. tipping points and critical mass). We will describe three processes for generating tail impacts: multiplicative processes, preferential attachment, and the edge of chaos.

Multiplicative processes

Sometimes forces are additive, where additional resources, effort, or expenditure in any one variable can be expected to drive the overall system forward in a linear way. In cases like this, the Central Limit Theorem often holds, and we should expect that outcomes will be normally distributed–in these cases one variable tends not to dominate. However, sometimes variables are multiplicative or interact nonlinearly: if one variable is close to zero, increasing other factors will not make much of a difference.

In multiplicative scenarios, outcomes will be dominated by the combinations of variables where each of the variables is relatively high. For example, adding three normally distributed variables together will produce another normal distribution with a higher variance; multiplying them together will produce a long-tailed distribution.

As a concrete example, consider the impact of an individual researcher with respect to the variables that impact their work: time, drive, GPUs, collaborators, collaborator efficiency, taste/instincts/tendencies, cognitive ability, and creativity/the number of plausible concrete ideas to explore. In many cases, these variables can interact nonlinearly. For example, it doesn’t matter if a researcher has fantastic research taste and cognitive ability if they have no time to pursue their ideas. This kind of process will produce long tails, since it is hard for people to get all of the many different factors right (this is also the case in startups).

The impl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Six Dimensions of Operational Adequacy in AGI Projects, published by Eliezer Yudkowsky on May 30, 2022 on The AI Alignment Forum.

Editor's note: The following is a lightly edited copy of a document written by Eliezer Yudkowsky in November 2017. Since this is a snapshot of Eliezer’s thinking at a specific time, we’ve sprinkled reminders throughout that this is from 2017.

A background note:

It’s often the case that people are slow to abandon obsolete playbooks in response to a novel challenge. And AGI is certainly a very novel challenge.

Italian general Luigi Cadorna offers a memorable historical example. In the Isonzo Offensive of World War I, Cadorna lost hundreds of thousands of men in futile frontal assaults against enemy trenches defended by barbed wire and machine guns. As morale plummeted and desertions became epidemic, Cadorna began executing his own soldiers en masse, in an attempt to cure the rest of their “cowardice.” The offensive continued for 2.5 years.

Cadorna made many mistakes, but foremost among them was his refusal to recognize that this war was fundamentally unlike those that had come before. Modern weaponry had forced a paradigm shift, and Cadorna’s instincts were not merely miscalibrated—they were systematically broken. No number of small, incremental updates within his obsolete framework would be sufficient to meet the new challenge.

Other examples of this type of mistake include the initial response of the record industry to iTunes and streaming; or, more seriously, the response of most Western governments to COVID-19.

As usual, the real challenge of reference class forecasting is figuring out which reference class the thing you’re trying to model belongs to.

For most problems, rethinking your approach from the ground up is wasteful and unnecessary, because most problems have a similar causal structure to a large number of past cases. When the problem isn’t commensurate with existing strategies, as in the case of AGI, you need a new playbook.

I've sometimes been known to complain, or in a polite way scream in utter terror, that "there is no good guy group in AGI", i.e., if a researcher on this Earth currently wishes to contribute to the common good, there are literally zero projects they can join and no project close to being joinable. In its present version, this document is an informal response to an AI researcher who asked me to list out the qualities of such a "good project".

In summary, a "good project" needs:

Trustworthy command: A trustworthy chain of command with respect to both legal and pragmatic control of the intellectual property (IP) of such a project; a running AGI being included as "IP" in this sense.

Research closure: The organizational ability to close and/or silo IP to within a trustworthy section and prevent its release by sheer default.

Strong opsec: Operational security adequate to prevent the proliferation of code (or other information sufficient to recreate code within e.g. 1 year) due to e.g. Russian intelligence agencies grabbing the code.

Common good commitment: The project's command and its people must have a credible commitment to both short-term and long-term goodness. Short-term goodness comprises the immediate welfare of present-day Earth; long-term goodness is the achievement of transhumanist astronomical goods.

Alignment mindset: Somebody on the project needs deep enough security mindset plus understanding of AI cognition that they can originate new, deep measures to ensure AGI alignment; and they must be in a position of technical control or otherwise have effectively unlimited political capital. Everybody on the project needs to understand and expect that aligning an AGI will be terrifically difficult and terribly dangerous.

Requisite resource levels: The project must have adequate resources to compete at the ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Will working here advance AGI? Help us not destroy the world!, published by Yonatan Cale on May 29, 2022 on LessWrong.

I often talk to developers who prefer not destroying the world by accident (specifically by accelerating AGI risk), but neither them nor me can decide if specific companies qualify for this.

Could someone knowledgable help? A few short replies could probably change someone's career decisions

Can you help with future questions?

Please subscribe to the this comment. I'll reply to it only when there's a new open question.

Thank you!

Adding: Reply anonymously here

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing EAecon: Community-Building Project, published by Brian Jabarian on May 30, 2022 on The Effective Altruism Forum.

EAecon: Background

For my first post on the forum, I am very excited to share that my community-building project, EAecon, received a major grant from CEA Events.

I am grateful to my grantmaker, Ollie Base, Will MacAskill, Rossa O'Keefe O'Connor, and my EA economist friends and colleagues (in particular, David, Charlotte, Ruru, Ben G., Ben T., Phil, Zach, Danny, Loren) for their support in this project.

I am a Ph.D. Candidate in economics at PSE, writing a thesis titled "Economics of Complex Uncertainty" and a Global Priority Fellow in Economics at Forethought Foundation (18') and was a visitor to GPI (18'). On the side, I also just submitted a doctoral thesis in philosophy at Sorbonne, titled "Operationalizing Normative Uncertainty".

EAecon: Why

I got interested in EA through my Ph.D. topic and started to interact with EA based on my first-year paper. Since then, I have benefited from the EA community, alongside three main dimensions:

Building long-term connections with other EA economists, academics, and non-academic actors.

Exploring an original and useful research agenda for future generations thanks to generous funding.

Discussing freely and openly all academic and non-academic topics with all my EA community.

I am about going to finish my doctoral studies. I thought younger econ generations could also benefit from these three dimensions, without having to wait to be far advanced in their academic career or to have found a very specific topic on their own that exactly matched GPR and EA researches, but rather connecting, exploring, and discussing at an early stage and progressing from there.

EAecon: Goals

This is what EAecon aims to provide: reinforcing the community of EAEconomists across all EA-orgs and welcoming new economists on board, sharing specific EA jobs opportunities and career advice for EA economists, and organizing events for advanced EAecon undergraduates, master, and early Ph.D.

GPREcon Learning: presenting EAEcon academic work.

Academic Networking: facilitating cross-level networking (between juniors and seniors), matching people for collaboration (RA/co-authors).

Pitching Projects: screening good EAEcon academic, charity, and social entrepreneurship projects through pitching games, and meeting grantmakers and funders.

Career Development: career sessions on how to write EAEcon papers and to apply to grad school, discovering academic and non-academic EA jobs available for economists, and meeting leading EA orgs and actors.

EAecon: Community Formats

We aim to develop EAecon through a hybrid format: building an online community and organizing different types of physical events of different capacities throughout the year (EAecon Retreats and EAecon Meetings).

Upcoming Event: EAecon Retreat 2022

Based on these goals, we are launching our first event: EAecon Retreat 2022. You can read more about it (when, application,) here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We should expect to worry more about speculative risks, published by Ben Garfinkel on May 29, 2022 on The Effective Altruism Forum.

(This is a repost of a short-form, which I realized might be worth making into its own post. It’s partly inspired by Greg Lewis’s recent post “Rational Predictions Often Update Predictably.”)[1]

The existential risk community’s level of concern about different possible risks is correlated with how hard-to-analyse these risks are. For example, here is The Precipice’s ranking of the top five most concerning existential risks:

Unaligned artificial intelligence[2]

Unforeseen anthropogenic risks (tied)

Engineered pandemics (tied)

Other anthropogenic risks

Nuclear war (tied)

Climate change (tied)

This isn’t surprising.

For a number of risks, when you first hear and think a bit about them, it’s reasonable to have the reaction “Oh, hm, maybe that could be a huge threat to human survival” and initially assign something on the order of a 10% credence to the hypothesis that it will by default lead to existentially bad outcomes. In each case, if we can gain much greater clarity about the risk, then we should think there’s about a 90% chance this clarity will make us less worried about it. We’re likely to remain decently worried about hard-to-analyze risks (because we can’t get greater clarity about them) while becoming less worried about easy-to-analyze risks.

In particular, our level of worry about different plausible existential risks is likely to roughly track our ability to analyze them (e.g. through empirical evidence, predictively accurate formal models, and clearcut arguments).

Some plausible existential risks also are far easier to analyze than others. If you compare 80,000 Hours’ articles on climate change and artificial intelligence, for example, then I think it is pretty clear that people analyzing existential risks from climate change simply have a lot more to go on. When we study climate change, we can rely on climate models that we have reason to believe have a decent amount of validity. We can also draw on empirical evidence about the historical effects of previous large changes in global temperature and about the ability of humans and other species to survive under different local climate conditions. As a conceptual foundation, we are also lucky to have a set of precise and scientifically validated concepts (e.g. “temperature” and "sea-level") that we can use to avoid ambiguity in our analysis. And so on.

We’re in a much worse epistemic position when it comes to analyzing the existential risk posed by misaligned AI: we’re reliant abstract arguments that use ambiguous concepts (e.g. “objectives” and “intelligence”), rough analogies, observations of the behaviour of present-day AI systems (e.g. reinforcement learners that play videogames) that will probably be very different than future AI systems, a single datapoint (the evolution of human intelligence and values) that has a lot of important differences with the case we’re considering, and attempts to predict the incentives and beliefs of future actors in development scenarios that are still very opaque to us. Even if misaligned AI actually poses very little risk to continued human survival, then it’s hard to see how we could become really confident of that.

Some upshots:

The fact that the existential risk community is particularly worried about misaligned AI might largely reflect the fact that it’s hard to analyze risks from misaligned AI.

Nonetheless, even if the above possibility is true, it doesn't at all follow that the community is irrational to worry more about misaligned AI than other potential risks. It’s completely coherent to have something like this attitude: “If I could think more clearly about the risk from misaligned AI, then I would probably come to realize it’s not a ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why You Should Earn to Give in Tulsa, OK, USA, published by NicoleJaneway on May 29, 2022 on The Effective Altruism Forum.

TL;DR: Tulsa Remote is a one-year program that offers $10,000 (after tax) to people who are willing to move to Tulsa and work remotely here.

To qualify, you must be eligible to work in the US and have a remote job or full-time self employment outside the state of Oklahoma. If you apply and get accepted, you must move to Tulsa within 12 months.

Interested EAs could use the stipend to defray the cost of moving and put the rest into saving or donation. Tulsa has all the things you would reasonably want from a mid-size city. There's a burgeoning tech scene and a lot of entrepreneurial excitement.

Acceptance into Tulsa Remote includes three years free access to a well-apportioned coworking space. It is extremely easy to meet people there, in the bustling Slack community, and through Tulsa Remote events.

Due to the low cost of living (83.2 versus US average score of 100), Tulsa is a great place to earn to give or work towards FIREA (Financial Independence, Retire, Effective Altruism, which is an acronym I just made up to describe the career of someone who has generated enough savings that they can comfortably shift to doing EA work full time).

Tulsa is the perfect place for this kind of remote work incentive program. Its infrastructure was mostly built during a brief mid-century oil boom, and the population never grew to projected levels. Tulsa Remote members grow the city's tax base with relatively less negative impact than in a space constrained city like Washington, DC where gentrification would be a concern.

It's also worth noting that the program is largely privately funded, so it's not like the money is directly coming out of tax-payers' pockets. I've been asking around and Tulsa Remote seems to be looked on favorably by the locals as a path toward economic development.

Personal experience: In January 2022, I started in the Tulsa Remote program as a frontend engineer. My apartment, which I found through the TR Slack, is a two bedroom located in Midtown that I split with another transplant. My rent is $425, which includes all utilities. This is about a third of what I paid for a comparably sized apartment in Washington, DC (cost of living 152.1 versus US average score of 100).

I was surprised by how quickly I found friends and fun things to do after work and on weekends. I go into the coworking space about once every two weeks, but I always find it to be an effective work environment. I've gotten involved in Code for Tulsa and feel a sense of community at the local YWCA where I workout in the mornings.

I bike commute and do not own a car. If I need to get to the airport or run a major errand, I'll take an Uber or Lyft, which is much cheaper than car ownership.

Reflections on the bike commute: I find it kind of funny that folks assume you must have a car to live in Tulsa. When I tell people I've just biked to their event, they look stunned. Getting covid tested in the winter was interesting because I kept being told to remain in my vehicle, and I had to keep explaining that I'd biked. I think the bike commuter infrastructure will continue to improve over time as more people move here from out of state and show that it's possible to live outside downtown without a car.

Culturally, Tulsa is very family oriented. It seems like a lot of the native Tulsans my age are married with children. Some businesses close surprisingly early (6pm) and/or aren't open on the weekends, which I guess reflects the fact that people may not work as much and can run errands during the work day.

This is highly subjective, but from my perspective, the only major downside to Tulsa that's truly insuperable is lack of access to nature. Between Rock Creek Park and Kenilworth Gar...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reshaping the AI Industry, published by Thane Ruthenis on May 29, 2022 on The AI Alignment Forum.

The wider AI research community is an almost-optimal engine of apocalypse. The primary metric of a paper's success is how much it improves capabilities along concrete metrics, publish-or-perish dynamics supercharge that, the safety side of things is neglected to the tune of 1:49 rate of safety to other research, and most results are made public so as to give everyone else in the world a fair shot at ending it too.

It doesn't have to be this way. The overwhelming majority of the people involved do not actually want to end the world. There must exist an equilibrium in which their intentions match their actions.

Even fractionally shifting the status quo towards that equilibrium would have massive pay-offs, as far as timelines are concerned. Fully overturning it may well constitute a sufficient condition for humanity's survival. Yet I've seen precious little work done in this area, compared to the technical questions of AI alignment. It seems to be picking up in recent months, though — and I'm happy to contribute.

This post is an attempt at a comprehensive high-level overview of the tactical and strategic options available to us.

1. Rationale

Why is it important? Why is it crucial?

First. We, uh, need to make sure that if we figure alignment out people actually implement it. Like, imagine that tomorrow someone comes up with a clever hack that robustly solves the alignment problem... but it increases the compute necessary to train any given ML model by 10%, or it's a bit tricky to implement, or something. Does the wider AI community universally adopt that solution? Or do they ignore it? Or do the industry leaders, after we extensively campaign, pinky-swear to use it the moment they start training models they feel might actually pose a threat, then predictably and fatally misjudge that?

In other words: When the time comes, we'll need to convince people that safety is important enough to fuss around a bit for its sake. But if we can't convince them to do that now, what would change then?

I suppose having a concrete solution, instead of vague prophecies of doom, would give us more credibility. But... would it, really? And what if we won't have a concrete solution even then, just a bunch of weird heuristics that may nonetheless measurably improve our odds?

The latter seems reasonably likely, too. As this excellent post points out, most of the contemporary deep-learning progress comes from "messy bottom-up atheoretical empirical tinkering". If AGI comes from DL, it's plausible that, even if we arrive at the solution to alignment from mathematical foundations, the actual implementation will take the form of messy hacks. Ones that will probably need to be fine-tuned for any given model architecture. And given the no-fire-alarm principle, to be safe, we'll need to ensure that any sufficiently big model is only run with these hacks built-in.

If any given AI researcher is still not taking alignment seriously by then, how will we make them bother with all of that every time they run an experiment? How will we ensure they don't half-ass it even once?

Second. If that figure I quoted, the 49:1 ratio, is even remotely correct, there's plenty of room to upscale our research efforts. Imagine if every researcher started spending 2% more of their time thinking about alignment. That'd double the researcher-hours spent on the problem!

Which doesn't directly translate into 2x progress, I'll grant. Given the field's pre-paradigmic status, the returns to scale might be relatively small... but by no means negligible. Even if we don't necessarily have research directions clearly outlined, having much more people stumbling around in the dark still increases the chances of bumping into something useful....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Revisiting the karma system, published by Arepo on May 29, 2022 on The Effective Altruism Forum.

Epistemic status: grumpy, not committed.

There was quite a lot of discussion of the karma system in the comments to the forum 2.0 announcement, but it didn’t seem very conclusive and as far as I know, hasn’t been publicly discussed since.

That seems enough concern that it’s worth revisiting. My worries are:

Karma concentration exacerbates groupthink by allowing a relatively small number of people to influence which threads and comments have greatest visibility

It leads to substantial karma inflation over time, strongly biasing recent posts to get more upvotes

Point 1) was discussed a lot in the original comments. The response was that because it’s a pseudo-logarithmic scale, this shouldn’t be much of a concern. I think we now have reasons to be sceptical of this response:

There are plenty of people with quite powerful upvotes now - mine are currently worth 5 karma, very close to 6, and I’ve posted less than a dozen top level posts. That will give me 3-6 times the strong voting power of a forum beginner, which seems like way too much.

While top level posts are the main concern, comments get a much lower level of interest, so the effect of one or two strong votes can stand out much more if you’re skimming through them.

The people with the highest karma naturally tend to be the most active users, who’re likely already the most committed EAs. This means we already have a natural source of groupthink (assuming the more committed you are to a social group the more likely you are to have bought into any given belief it tends to hold). So groupthinky posts would already tend to get more attention, and having these active users have greater voting power multiplies this effect.

Point 2) is confounded by the movement and user base having grown, so a higher proportion of posts having been made in later years, when there were more potential upvoters. Nonetheless, unless you believe that the number of posts has proliferated faster than the number of users (so that karma is stretched evenly), it seems self-evident that there is at least some degree of karmic inflation.

So my current stance is that, while the magnitude of both effects is difficult to gauge because of complementary factors, both effects are probably in themselves net negative, and therefore things we should not be using tools to complement - we might even want to actively counteract them. I don’t have a specific fix in mind, though plenty were discussed in the comments section linked above. This is just a quick post to encourage discussion of alternative. so over to you, commenters!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What have been the major "triumphs" in the field of AI over the last ten years?, published by lc on May 28, 2022 on LessWrong.

Contrary to what seems to be the experience of others, when I'm talking to normies about AI safety, the most common dissenting reaction I get isn't that they think AI will be controllable, or safe. Convincing them that computers with human-level intelligence won't have their best interests at heart by default tends to be rather easy.

More often the issue is that AGI seems very far away, and so they don't think AI safety is particularly important. Even when they say that's not their sticking point, alerting them to the existence of tools like GPT3 tends to impart a sense of urgency and "realness" to the problem that makes them take it a bit more seriously. There's this significant qualitative difference in the discussion before and after I show them all of the crazy things OpenAI and DeepMind have built.

I have a general sense that progress has been speeding up, but I'd like to compile a list of relevant highlights. Anybody here willing to help?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Distributed Decisions, published by johnswentworth on May 29, 2022 on The AI Alignment Forum.

Consider two prototypical “agents”: a human, and a company.

The human is relatively centralized and monolithic. As a rough approximation, every 100 ms or so observations flow into the brain from the eyes, ears, etc. This raw input data updates the brain’s world-model, and then decisions flow out, e.g. muscle movements. This is exactly the sort of “state-update model” which Against Time In Agent Models criticized: observations update one central internal state at each timestep, and all decisions are made based on that central state. It’s not even all that accurate a model for a human, but let’s set that aside for now and contrast it to a more obviously decentralized example.

In a company, knowledge and decisions are distributed. A cashier sees and hears customers in the store, and interacts with them in order to sell things. Meanwhile, a marketing editor tweaks some ad copy. Each mostly makes decisions based on their local information; most of that local information is not propagated to other decision-makers. Observations don’t update a single centralized state which informs all decisions. Instead, different decisions have different input information from different sources.

In Optimization at a Distance, I suggested a mental picture of agents kinda like this:

It’s like a phased array: there’s lots of little actions distributed over space/time, all controlled in such a way that their influence can add up coherently and propagate over a long distance to optimize some far-away target. Optimization at a Distance mainly emphasized the “height” of this picture, i.e. the distance between optimizer and target. This post is instead about the “width”: not only are the actions far from the optimization target, the actions themselves are also distributed in spacetime and potentially far apart from each other.

Contrast: Bayesian Updates

Suppose I want to watch my favorite movie, 10 Things I Hate About You, in the evening. To make this happen, I do some optimization - I steer myself-in-the-evening and my-immediate-environment-in-the-evening into the relatively small set of states in which I’m watching the movie. Via the argument in Utility Maximization = Description Length Minimization, we should expect that I approximately-act-as-though I’m a Bayesian reasoner maximizing some expected utility over myself-in-the-evening and my-immediate-environment-in-the-evening. (Note that it’s a utility function over myself-in-the-evening and my-immediate-environment-in-the-evening, not just any old random utility function; something like e.g. a rock would not be well-described by such a utility function.)

While arranging my evening, I may perform some Bayesian updates. Maybe I learn that the movie is not available on Netflix, so I ask a friend if they have a copy, then check Amazon when they don’t. This process is reasonably well-characterized as me having a centralized model of the places I might find the movie, and then Bayes-updating that model each time I learn another place where I can/can’t find it. (If I had checked Netflixed, then asked my friend, then checked Netflix again because I forgot whether it was on Netflix, that would not be well-modeled as Bayesian updates.)

By contrast, imagine that myself and some friends are arranging to watch 10 Things I Hate About You in the evening. I check to see if the movie is on Netflix, and at the same time my friend checks their parents’ pile of DVDs. My friend doesn’t find it in their parents’ DVD pile, and doesn’t know I already checked Netflix, so they also check Netflix. My friends and I, as a system, are not well-modeled as Bayesian updates to a single central knowledge-state; otherwise we wouldn’t check Netflix twice. And yet, it’s not obviously sub...