Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maxent and Abstractions: Current Best Arguments, published by johnswentworth on May 18, 2022 on The AI Alignment Forum.
This post is not-very-distilled and doesn’t contain much background; it’s intended for people who already have the context of at least these four posts. I’m putting it up mainly as a reference for people who might want to work directly on the math of natural abstractions, and as a technical reference post.
There’s various hints that, in most real-world cases, the distribution of low-level state given high-level natural abstractions should take the form of a maximum entropy distribution, in which:
The “features” are sums over local terms, and
The high-level variables are (isomorphic to) the Lagrange multipliers
More formally: we have a low-level causal model (aka Bayes net) P[XL]=∏iP[XLi|XLpa(i)]. Given the high-level variables XH, the distribution of low-level variable values should look like
P[XL|XH]=1ZP[XL]eλT(XH)∑ifi(XLi,XLpa(i))
. i.e. the maximum-entropy distribution subject to constraints of the form E[∑ifi(XLi,XLpa(i))|XH]=μ(XH). (Note: λ, fi, and μ are all vector-valued.)
This is the sort of form we see in statistical mechanics. It’s also the form which the generalized Koopman-Pitman-Darmois (gKPD) theorem seems to hint at.
I don’t yet have a fully-satisfying general argument that this is the main form which abstractions should take, but I have two partial arguments. This post will go over both of them.
Maxent Telephone Argument
Quick recap of the Telephone Theorem: information about some variable X passes through a nested sequence of Markov blankets M1,M2,.. Information about X can only be lost as it propagates. In the limit, all information is either perfectly conserved or completely lost. Mathematically, in the limit P[X|Mn]=P[X|Fn(Mn)] for some F such that Fn(Mn)=Fn+1(Mn+1) with probability approaching 1 as n→∞; F is the perfectly-conserved-in-the-limit information carrier.
In this setup, we can also argue that the limiting distribution limn→∞P[X|Mn] should have a maxent form. (Note: this is a hand-wavy argument, not a proper proof.)
Think about how the distribution (x↦P[X=x|Mn]) transforms as we increment n by 1. We have
P[X|Mn+1]=∑MnP[X|Mn]P[Mn|Mn+1]
First key property of this transformation: it’s a convex combination for each Mn+1 value, i.e. it’s mixing. Mixing, in general, cannot decrease the entropy of a distribution, only increase it or leave it the same. So, the entropy of P[X|Mn] will not decrease with n.
When will the entropy stay the same? Well, our transformation may perfectly conserve some quantities. Since the transformation is linear, those quantities should have the form ∑Xf(X)P[X|Mn] for some f, i.e. they’re expected values. They’re conserved when E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1.
Intuitively, we’d expect the entropy of everything except the conserved quantities to strictly increase. So, we’d expect the distribution P[X|Mn] to approach maximum entropy subject to constraints of the form E[f(X)|Mn]=μ(Mn), where E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1 (at least in the limit of large n). Thus, we have the maxent form
P[X|Mn]=1ZP[X]eλT(Mn)f(X)
(Note on the P[X] in there: I’m actually maximizing relative entropy, relative to the prior on X, which is almost always what one should actually do when maximizing entropy. That results in a P[X] term. We should find that E[lnP[X]|Mn] is a conserved quantity anyway, so it shouldn’t actually matter whether we include the P[X] multiplier or not; we’ll get the same answer either way.)
Shortcomings of This Argument
Obviously it’s a bit handwavy. Other than that, the main issue is that the Telephone Theorem doesn’t really leverage the spatial distribution of information; information only propagates along a single dimension. As a result, there’s not really a way to talk about th...