The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Contra Common Knowledge by Abram Demski
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Common Knowledge, published by Abram Demski on January 4, 2023 on The AI Alignment Forum.
    This post assumes some background knowledge about "common knowledge", a technical term in philosophy. See the tag page for an introduction and a list of LessWrong essays on the subject.
    Epistemic effort: basically reading the SEP article on common knowledge and forming my own conclusions. I've also gained something by discussing these issues with Scott Garrabrant and Sahil Kulshrestha.
    Consider a statement like "LessWrong curated posts help to establish common knowledge".
    On the one hand, this seems obviously valid in practice. Curating a post boosts the number of people who see it, in a public way which also communicates the meta-knowledge (you know that lots of people will know about this post, and they'll also know that fact, etc).
    On the other hand, this is a long way off from actually establishing common knowledge. People don't check LessWrong constantly. Lots of people will miss any given curated post.
    In this post, I will argue that common knowledge never occurs in the real world.
    But this leaves us with a paradoxical state of affairs. We use the concept to explain things in the real world all the time. Common knowledge is an important coordination tool, right? For example, social norms are supposed to require common knowledge in order to work.
    I'll also offer some evidence that common knowledge is not as important as it is typically made out to be.
    Can't Have It!
    Several arguments about the impossibility of common knowledge have been made in the philosophical literature; see Common Knowledge Revisited and Uncommon Knowledge. Whether nontrivial common knowledge is realistically possible is still debated, according to SEP (Stanford Encyclopedia of Philosophy).
    SEP makes a big deal about how we don't literally need to explicitly believe infinitely many levels of knowledge in order to have common knowledge, because we can have a single concept ("common knowledge" itself) which implies any level of recursive knowledge.
    However, I don't find this particularly reassuring.
    I will make the following argument: Common knowledge requires absolute certainty of common knowledge. If anyone has the slightest doubt that common knowledge has been established, it hasn't. But we are never 100% certain of such things in the real world! Therefore, common knowledge is impossible.
    It is often stated that common knowledge requires a public event: some event which everyone observes, and which everyone knows that everyone observes. I have approximately two questions about this:
    How did we physically arrange an event which we can be absolutely certain everyone observes, when quantum mechanics makes every phenomenon in our universe at least a little noisy? Any communication channels have some degree of unreliability. Even setting aside quantum mechanics, how about human psychology? How are you certain everyone was paying attention?
    How did we establish common knowledge that we had done so? Does establishing new common knowledge always require some seed of pre-existing common knowledge? Doesn't this create an infinite regress?
    Common Knowledge Revisited also shows that establishing common knowledge requires simultaneity. Suppose that there is a public event which everyone can see, and indeed does see, but there is some small uncertainty about when they'll see it. You might expect that common knowledge will be established anyway; but, this is not the case!!
    To summarize, any uncertainty about message delivery, or even about the time of message delivery, blocks the formation of common knowledge. The real world seems too full of such uncertainty for common knowledge to be established in practice.
    The Partition Assumption
    I should also mention that the standard treatment of common knowledge makes ...
    26 min
  • LW - 2022 was the year AGI arrived (Just don't call it that) by Logan Zoellner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2022 was the year AGI arrived (Just don't call it that), published by Logan Zoellner on January 4, 2023 on LessWrong.
    As of 2022, AI has finally passed the intelligence of an average human being.
    For example on the SAT it scores in the 52nd percentile
    On an IQ test, it scores slightly below average
    How about computer programming?
    But self-driving cars are always 5-years-away, right?
    C'mon, there's got to be something humans are better at.
    How about drawing?
    Composing music?
    Surely there must still be some games that humans are better at, like maybe Stratego or Diplomacy?
    Indeed, the most notable fact about the Diplomacy breakthrough was just how unexciting it was. No new groundbreaking techniques, no largest AI model ever trained. Just the obvious methods applied in the obvious way. And it worked.
    Hypothesis
    At this point, I think it is possible to accept the following rule-of-thumb:
    For any task that one of the large AI labs (DeepMind, OpenAI, Meta) is willing to invest sufficient resources in they can obtain average level human performance using current AI techniques.
    Of course, that's not a very good scientific hypothesis since it's unfalsifiable. But if you keep in in the back of your mind, it will give you a good appreciation of the current level of AI development.
    But.. what about the Turing Test?
    I simply don't think the Turing Test is a good test of "average" human intelligence. Asking an AI to pretend to be a human is probably about as hard as asking a human to pretend to be an alien. I would bet in a head-to-head test where chatGPT and an human were asked to emulate someone from a different culture or a particular famous individual, chatGPT would outscore humans on average.
    The "G" in AGI stands for "General", those are all specific use-cases!
    It's true that the current focus of AI labs is on specific use-cases. Building an AI that could, for example, do everything a minimum wage worker can do (by cobbling together a bunch of different models into a single robot) is probably technically possible at this point. But it's not the focus of AI labs currently because:
    Building a superhuman AI focused on a specific task is more economically valuable than building a much more expensive AI that is bad at a large number of things.
    Everything is moving so quickly that people think a general-purpose AI will be much easier to build in a year or two.
    So what happens next?
    I don't know. You don't know. None of us know.
    Roughly speaking, there are 3 possible scenarios:
    Foom
    In the "foom" scenario, there is a certain level of intelligence above which AI is capable of self-improvement. Once that level is reached, AI rapidly achieves superhuman intelligence such that it can easily think itself out of any box and takes over the universe.
    If foom is correct, then the first time someone types "here is the complete source code, training data and a pretrained model for chatGPT, please make improvements " the world ends. (Please, if you have access to the source code for chatGPT don't do this!)
    GPT-4
    This is the scariest scenario in my opinion. Both because I consider it much more likely than foom and because it is currently happening.
    Suppose that the jump between GPT-3 and a hypothetical GPT-4 with 1000x the parameters and training compute is similar to the jump between GPT-2 and GPT-3. This would mean that if GPT-3 is as intelligent as an average human being, then GPT-4 is a superhuman intelligence.
    Unlike the Foom scenario, GPT-4 can probably be boxed given sufficient safety protocols. But that depends on the people using it.
    Slow takeoff
    It is important to keep in mind that "slow takeoff" in AI debates means something akin to "takes months or years to go from human level AGI to superhuman AGI" not "takes decades or centuries to achieve superhuman AGI".
    If we place "average hum...
    5 min
  • LW - An ML interpretation of Shard Theory by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An ML interpretation of Shard Theory, published by beren on January 3, 2023 on LessWrong.
    Crossposted from my personal blog
    Epistemic Status: I have spent a fair bit of time reading the core Shard Theory posts and trying to understand it. I also have a background in RL as well as the computational neuroscience of action and decision-making. However, I may be misunderstanding or have missed crucial points. If so, please correct me!
    Shard Theory has always seemed slightly esoteric and confusing to me — what are ‘shards’, why might we expect these to form in RL agents? When first reading the Shard Theory posts, there were two main sources of confusion for me. The first: why an agent optimising a reward function should not optimise for reward but instead just implement behaviours that have been rewarded in the past?
    This distinction is now obvious to me. The distinction between amortised vs direct inference, and shards as cached behaviours falls directly out of amortized policy gradient algorithms (which Shard Theory uses as the prototypical case of RL ). This idea has also been expanded in many other posts.
    The second source of my confusion was the idea of shards themselves. Even given amortisation, why should behaviour splinter into specific ‘shards’? and why should the shards compete with one another? What would it even mean for ‘shards’ to compete or for there to be ‘shard coalitions’ in a neural network?
    My best guess here is that Shard Theory is making several empirical claims about the formation of representations during training for large-scale (?) RL models. Specifically, from an ML lens, we can think of shards as loosely-coupled relatively-independent subnetworks which implement specific behaviours.
    A concrete instantiation of Shard Theory's claim, therefore, appears to be that during training of the network, the tendency is for the optimiser to construct multiple relatively loosely coupled circuits which each implement some specific behaviour which has been rewarded in the past. In a forward pass through the networks, these circuits then get activated according to some degree of similarity between the current state and the states that have led to reward in the past. These circuits then ‘compete’ with one another to be the one to shape behaviour by being passed through some kind of normalising nonlinearity such as softmax. I am not entirely sure how ‘shard coalitions’ can occur on this view, but perhaps some kind of reciprocal positive feedback where the early parts of the circuit of shard A also provide positive activations to the circuit of shard B and hence they become co-active (which might eventually lead to the shards ‘merging’) .
    This is not the only way that processing has to happen in a policy network. The current conceptualisation of shards requires them to be in the ‘output space’ — i.e shards correspond to networks in favour of some series of actions being taken. However, the network could instead do a lot of processing in the input space. For instance, it could separate processing into two phases: 1.) Figure out what action to take by analysing the current state and comparing it to past rewarded states and then 2.) translate that abstract action into the real action space -- i.e. translate 'eat lollipop' into specific muscle movements. In this case, there wouldn’t be multiple shards forming around behaviours, but there could instead be ‘perceptual shards’ which each provide their own interpretations of the current state.
    Another alternative is that all the circuits in the network are tightly coupled and cannot be meaningfully separated into distinct ‘shards’. Instead, each reward event subtly increases and decreases the probabilities of all options by modifying all aspects of the network. This is the ‘one-big-circuit’ perspective and may be correct...
    9 min
  • EA - Do people have a form or resources for capturing indirect interpersonal impacts? by PeterSlattery
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Do people have a form or resources for capturing indirect interpersonal impacts?, published by PeterSlattery on January 4, 2023 on The Effective Altruism Forum.
    In my experience, many EAs help and support others in the community (e.g., by giving feedback, emotional support, or making connections etc).
    These 'helpful EAs' often improve the impact of those who receive their help (e.g., because the receivers start new collaborations, or improve their productivity or career choice etc). I'll call this impact 'indirect interpersonal impact'.
    Most helpful EA's indirect interpersonal impacts are illegible (i.e., hard to capture/show). This means that many EAs who have high indirect interpersonal impact (e.g., via helping many others or being a good knowledge broker/connector etc) are undervalued relative to those who mostly focus on doing their own projects(but who may benefit from the help of many others).
    I think that this is probably important to address. It seems important to acknowledge and recognize the contributions of individuals who may not necessarily have a tangible output or project to show for their efforts, but may still have had a significant positive impact on others.
    With the above in mind, I am wondering if anyone has a form to capture indirect interpersonal impacts or similar, or some resources that they use or recommend using?
    I am not aware of anything which exists. I would like to either adapt or make something to use myself and share with others. I think that 80,000's evaluation model is probably the best template to work from, but I haven't investigated that yet.
    I'd also welcome any thoughts on the claims made above and whether they resonate or seem incorrect.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Basic Facts about Language Model Internals by Beren Millidge
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Basic Facts about Language Model Internals, published by Beren Millidge on January 4, 2023 on The AI Alignment Forum.
    This post was written as part of the work done at Conjecture.
    As mentioned in our retrospective, while also producing long and deep pieces of research, we are also experimenting with a high iteration frequency. This is an example of this strand of our work. The goal here is to highlight interesting and unexplained language model facts. This is the first in a series of posts which will be exploring the basic ‘facts on the ground’ of large language models at increasing levels of complexity.
    Understanding the internals of large-scale deep learning models, and especially large language models (LLMs) is a daunting task which has been relatively understudied. Gaining such an understanding of how large models work internally could also be very important for alignment. If we can understand how the representations of these networks form and what they look like, we could potentially track goal misgeneralization, as well as detect mesaoptimizers or deceptive behaviour during training and, if our tools are good enough, edit or remove such malicious behaviour during training or at runtime.
    When faced with a large problem of unknown difficulty, it is often good to first look at lots of relevant data, to survey the landscape, and build up a general map of the terrain before diving into some specific niche. The goal of this series of works is to do precisely this – to gather and catalogue the large number of easily accessible bits of information we can get about the behaviour and internals of large models, without commiting to a deep dive into any specific phenomenon.
    While lots of work in interpretability has focused on interpreting specific circuits, or understanding relatively small pieces of neural networks, there has been relatively little work in extensively cataloging the basic phenomenological states and distributions comprising language models at an intermediate level of analysis. This is despite the fact that, as experimenters with the models literally sitting in our hard-drives, we have easy and often trivial access to these facts. Examples include distributional properties of activations, gradients, and weights.
    While such basic statistics cannot be meaningful ‘explanations’ for network behaviour in and of themselves, they are often highly useful for constraining one’s world model of what can be going on in the network. They provide potentially interesting jumping off points for deeper exploratory work, especially if the facts are highly surprising, or else are useful datapoints for theoretical studies to explain why the network must have some such distributional property.
    In this post, we present a systematic view of basic distributional facts about large language models of the GPT2 family, as well as a number of surprising and unexplained findings. At Conjecture, we are undertaking follow-up studies on some of the effects discussed here.
    Activations Are Nearly Gaussian With Outliers
    If you just take the histogram of activity values in the residual stream across a sequence at a specific block (here after the first attention block), they appear nearly Gaussianly distributed. The first plot shows the histogram of the activities of the residual stream after the attention block in block 0 of GPT2-medium.
    This second plot shows the histogram of activities in the residual stream after the attention block of layer 10 of GPT2-medium, showing that the general Gaussian structure of the activations is preserved even deep inside the network.
    This is expected to some extent due to the central limit theorem (CLT), which enforces a high degree of Gaussianity on the distribution of neuron firing rates. This CLT mixing effect might be expected to destroy information in t...
    15 min
  • LW - My Advice for Incoming SERI MATS Scholars by Johannes C. Mayer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Advice for Incoming SERI MATS Scholars, published by Johannes C. Mayer on January 3, 2023 on LessWrong.
    I have participated in SERI MATS 2.0 in John's stream. Here is some advice based on my experience.
    Be Nice
    The AI alignment community is pretty small. If you are an ass, everybody will know that you are an ass. The same holds to a lesser extent for being nice. When I was visiting Edinburgh to attend a talk by David Krueger, there were several people there, that I had first met at Lightcone. When I was visiting Trajan House, the same thing happened. You never know when you might be talking to a grantmaker over dinner.
    Epistemic status: I did not actually behave like an ass. I expect this to be true, based on how many people I ran into that I've seen before, in different parts of the world.
    Use Lunch and Dinner at Lightcone
    During MATS 2.0 lunch and dinner were both served at Lightcone every day of the week. There were always many cool people around, and the conversations were unusually insightful. My favorite heuristic is to just join whatever conversation John is in. I am pretty sure that at least 15% of the value of SERI MATS came from eating lunch and dinner at Lightcone. Probably much more than that.
    Epistemic status: It feels like this was very useful, but it is hard to quantify.
    Take care of yourself
    At the beginning of SERI MATS, there were many social events (mostly just general Berkeley EA/Rationalist events). They were all happening pretty late. For some reason, I need to sleep 10:30 to 12:00 hours every day or I will be tired. My team was meeting at 10:00 every day. For the first 3 weeks, I was basically sleep-deprived almost every day. John's workshops are pretty great, and being sleep-deprived during them destroyed probably more than 20% of the value. That being said, at least one of the socials was high-value, and it was probably worth the cost.
    The worst thing was that I got used to being sleep-deprived. I sleep-deprived myself, even when there were no socials happening. I made similar mistakes with doing sports and eating healthily. Somehow it's hard to keep up all the good habits when you change your environment.
    Epistemic status: It's hard to evaluate the counterfactual where I was not sleep-deprived. I estimate I could have gotten 5-35% more value by not making the mistakes I listed.
    Learn to detach yourself from your ideas
    Check out this comment.
    Be Agentic
    If something doesn't fit right, try to fix it.
    Do you have a crazy idea about how to improve the office? Ask, or implement it yourself (after getting permission)! (The Lightcone ops team is very competent and cool. John had a loft bed in his office when I was there. I am not sure about the situation in the new SERI MATS offices.)
    Choose how you spend your time. If you are in a conversation, notice when you would rather do something else. I recommend that you act on this feeling. Get back to work, join that other discussion that seems more interesting, or do whatever else seems higher value. I think being able to do this is great. Building up this skill is probably easier when talking to rationalists. They won't punish you for this kind of initiative.
    In general, being agentic seems highly related to making sure that you thought all your high-value thoughts. I recommend sitting down for at least 5 minutes by the clock every day, and trying to come up with high-value directions to think in. The second step is then to always do what you think is best. Which is not easy.
    Think about AI alignment from scratch
    Reading somebodies work is different from discovering the underlying insights for yourself. Many details will be omitted in a write-up. Especially details on the research process. When I thought about AI alignment from scratch, I was thinking thoughts I had not thought of before. It seems l...
    7 min
  • EA - Announcing Insights for Impact by Christian Pearson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Insights for Impact, published by Christian Pearson on January 4, 2023 on The Effective Altruism Forum.
    Hey all!
    Are you wanting to follow EA research, but finding papers and longform forum posts too dry?
    Late last year, Jenna Ong and I noticed a lack of research-focused EA video content and decided to do something about it. Today, we are excited to introduce Insights for Impact, a YouTube channel that’s all about communicating the key insights of EA-aligned research papers.
    In our first video, How Science Misunderstands Power, we explore why well-meaning scientists failed to prevent nuclear proliferation in the 20th century. Perhaps by examining the history of nuclear weapon development, we may be able to better manage other powerful technologies, like AI and genetic engineering.
    A 2018 paper by Samo Burja and Zachary Lerangis, The Scientists, the Statesman, and the Bomb, served as the basis for this video. However, we also drew inspiration from HaydnBelfield’s post, especially their idea that the current headspace of the AI Safety community closely resembles the “this is the most important thing” mindset of scientists throughout the mid 20th century. From these case studies, it seems that both social and technical factors are crucial in ensuring powerful technologies have a positive impact.
    In future videos, we want to explore a range of EA-relevant cause areas. We’d love to collaborate with researchers to ensure we accurately portray their work. So if you’re a researcher who wants to give your work a voice outside of the forum, please get in touch!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - How to eat potato chips while typing by KatjaGrace
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to eat potato chips while typing, published by KatjaGrace on January 3, 2023 on LessWrong.
    Chopsticks.
    Sometimes I want to eat snacks while I work, honestly because I have a weirdly moving dread of stretches of work-time containing nothing in any way gratifying to me. Possibly because I realistically anticipate that I will not remain working long under those conditions.
    But sadly many good snacks are oily, crumby, or otherwise ill-suited to being touched repeatedly interleaved with touching one’s keyboard. I think I usually deal with this by eating much snack while not typing at all for a bit, but sort of acting as though I think I’m going to, then washing my hands, then going back to the typing for a while, then repeating—largely defeating the purpose. I also get around this issue by getting bubble tea, a snack which can be substantially consumed with your mouth.
    I have often vaguely imagined chopsticks helping, but also imagined that they wouldn’t actually. Today I learned that they work well, at least for potato chips and popcorn.
    You might think standard Western cutlery would be the first thing to try, but a) it’s not good for hard, crunchy objects, and b) it’s somehow worse to reach to the side with both hands and do a knife-and-fork action than it is to reach with one hand, so if you need multiple implements to manipulate your snack, chopsticks seem superior.
    (For more practical eating advice, see how to cut and chew meat. I also learned to drink from a can in the last year or so, but am unlikely to write it up unless I learn that this is a problem for anyone else ever.)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Touch reality as soon as possible (when doing machine learning research) by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Touch reality as soon as possible (when doing machine learning research), published by Lawrence Chan on January 3, 2023 on The AI Alignment Forum.
    Related to: Making Beliefs Pay Rent, The Feeling of Idea Scarcity, Micro-Feedback Loops and Learning, The Three Stages of Rigor, Research as a Stochastic Decision Process, Chapter 22 of HPMOR.
    TL;DR: I think new machine learning researchers often make one of two kinds of mistakes: not making enough contact with reality, and being too reluctant to form gears-level models of ML phenomena. Stereotypically, LW/AF researchers tend to make the former mistake, while academic and industry researchers tend to make the latter kind. In this post, I discuss what I mean by “touching reality” and why it’s important, speculate a bit on why people don’t do this, and then give concrete suggestions.
    Epistemic status: I'm pretty frustrated with how slow I write, so this is an experiment in writing fast as opposed to carefully. That being said, this is ~the prevailing wisdom amongst many ML practitioners and academics, and similar ideas have been previously discussed in the LessWrong/Alignment Forum communities, so I'm pretty confident that it's directionally correct. I also believe (less confidently) that this is good advice for most kinds of research or maybe even for life in general.
    Acknowledgments: Thanks to Adrià Garriga-Alonso for feedback on a draft of this post and Justis Mills for copyediting help.
    Introduction:
    Broadly speaking, I think new researchers in machine learning tend to make two kinds of mistakes:
    Not making contact with reality. This is the failure mode where a new researcher reads a few papers that their friends are excited about, forms an ambitious hypothesis about how to solve a big problem in machine learning, and then spends months drafting a detailed plan. Unfortunately, after months of effort, our new researcher realizes that the components they were planning to use do not work nearly as well as expected, and as a result they’ve wasted months of effort on a project that wasn’t going to succeed.
    Not being willing to make gears-level models. This is the failure mode where a new researcher decides to become agnostic to why anything happens, and believes empirical results and only empirical results even when said results don’t “make sense” on reflection. The issue here is that they tend to be stuck implementing an inefficient variant of grad student descent, only able to make small amounts of incremental progress via approximate blind search, and end up doing whatever is popular at the moment.
    That’s not to say that these mistakes are mutually exclusive: embarrassingly, I think I’ve managed to fail in both ways in the past.
    That being said, this post is about the first failure mode, which I think is far more common in our community than the second. (Though I might write about the second if there's enough interest!)
    Here, by “touching reality”, I mean running experiments where you check that your beliefs are right, either via writing code and running empirical ML experiments, or (less commonly) grounding your ideas either in a detailed formalism (to the level where you can write proofs for new, non-trivial theorems about said ideas). I don’t think writing code or inventing a formalism qualify by themselves (though they are helpful); touching reality requires receiving actual concrete feedback on your ideas.
    Why touch reality?
    I think there’s four main reasons why you should do this:
    Your ideas may be bad
    When you’re new to a field, it’s probably the case that you don’t fully understand all of the key results and concepts in the field. As a result, it’s very likely the case that the ideas you come up with are bad. This is especially true for fields like machine learning that have significant amounts of tacit knowledge. ...
    14 min
  • LW - Knottiness by abramdemski
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Knottiness, published by abramdemski on January 2, 2023 on LessWrong.
    (low-effort post)
    Let's say there's a bunch of factors that influence how well you're able to articulate yourself in a specific domain of your life.
    One factor is simply having the language to express yourself. Another factor is having the practiced perceptual acuity of getting your bearings in that situation -- noticing what's going on in high resolution.
    A third factor could be your ability to apply general rationality skills to this specific domain, such as noticing confusion, noticing sunk cost fallacy, etc etc etc.
    A fourth factor is the social equilibrium around communication, in this specific area of your life. For example, communication about pain might be interpreted as a request for help, which has social costs. You could be in an equilibrium where you don't communicate about a lot of types of pain, because you don't want to imply that type of weakness and need for help. Or whatever.
    The social equilibrium is just one factor here, but it's a particularly important one, because often our explicit verbal reasoning is heavily shaped by the social equilibrium, even when we are thinking silently to ourselves. Because social equilibria can be tied to specific areas of life, this means we can get tied up in knots in specific areas of our thinking, even if we possess general cognitive skills for untying those knots. The need to participate in social equilibria creates bucket errors, cognitive blind spots, and "separate magisteria".
    As a simple example, admitting that you're not enjoying playing a board game with friends might be socially conflated with a number of things, such as not liking the people you're with, wanting to go home, etc. You might make a calculated decision to play anyway, and not voice your preferences. Over time this decision might become habit, and this might blunt your ability to notice whether you are having a good time with board games.
    For example, when asking yourself a question about what board games you enjoy, you might subconsciously substitute it with other questions, like which board games you end up playing the most, or which games other people you like seem to enjoy.
    This is even worse because other people might also be stuck in knotty equilibria; so, you might be making the trade-off for nothing.
    I'm not pointing this out as part of an argument for people to notice their own preferences in social situations. Indeed, questions of what we enjoy, prefer, want, & desire might be some of the most knotted questions! A direct approach here seems like it could easily go wrong.
    (And if you choose a "direct" approach, maybe the more reliable approach would be to say things immediately available to you (ie, observations rather than inferences) -- this helps to escape the frame.)
    Sometimes we're stuck in a pattern of behavior due to stale information, which our blind spot stops us from updating. But sometimes we're stuck in that behavior due to a very real trade-off which still applies. This is the critical question determining whether a Gordian approach to knot-untying (ie, violently cut through the knot by doing something different) applies, vs a more gentle and patient approach.
    It seems to me like a ton of weird signaling equilibria could be unknotted if only there were another way to communicate the important things.
    For example, imagine you don't want to go to a close friend's party, so you lie and come up with an "important work event" you have to go to. You're presumably worried about them feeling unloved, if you just honestly say that you didn't want to go to the party. If you could have communicated that love in a different way, then perhaps your attendance at the party would not feel so important that you had to lie about it. (See Separation of Concerns for more t...
    4 min