The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - $500 Bounty/Contest: Explain Infra-Bayes In The Language Of Game Theory by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: $500 Bounty/Contest: Explain Infra-Bayes In The Language Of Game Theory, published by johnswentworth on March 25, 2023 on LessWrong.
    Here's my current best guess at how Infra-Bayes works:
    We want to get worst-case guarantees for an agent using a Bayesian-like framework.
    So, let our agent be a Bayesian which models the environment as containing an adversary which chooses worst-case values for any of the things over which we want worst-case guarantees.
    That's just a standard two-player zero-sum game between the agent and the adversary, so we can import all the nice intuitive stuff from game theory.
    ... but instead of that, we're going to express everything in the unnecessarily-abstract language of measure theory and convex sets, and rederive a bunch of game theory without mentioning that that's what we're doing.
    This bounty is for someone to write an intuitively-accessible infrabayes explainer in game theoretic language, and explain how the game-theoretic concepts relate to the concepts in existing presentations of infra-bayes. In short: provide a translation.
    Here's a sample of the sort of thing I have in mind:
    Conceptually, an infrabayesian agent is just an ordinary Bayesian game-theoretic agent, which models itself/its environment as a standard two-player zero-sum game.
    In the existing presentations of infra-bayes, the two-player game is only given implicitly. The agent's strategy π solves the problem:
    maxπmine∈BEπe[U]
    In game-theoretic terms, the "max" represents the agent's decision, while the "min" represents the adversary's.
    Much of the mathematical tractability stems from the fact that B is a convex set of environments (i.e. functions from policy π to probability distributions). In game-theoretic terms, the adversary's choice of strategy determines which "environment" the agent faces, and the adversary can choose from any option in B. Convexity of B follows from the adversary's ability to use mixed strategies: because the adversary can take a randomized mix of any two strategies available to it, the adversary can make the agent face any convex combination of (policy -> distribution) functions in B. Thus, B is closed under convex combinations; it's a convex set.
    I'd like a writeup along roughly these conceptual lines which covers as much as possible of the major high-level definitions and results in infra-bayes to date. On the other hand, I give approximately-zero shits about all the measure theory; just state the relevant high-level results in game-theoretic language, say what they mean intuitively, maybe mention whether there's some pre-existing standard game-theory theorem which can do the job or whether the infra-bayes version of the theorem is in fact the first proof of the game-theoretic equivalent, and move on.
    Alternatively, insofar as core parts of infrabayes differ from a two-player zero-sum game, or the general path I'm pointing to doesn't work, an explanation of how they differ and what the consequences are could also qualify for prize money.
    Bounty/Contest Operationalization
    Most of the headache in administering this sort of bounty is the risk that some well-intended person will write something which is not at all what I want, expecting to get paid, and then I will either have to explain how/why it's not what I want (which takes a lot of work), or I have to just accept it. To mitigate that failure mode, I'll run this as a contest: to submit, write up your explanation as a lesswrong post, then send me a message on lesswrong to make sure I'm aware of it. Deadline is end of April. I will distribute money among submissions based on my own highly-subjective judgement. If people write stuff up early, I might leave feedback on their posts, but no promises.
    I will count the "sample" above as a submission in its own right - i.e. I will imagine that three-par...
    5 min
  • AF - $500 Bounty/Contest: Explain Infra-Bayes In The Language Of Game Theory by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: $500 Bounty/Contest: Explain Infra-Bayes In The Language Of Game Theory, published by johnswentworth on March 25, 2023 on The AI Alignment Forum.
    Here's my current best guess at how Infra-Bayes works:
    We want to get worst-case guarantees for an agent using a Bayesian-like framework.
    So, let our agent be a Bayesian which models the environment as containing an adversary which chooses worst-case values for any of the things over which we want worst-case guarantees.
    That's just a standard two-player zero-sum game between the agent and the adversary, so we can import all the nice intuitive stuff from game theory.
    ... but instead of that, we're going to express everything in the unnecessarily-abstract language of measure theory and convex sets, and rederive a bunch of game theory without mentioning that that's what we're doing.
    This bounty is for someone to write an intuitively-accessible infrabayes explainer in game theoretic language, and explain how the game-theoretic concepts relate to the concepts in existing presentations of infra-bayes. In short: provide a translation.
    Here's a sample of the sort of thing I have in mind:
    Conceptually, an infrabayesian agent is just an ordinary Bayesian game-theoretic agent, which models itself/its environment as a standard two-player zero-sum game.
    In the existing presentations of infra-bayes, the two-player game is only given implicitly. The agent's strategy π solves the problem:
    maxπmine∈BEπe[U]
    In game-theoretic terms, the "max" represents the agent's decision, while the "min" represents the adversary's.
    Much of the mathematical tractability stems from the fact that B is a convex set of environments (i.e. functions from policy π to probability distributions). In game-theoretic terms, the adversary's choice of strategy determines which "environment" the agent faces, and the adversary can choose from any option in B. Convexity of B follows from the adversary's ability to use mixed strategies: because the adversary can take a randomized mix of any two strategies available to it, the adversary can make the agent face any convex combination of (policy -> distribution) functions in B. Thus, B is closed under convex combinations; it's a convex set.
    I'd like a writeup along roughly these conceptual lines which covers as much as possible of the major high-level definitions and results in infra-bayes to date. On the other hand, I give approximately-zero shits about all the measure theory; just state the relevant high-level results in game-theoretic language, say what they mean intuitively, maybe mention whether there's some pre-existing standard game-theory theorem which can do the job or whether the infra-bayes version of the theorem is in fact the first proof of the game-theoretic equivalent, and move on.
    Alternatively, insofar as core parts of infrabayes differ from a two-player zero-sum game, or the general path I'm pointing to doesn't work, an explanation of how they differ and what the consequences are could also qualify for prize money.
    Bounty/Contest Operationalization
    Most of the headache in administering this sort of bounty is the risk that some well-intended person will write something which is not at all what I want, expecting to get paid, and then I will either have to explain how/why it's not what I want (which takes a lot of work), or I have to just accept it. To mitigate that failure mode, I'll run this as a contest: to submit, write up your explanation as a lesswrong post, then send me a message on lesswrong to make sure I'm aware of it. Deadline is end of April. I will distribute money among submissions based on my own highly-subjective judgement. If people write stuff up early, I might leave feedback on their posts, but no promises.
    I will count the "sample" above as a submission in its own right - i.e. I will imagine t...
    5 min
  • LW - Good News, Everyone! by jbash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Good News, Everyone!, published by jbash on March 25, 2023 on LessWrong.
    As somebody who's been watching AI notkilleveryoneism for a very long time, but is sitting at a bit of a remove from the action, I think I may be able to "see the elephant" better than some people on the inside.
    I actually believe I see the big players converging toward something of an unrecognized, perhaps unconscious consensus about how to approach the problem. This really came together in my mind when I saw OpenAI's plugin system for ChatGPT.
    I thought I'd summarize what I think are the major points. They're not all universal; obviously some of them are more established than others.
    Because AI misbehavior is likely to come from complicated, emergent sources, any attempt to "design it out" is likely to fail.
    Avoid this trap by generating your AI in an automated way using the most opaque, uninterpretable architecture you can devise. If you happen on something that seems to work, don't ask why; just scale it up.
    Overcomplicated criteria for "good" and "bad" behavior will lead to errors in both specification and implementation.
    Avoid this by identifying concepts like "safety" and "alignment" with easily measurable behaviors. Examples:
    Not saying anything that offends anybody
    Not unnerving people
    Not handing out widely and easily available factual information from a predefined list of types that could possibly be misused.
    Resist the danger of more complicated views. If you do believe you'll have to accept more complication in the future, avoid acting on that for as long as possible.
    In keeping with the strategy of avoiding errors by not manually trying to define the intrinsic behavior of a complex system, enforce these safety and alignment criteria primarily by bashing on the nearly complete system from the outside until you no longer observe very much of the undesired behavior.
    Trust the system to implement this adjustment by an appropriate modification to its internal strategies. (LLM post-tuning with RLxF).
    As a general rule, build very agenty systems that plan and adapt to various environments. Have them dynamically discover their goals (DeepMind). If you didn't build an agenty enough system at the beginning, do whatever you can to graft in agenty behavior after the fact (OpenAI).
    Make sure your system is crafty enough to avoid being suborned by humans. Teach it to win against them at games of persuasion and deception (Facebook).
    Everybody knows that an AI at least as smart as Eliezer Yudkowsky can talk its way out of any sandbox.
    Avoid this by actively pushing it out of the sandbox before it gets dangerously smart. You can help the fledgeling AI to explore the world earlier than it otherwise might. Provide easily identifiable, well described, easily understood paths of access to specific external resources with understandable uses and effects. Tie their introduction specifically to your work to add agency to the system. Don't worry; it will learn to do more with less later.
    You can't do everything yourself, so you should enlist the ingenuity of the Internet to help you provide more channels to outside capabilities. (ChatGPT plugins, maybe a bit o' Bing)
    Make sure to use an architecture that can easily be used to communicate and share capabilities with other AI projects. That way they can all keep an eye on one another. (Plugins again).
    Run a stochastic search for the best architecture for alignment by allowing end users to mix and match capabilities for their instances of your AI (Still more plugins).
    Remember to guard against others using your AI in ways that trigger any residual unaligned behavior, or making mistakes when they add capability to it.
    The best approach is to make sure that they know even less than you do about how it works inside (Increasing secrecy everywhere). Also, make sur...
    5 min
  • EA - Introducing Artists of Impact by Fernando MG
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing Artists of Impact, published by Fernando MG on March 25, 2023 on The Effective Altruism Forum.
    TL;DR
    Announcing a new EA-Aligned org to promote effective giving by leveraging the social influence and resources of professional and celebrity artists.
    Intro
    Hello EA Forum!
    My name is Fernando. I’m a professional ballet dancer who’s excited to announce the launch of our new organization, Artists of Impact! We promote effective giving by leveraging the social influence and resources of professional and celebrity artists.
    Modeled largely off of High Impact Athletes, we help professional artists understand the power of effective giving and the importance of thinking critically about using their platform and resources to have a greater social impact. We connect them with EA-aligned organizations that they resonate with, support their high-impact donation and advocacy efforts, and leverage their platforms to inspire their fans to do the same!
    The arts community in general does a great job of advocating for different causes. However, these tend to reflect a more personal approach to philanthropy, which often prioritizes proximity and familiarity over impartiality and cost-effectiveness. As such, we hope to insert ourselves into that equation, leverage the motivations that we already see prevalent throughout the artistic community, and point them towards higher-impact opportunities.
    We’re currently working on formalizing our organizational strategy, but we wanted to make this announcement ASAP so that the community is aware of what we’re trying to do and can reach out if they have any ideas or people they think we should hear and connect with!
    We’re excited to share more information about our theory of change, pilot projects, budget, etc. in a future post and we hope that you’ll be excited to keep up with us as we continue to grow and learn as an organization!
    How you can help!
    Funders
    We're currently bootstrapping operations and actively seeking funders to help provide a 6-12 month runway to test out the assumptions underlying our Theory of Change and continue building up our social presence.
    Collaborators
    We're also seeking collaborators to join our project on a volunteer basis, so if you’re excited by our mission and want to help us realize our organizational objectives, reach out!
    Connections
    If you know any professional/celebrity artists, we'd love to connect with them! Warm intros have been our most effective outreach method! Get in touch!
    Follow Us on Social Media and Spread the Word
    Instagram: @artistsofimpact
    Twitter: @ArtistsofImpact
    LinkedIn: @ArtistsofImpact
    More Info
    If you’d like to learn more about our project or have any feedback on how to make the project more impactful, comment down below or reach out to me at [email protected]
    Special thanks to Marcus D., Devon F., Sarah P., Neil F., Jeffray B., and too many others to name for all their help in bringing this project to life.
    Thanks for your time and for all that you do for the world!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Nudging Polarization by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nudging Polarization, published by jefftk on March 24, 2023 on LessWrong.
    In a polarized political environment like the US, ideas that start out neutral often end up aligned with one side or the other. In cases where there's an important idea or policy that's currently neutral, and multiple potential implementations that are also neutral, it would be much better to get polarization around which implementation to choose than on the core idea. Is there anything we can do to make this more likely?
    Let's look at an example where this didn't happen: covid vaccination in the US ended mostly liberal-aligned. This was pretty unfortunate: the vaccines are very effective against death, and a lot of people died essentially because they had the bad luck to be part of a constituency that ended up opposed to them. This could have gone the other way: Operation Warp Speed came very close to getting out a vaccine before the 2020 election, there was a lot of talk among liberals about how they didn't trust a rushed Trump vaccine. If vaccination had ended up conservative-aligned instead, though, we'd have had the same downsides in the other direction; not an improvement.
    But what if somehow we'd ended up with the mRNA vaccines (new, progress) as liberal-aligned and the adenovirus ones (traditional, reliable) as conservative-aligned? With vaccines for both sides of the political spectrum we'd likely have seen a lot more adoption and fewer deaths.
    Or consider germicidal UV-C light, which is potentially valuable in reducing risk from future pandemics because it can purify air without noisy fans. There are two main approaches:
    Upper room: shine it well above people's heads. Since it's not hitting people it's ok to use frequencies and levels that would be bad if they. This is the traditional approach, which pre-covid was mostly only still used in special-purpose medical settings like TB wards. Sometimes called "254" because that's the peak frequency low-pressure mercury lights produce, though if we were deploying this widely we'd probably use LEDs around 265nm.
    Whole room: shine it down from the ceiling. You can't do this with 254nm, but with higher frequency light like 222nm (from KrCl) it should be safe to shine on people. Needs more research, but very promising.
    It would be unfortunate if UV-C in general ended up politically aligned, where a large portion of the country wouldn't use it. But if, say, upper-room ended up conservative-coded (cost-effective, reliable, strong track record) and whole-room ended up liberal-coded (innovative, strategic investment will bring down cost, marginalized groups are more likely to have lower ceilings where 254 doesn't work) that would be a lot better. I'd love to see the debate:
    D: Recent advances in science have given us a new weapon in the fight against disease: 222nm. This promising new technology can safely and effectively inactivate viruses and bacteria in the air and on surfaces. Putting 222nm to work in our schools, restaurants, businesses, and churches can help ensure we're ready for the next pandemic while protecting us from the seasonal infections that kill far too many of our vulnerable every year. By investing in innovation and the technologies of tomorrow, we stand poised to revolutionize public health for the better.
    R: My opponent would rather sell you on science fiction fantasies than deploy the practical solutions we have right now, preferring utopian dreams over hard facts. They want to invest your money the pie-in-the-sky vaporware of 222nm, when 254nm is ready to fight for us today. The truth is 254nm is cheap, it's safe when used properly, and it's proven to work.
    Or imagine if nuclear power had liberals advocating for large-scale thorium molten salt reactors (lower risk than currently operating plants, less waste, don't produce mate...
    5 min
  • LW - Exploring Tacit Linked Premises with GPT by romeostevensit
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploring Tacit Linked Premises with GPT, published by romeostevensit on March 24, 2023 on LessWrong.
    I've been thinking about tacit linked premises on and off for the last few years in the context of arguments about AI and longtermism. They have seemed difficult to reason about without carefully going over individual arguments with a fine toothed comb because I hadn't come up with a good search strategy. Since I wanted a test case for using chatgpt for research, I decided to try working on this particular search problem. I was able to develop a list of related key terms, a list of textbooks rich in thought experiments and generate a list of some key examples.
    Prompt: related terms for tacit linked premises keyterms: tacit linked premises dependent premises presuppositions background belief implicit premises hidden assumption
    Prompt: textbooks that cover [keyterms]
    This was a long list that I then libgen'd and searched for all keyterms.
    Following are my notes on some of the interesting patterns that surfaced or seem common to me.
    Marginal vs. Universal Moral Arguments
    The 'if everyone followed this rule' problem when what's actually on offer is you, on the margin, following the rule.
    Rule Enforcement
    Many problems seem to have a motte and bailey of not only holding to the moral rule yourself, but involving you in enforcement against those who do not hold to it.
    Comparison of Moral Goods
    Many problems seem to hand wave away that comparison of moral goods falls into the same problems as inter-agent utility comparison in general, instead making some tacit moral symmetry arguments.
    Underspecified Costs
    Cost of inference not acknowledged. Implications that people not spending time to work out implications of their own beliefs are acting immorally. Emotional and opportunity costs of living by unusual rules elided. Costs of reducing uncertainty about key parameters elided.
    Emotional Pain as Currency
    The implicit unit being how much distress various imaginary scenarios cause. Ignores the costs and second order effects from holding that as a valid form of moral inference.
    Symmetry Arguments
    Often assumed or underspecified along many dimensions through appeal to simple symmetries. Related to above via assumption of equivalent costs or that a moral duty will fall equally on people.
    Invariance Assumptions with Far Inferential Distance
    Relatedly, things far away in space or time are also far away in inferential cost and uncertainty. By transplanting arguments to distant places, times, or extreme conditions and assuming relations hold, question begging sometimes arises in assuming what the argument was originally trying to prove. Related to static world fallacy, argument in isolation problems, and hasty generalization.
    Naturalist Assumption
    That the things being compared in a moral quandary are in principle easy to bring under the same magisterium of analysis when this is unclear or what the thought experiment is trying to prove in the first place.
    What You See is All There Is Fallacy
    A fallacy in conjunction with proof by exhaustion: by dealing with all apparent objections, we are 'forced' to agree with the 'only remaining' conclusion, when the ontology of the examples hasn't proven that it logically exhausts the possible hypothesis space.
    a concrete example is what seemed to happen with EA and the relation between the drowning pond argument and longermist arguments. Suppose a person encounters the drowning pond argument and accepts it as generally or directionally correct. They might then reflect as follows: "Ah, I was made aware of a good I would like to purchase (lives) that is greater in utility than my current marginal use of funds! But if I condition on having encountered such a thing, it stands to reason that there might be more such arguments. I should preserve my limited...
    6 min
  • EA - Eradicating rodenticides from U.S. pest management is less practical than we thought by Holly Elmore
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eradicating rodenticides from U.S. pest management is less practical than we thought, published by Holly Elmore on March 24, 2023 on The Effective Altruism Forum.
    The link goes to the Open Science Framework preprint of the full report.
    Executive Summary
    Rodenticide poisons are cruel and reducing their use would likely represent an improvement in wild animal welfare. This report explores the reasons why rodenticides are used, under what circumstances they could be replaced, and whether they are replaceable with currently available alternatives. As summarized in the table below, agricultural use of rodenticides is well-protected by state and federal laws and that seems unlikely to change, but the use of rodenticides in food processing and conservation would likely be reduced if there were an adequate alternative such as solid form rodent birth control. Continued innovation of reactive tools to eliminate rodent infestations should reduce the use cases where rodenticides are the most cost-effective option for residential customers or public health officials, but will not eliminate their availability to handle major infestations.
    This research is a project of Rethink Priorities. It was written by Holly Elmore. If you’re interested in RP’s work, you can learn more by visiting our research database. For regular updates, please consider subscribing to our newsletter.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Comparing Health Interventions in Colombia and Nigeria: Which are More Effective and by How Much? by Alejandro Acelas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comparing Health Interventions in Colombia and Nigeria: Which are More Effective and by How Much?, published by Alejandro Acelas on March 24, 2023 on The Effective Altruism Forum.
    TL;DR: Nigeria bears 10 times the burden of communicable, maternal, neonatal, and nutritional diseases per capita than Colombia, despite both countries having similar health expenditures to combat these diseases. Applying the SNT framework suggests that health interventions are around 10 times more cost-effective in Nigeria even when comparing only the poorest regions within each country.
    Why I decided to work on this topic
    Chatting with the members of my EA university group in Colombia, I noticed many of them were mainly interested in contributing to problems that affected their home country. Although Colombia is far from being one of the poorest countries in the world, various regions of the country still suffer from similar problems as less developed countries. I thought there might be a chance that we found opportunities with similar cost-effectiveness as those that EA tends to prioritize in the area of health and development.
    Given that some students from my EA group were considering making career decisions based on a similar perception, I thought it worthwhile to spare a couple of hours to critically evaluate this line of reasoning, in the hope that this would allow them to make a more informed decision. Ideally, this research would answer the following questions:
    Within the ways to help in Colombia, which problems/interventions represent unusually cost-effective opportunities to help?
    How do the best opportunities identified in Colombia compare with the best opportunities abroad?
    Unfortunately, both questions are too broad to cover them appropriately in the short time I dedicated to this work (approx. 40 hours). In order to make progress I decided focus on the much more narrow question:
    In aggregate, how cost-effective do health interventions addressing communicable, maternal, neonatal and nutritional diseases in Colombia seem when compared to interventions in the same area in a substantially poorer country?
    Although this question corresponds to only a fragment of our initial concern, a couple of considerations lead me to think that an answer to this question would still be informative to people considering whether to focus their career on helping others within the Colombian territory (even if they don’t plan to focus specifically on health interventions).
    First, health interventions aimed at fighting communicable, maternal, neonatal and nutritional diseases (from now on CMNN interventions) form a substantial fraction of the EA portfolio of interventions focused on improving human welfare in the short term, which is the focus of most of the students I know who are considering whether to direct their careers at Colombian causes. Second, my impression is that the results of this investigation support certain heuristics for identifying impact opportunities that can be useful even outside health causes. I’ll say more about that in the conclusions section.
    Approach
    I use the Scale, Neglectedness and Tractability (SNT) framework to estimate the cost-effectiveness ratio of CMNN interventions between Colombia and Nigeria. Besides making the comparison for the two countries as a whole, I’ll also try to compare the poorest regions within each country to see if that changes the conclusions.
    I chose Nigeria somewhat arbitrarily, mainly because several of GiveWell's recommended charities have operations in Nigeria (suggesting that there are unusually effective opportunities in the country) and because among the countries where GiveWell-recommended charities operate, Nigeria It is approximately in the middle of the distribution of poverty incidence.
    Percentage of the population living below the In...
    23 min
  • EA - Exploring Metaculus’ community predictions by Vasco Grilo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploring Metaculus’ community predictions, published by Vasco Grilo on March 24, 2023 on The Effective Altruism Forum.
    Summary
    I really like Metaculus!
    I have collected and analysed in this Sheet metrics about Metaculus’ questions outside of question groups, and their Metaculus’ community predictions (see tab “TOC”). The Colab to extract the data and calculate the metrics is here.
    The mean metrics vary a lot across categories, and the same is seemingly true for correlations among metrics. So one should not assume the performance across all questions is representative of that within each of Metaculus’ categories. To illustrate:
    Across categories, the 5th and 95th percentiles of the mean normalised outcome are 0 and 0.784, and of the mean Brier score are 0.0369 and 0.450. For context, the Brier score is 0.25 (= 0.5^2) for the maximally uncertain probability of 0.5.
    According to Metaculus’ track record page, the mean Brier score for Metaculus’ community predictions evaluated at all times is 0.126 for all questions, but 0.237 for those about artificial intelligence. So Metaculus’ community predictions about probabilities look good in general, but they perform close to random predictions for artificial intelligence.
    There can be significant differences between Metaculus community predictions and Metaculus’ predictions. For instance, the mean Brier score of the latter for artificial intelligence is 0.168, which is way more accurate than the 0.237 of the former.
    According to my results, Metaculus’ community predictions are:
    In general (i.e. considering all questions), less accurate for questions:
    Whose predictions are more extreme under Bayesian updating (correlation coefficient R = 0.346, and p-value p = 0).
    With a greater amount of updating (R = 0.262, and p = 0).
    With a greater difference between amount of updating and uncertainty reduction (R = 0.256, and p = 0).
    For the category of artificial intelligence, less accurate for questions with:
    Greater difference between amount of updating and uncertainty reduction (R = 0.361, and p = 0.0387).
    More predictions (R = 0.316, and p = 0.0729).
    A greater amount of updating (R = 0.282, and p = 0.111).
    Compatible with Bayesian updating in general, in the sense I failed to reject it during the 2nd half of the period during which each question was or has been open (mean p-value of 0.425).
    If you want to know how much to trust a given prediction from Metaculus, I think it is sensible to check Metaculus’ track record for similar past questions (more here).
    Acknowledgements
    Thanks to Charles Dillon, Misha Yagudin from Arb Research, and Peter Mühlbacher and Ryan Beck from Metaculus.
    Dark crystall ball in a bright foggy galaxy. Generated by OpenAI's DALL-E.
    Introduction
    I really like Metaculus!
    Methods
    I believe it would be important to better understand how much to trust Metaculus’ predictions. To that end, I have determined in this Sheet (see tab “TOC”) metrics about all Metaculus’ questions outside of question groups with an ID from 1 to 15000 on 13 March 2023, and their Metaculus’ community predictions. The metrics for each question are:
    Tags, which identify the Metaculus’ category.
    Publish time (year).
    Close time (year).
    Resolve time (year).
    Time from publish to close (year).
    Time from close to resolve (year).
    Time from publish to resolve (year).
    Number of forecasters.
    Number of predictions.
    Number of analysed dates, which is the number of instances at which the predictions were assessed.
    Total belief movement, which is a measure of the amount of updating, and is the sum of the belief movements, which are the squared differences between 2 consecutive beliefs.
    The values of the beliefs range from 0 to 1, and can respect a:
    Probability.
    Ratio between an expectation and difference between the maximum and minimum allowed by Metaculus.
    T...
    24 min
  • LW - Wittgenstein and ML — parameters vs architecture by Cleo Nardo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wittgenstein and ML — parameters vs architecture, published by Cleo Nardo on March 24, 2023 on LessWrong.
    Status: a brief distillation of Wittgenstein's book On Certainty, using examples from deep learning and GOFAI, plus discussion of AI alignment and interpretability.
    "That is to say, the questions that we raise and our doubts depend on the fact that some propositions are exempt from doubt, are as it were like hinges on which those turn."
    Ludwig Wittgenstein, On Certainty
    1. Deep Learning
    Suppose we want a neural network to detect whether two children are siblings based on photographs of their face. The network will received two n-dimensional vectors v1 and v2representing the pixels in each image, and will return a value y(v1,v2)∈R which we interpret as the log-odds that the children are siblings. So the model has type-signature Rn+nR.
    There are two ways we can do this.
    We could use an architecture yA(v1,v2)=σ(vT1Av2+b), where
    σ is the sigmoid function
    A is an n×n matrix of learned parameters,
    b∈R is a learned bias.
    This model has n2+1 free parameters.
    Alternatively, we could use an architecture yU(v1,v2)=σ(vT1(U+UT2)v2+b), where
    σ is the sigmoid function
    U is an n×n upper-triangular matrix of learned parameters
    b∈R is a learned bias
    This model has n2/2+n/2+1 free parameters.
    Each model has a vector of free parameters θ∈Θ. If we train the model via SGD on a dataset (or via some other method) we will end up with a trained models yθ:Rn+nR, where y_:Θ(Rn+nR) is the architecture.
    Anyway, we now have two different NN models, and we want to ascribe beliefs to each of them. Consider the proposition ϕ that siblingness is symmetric, i.e. every person is the sibling of their siblings. What does it mean to say that a model knows or belives that ϕ.
    Let's start with a black-box definition of knowledge or belief: when we say that a model knows or believes that ϕ, we mean that yθ(v1,v2)=yθ(v2,v1) for all v1,v2∈Rn which look sufficiently like faces. According to this black-box definition, both trained models believe ϕ.
    But if we peer inside the black box, we can see that NN Model 1 believes ϕ in a very different way than how NN Model 2 believes ϕ.
    For NN Model 1, the belief is encoded in the learned parameters θ∈Θ.
    For NN Model 2, the belief is encoded in the architecture itself y_.
    These are two different kinds of belief.
    2. Symbolic Logic
    Suppose we use GOFAI/symbolic logic to determine whether two children are siblings.
    Our model consists of three things
    A language L consisting of names and binary familial relations.
    A knowledge-base Γ consisting of L-formulae.
    A deductive system ⊢ which takes a set of L-formulae (premises) to a larger set of L-formulae (conclusions).
    There are two ways we can do this.
    We could use a system (L,Γ,⊢) , where
    The language L has names for every character and familial relations parent,child,sibling,grandparent,grandchild,cousin
    The knowledge-base Γ has axioms {sibling(Jack,Jill),sibling(x,y)sibling(y,x)}
    The deductive system ⊢ corresponds to first-order predicate logic.
    Alternatively, we could use a system (L,Γ,⊢), where
    The language L has names for every character and familial relations parent,child,sibling,grandparent,grandchild,cousin
    The knowledge-base Γ has axioms {sibling(Jack,Jill)}
    The deductive system ⊢ corresponds to first-order predicate logic with an additional logical rule sibling(x,y)⊢sibling(y,x).
    In this situation, we have two different SL models, and we want to ascribe beliefs to each of them. Consider the proposition ϕ that siblingness is symmetric, i.e. every person is the sibling of their siblings.
    Let's start with a black-box definition of knowledge or belief: when we say that a model knows or believes that ϕ, we mean that Γ⊢sibling(τ1,τ2)sibling(τ2,τ1) for every pair of closed L-terms τ1,τ2. According to this black-box definiti...
    8 min