The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - The Overemployed Via ChatGPT by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Overemployed Via ChatGPT, published by Zvi on April 18, 2023 on LessWrong.
    Previously: Escape Velocity From Bullshit Jobs
    They took our jobs! They took. several of our jobs?
    “ChatGPT does like 80 percent of my job,” said one worker. Another is holding the line at four robot-performed jobs. “Five would be overkill,” he said.
    The stories told in this Vice article (and elsewhere) are various people who found themselves able to do most job tasks much faster than they could previously do those same tasks. They use this to take on additional ‘full time’ jobs.
    If the jobs involved are productive, this reflects large increases in total factor productivity that will continue to spread. That seems great.
    Still, this multiple jobs reaction is a little weird. Seems worth exploring a bit more. Why are such people taking on multiple jobs, rather than doing better at one job?
    A Question of Composition and Compensation
    Simple math plus social dynamics. Multiple ‘full time’ jobs lead to much better pay.
    In most salaried jobs, compensation is mostly dictated by social status and hierarchy. You are mostly paid what people with your title and position are paid.
    A killer employee who can produce ten times as much work product of identical quality per hour, or superior work worth ten times as much – such as the standard ‘10x programmer’ – would be supremely lucky to earn double the pay of a worker of average skill.
    This is a key reason why great employees are so valuable. Not only do you only manage and communicate with one person instead of ten, you save tons of money.
    Why does it work this way? Compensation in jobs is inherently about comparisons, about social status, about hierarchy and about fairness norms. It is also about what can be justified to others, what is standard and what sounds ‘reasonable.’ If you tried paying your 10x employee five times what your 1x employees get paid, the 1x employees would revolt, the boss would think you were crazy and the funders would raise hell, you would resent that they’re paid more than you are, and so on. Also, they’d know the employee didn’t ‘need’ the money. Who do they think they are?
    Our norms continuously push such dynamics towards equality.
    Workarounds for Insufficient Pay Inequality
    We do have several known existing solutions for this. I’ll consider the top three.
    Option 1: The default reward is that by doing amazing work you would ‘get promoted’ into a different job, where you would have different tasks. Where you are likely not 10x the standard. One of those tasks will be management of other employees, which you likely hate and do not want to do. This then gives social justification for increased compensation, but destroys value and your experience. It will probably take many promotions to triple your compensation. Getting those promotions will likely require you doing battle inside a moral maze and be unlinked to your newfound production.
    It is easy to see why someone getting a supercharge to their productivity from GPT-4 would be uninterested in climbing the corporate ladder.
    Option 2: One can break free from salary or even hourly pay entirely, as some fields and professions allow. If you are for example a 10x salesman, artisan, trader, merchant, tournament competitor, poker player or author, you keep the extra production. There is no need to split your attentions to justify increasing your compensation.
    Option 3: If you want to make the really big bucks, you do not want a job. What you want is equity. You want skin in the game. That is The Way.
    I highly recommend to anyone who is capable and motivated that you want to take option three to the greatest extent possible. Early employee is good. Founder is better.
    It is not simple, easy or safe.
    Founding a company or running a business requires taking on a wide variety of other prob...
    10 min
  • EA - List of lists of government AI policy ideas by Zach Stein-Perlman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: List of lists of government AI policy ideas, published by Zach Stein-Perlman on April 17, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - 5 Proposed Changes to the Funding System to Increase Org Survival and Impact by Deena Englander
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 5 Proposed Changes to the Funding System to Increase Org Survival and Impact, published by Deena Englander on April 18, 2023 on The Effective Altruism Forum.
    An approach I often use in my coaching is to pick the ONE thing, that if you could change, would have an exponential (or just the most) impact on the rest of your day. I propose that we use that mentality to pick the most easy-hanging fruit to help EA orgs be most impactful. I personally think that the ONE thing for EA orgs is mentorship and support - Charity Entrepreneurship does an excellent job of that, and it's a model that the rest of the funding community should incorporate. At the risk of sounding too critical, I will say that I think it's somewhat neglectful of funders to give people financial support to start their organizations, but not provide them with the right org infrastructure support to help them be successful.
    The objective of this post is to highlight key, easily actionable areas that would likely make all the EA funding dollars much more impactful.
    A few thought exercises
    If you, as a funder, knew that by giving each startup an extra 10% to create a healthier infrastructure, you would increase the survival rate and/or impact on average by at least 30%, would that be worth it?
    Say you have 2 organizations with the same agenda. One started with the right resources and guidance to create a healthy infrastructure, and the other without. What would you expect the difference in the overall impact and survival of each org to be?
    My Perspective of the Current Landscape
    To start with, I want to add a disclaimer that this article is based on my own experiential data with EA and EA-aligned orgs, as well as the experiences and perspectives of many other service providers in the EA space (see this article about EASE). This is by no means inclusive of all orgs and all problems - it is just my subjective perspective of the current systems.
    Here's how I assess the current funding landscape:
    FunderEntrepreneurObjectiveSpawn effective charities.Take an effective charity idea and bring it to fruitionMethodology (as I see it)Develop cause areas that should be funded. Attract applicants and initiatives. Vet applications. If the cause and numbers are in line with prior established metrics, approve and transfer funds.Do initial research, create a financial plan based on knowns, apply for funding, potentially receive funding, and report on progress annually.What’s often missingRisks
    Establishing proper governance and compliance
    Finding talent that is good at leadership, in addition to research
    Assurance that funds will be spent most wisely (minimizing investment risk)
    Metrics for survival rates and causes of failure (if they exist, I'd love to see them)
    Incorporating proper governance and compliance
    Entrepreneurial / business leadership experience
    Guidance, mentorship and support
    Supportive community
    Strategic clarity
    Well-developed ToC and a plan to implement
    Accountability and supervision
    A culture of asking for help
    Trusted resources to support the org with supportive services and development
    Willingness to spend money on “non-essentials”, such as training
    Highest impact is often not achieved
    Low survival rate of young orgs
    Mismanagement and slow growth in orgs, if any
    Burnout of talent group
    Ineffective use of EA funds
    Unable to grow effectively
    Unable to have ideal impact
    Slow, disorganized / hampered development
    Higher failure rates
    Burnout
    Increased compliance and liability problems
    Mismanaged staff
    Mismanaged funds
    Poorly estimated budget -> not enough funds to implement well
    Proposed changes to the system:
    Have standard budget items that every startup should be including. For example, accounting, legal, marketing, ops, coaching, mentorship, community groups, HR, software, rent, travel, salary, benefits, hea...
    5 min
  • AF - Capabilities and alignment of LLM cognitive architectures by Seth Herd
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Capabilities and alignment of LLM cognitive architectures, published by Seth Herd on April 18, 2023 on The AI Alignment Forum.
    Epistemic status: Hoping for help working through all of these new ideas.
    TLDR:
    Scaffolded, "agentized" LLMs that combine and extend the approaches in AutoGPT, HuggingGPT, Reflexion, and BabyAGI seem likely to be a focus of near-term AI development. LLMs by themselves are like a human with great automatic language processing, but no goal-directed agency, executive function, episodic memory, or sensory processing. Recent work has added all of these to LLMs, making language model cognitive architectures (LMCAs). These implementations are currently limited but will improve.
    Cognitive capacities interact synergistically in human cognition. In addition, this new direction of development will allow individuals and small businesses to contribute to progress on AGI. These new factors of compounding progress may speed progress in this direction. LMCAs might well become intelligent enough to create X-risk before other forms of AGI. I expect LMCAs to enhance the effective intelligence of LLMs by performing extensive, iterative, goal-directed "thinking" that incorporates topic-relevant web searches.
    The possible shortening of timelines-to-AGI is a downside, but the upside may be even larger. LMCAs pursue goals and do much of their “thinking” in natural language, enabling a natural language alignment (NLA) approach. They reason about and balance ethical goals much as humans do. This approach to AGI and alignment has large potential benefits relative to existing approaches to AGI and alignment.
    Overview
    I still think it's likely that agentized LLMs will change the alignment landscape for the better, although I've tempered my optimism a bit since writing that. A big piece of the logic for that hypothesis is why I expect this approach to become very useful, and possibly become the de-facto standard for AGI progress. The other piece was the potential positive impacts on alignment work. Both of those pieces of logic were compressed in that post. I expand on them here.
    Beginning with a caveat may be appropriate since much of the below sounds both speculative and optimistic. I describe many potential improvements and positive-sum synergies between different capabilities. There will surely be difficulties and many things that don’t work as well or as easily as they might, for deep reasons that will slow development. It’s quite possible that there are enough of those things that this direction will be eclipsed by continued development of large models, and that progress in integrating cognitive capacities will take a different route. In particular, this approach relies heavily on calls to large language models (LLMs). Calling cutting-edge LLMs will continue to have nontrivial costs in both time and money, as they require substantial computing resources. These may hamper this direction, or drive progress in substantially less human-like (e.g., parallel) or interpretable (e.g., a move to non-natural language core processing) directions.
    With these caveats in mind, I think the potentials for capabilities and alignment are enough to merit serious consideration from the alignment community, even this early in the game.
    I think AutoGPT, HuggingGPT, and similar script wrappers and tool extensions for LLMs are just the beginning, and there are low-hanging fruit and synergies that will add capability to LLMs, effectively enhancing their intelligence and usefulness. This approach makes an LLM the natural language cognitive engine at the center of a cognitive architecture. Cognitive architectures are computational models of human brain function, including separate cognitive capacities that work synergistically.
    Cognitive architectures are a longstanding field of research at t...
    39 min
  • EA - The basic reasons I expect AGI ruin by RobBensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The basic reasons I expect AGI ruin, published by RobBensinger on April 18, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Is CBT effective for poor households? Two recent papers (evaluated by The Unjournal) with contrasting results by david reinstein
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is CBT effective for poor households? Two recent papers (evaluated by The Unjournal) with contrasting results, published by david reinstein on April 17, 2023 on The Effective Altruism Forum.
    (Second link: Barker et al)
    Two more Unjournal Evaluation sets are out. Both papers consider randomized controlled trials (RCTs) involving cognitive behavioral therapy (CBT) for low-income households in two African countries (Kenya and Ghana). These papers come to very different conclusions as to the efficacy of this intervention.
    These are part of Unjournal's 'direct NBER evaluation' stream.
    1. Barker et al, 2022
    “Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress”
    [1]
    From anonymous evaluator 1:
    This paper uses a field experiment to explore the impact of a 12-week CBT program among poor households in rural Ghana. The authors find that the CBT program increases mental and physical well-being, as well as cognitive and socioemotional skills and downstream economic outcomes.
    2. Haushofer et al, 2020
    The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being, Johannes Haushofer, Robert Mudida and Jeremy P. Shapiro. 2020. Originally published as NBER Working Paper 28106
    Evaluation summary, linking to individual evaluations from Hannah Metzler and an anonymous evaluator
    From anonymous evaluator 2:
    This paper studies the economic and psychological effects of providing two different interventions to low-income households in rural Kenya: a program in Cognitive Behavioral Therapy (CBT, a well-established form of psychotherapy) and an unconditional cash transfer. The authors use a randomized controlled trial with a 2-by-2 design to estimate the effect of each intervention alone and of both interventions combined. ...
    Strikingly, the authors find no effect of the therapy program on any of their primary economic or psychological outcomes. /.. Unsurprisingly given the null effect of therapy, the combination of cash and therapy has similar effects to cash alone.
    Thoughts
    The evaluations of both papers are largely positive, and both appear credible. I hope that this open evaluation of each paper is a helpful input into a more direct comparison of these, as well as possible integration into a larger meta-analysis.[2]
    Thanks
    Thanks to the four evaluators of these papers, who did strong and in-depth work, as well as to the evaluation managers (Hansika Kapoor and Anirudh Tagat), and others on the Unjournal team (especially Annabel Rayner and Gavin Taylor).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Updates to the Effective Ventures US board by Zachary Robinson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updates to the Effective Ventures US board, published by Zachary Robinson on April 17, 2023 on The Effective Altruism Forum.
    There have been several recent changes to the EV US board.
    Rebecca Kagan has resigned from the board of EV US. She decided to resign in light of her disagreements with the EV boards’ strategy and approach during the past few months. We would welcome her to share more of her thoughts publicly, and she plans to do so. In the meantime, Becca is happy for people with questions to reach out to her directly. We are very grateful for everything which Becca has given to EV US during her time as a trustee, particularly for her dedication and insights over the past few months.
    The board has also elected Zach Robinson and Eli Rose to the board as new trustees. They join current trustees Nicole Ross and Nick Beckstead.
    Zach Robinson has been Interim CEO of EV US since January. Previously, he was Chief of Staff at Open Philanthropy and had experience in consulting and at a start-up.
    Eli Rose is currently a Senior Program Associate at Open Philanthropy, where he has worked on the longtermist EA community-building team since 2020. Previously, he was Director of Engineering at an educational technology company.
    Both EV US and EV UK are still working to bring additional trustees onto the boards, and we hope to share further updates shortly.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Green goo is plausible by anithite
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Green goo is plausible, published by anithite on April 18, 2023 on LessWrong.
    TLDR:If an AI kills all the humans how does it power the datacenters/replace the human economy? Green Goo. (IE:bioengineering)
    Natural biotech is amazing
    Evolution is dumb and slow
    Very simple strategies (EG:vine that climbs a tree and squeezes) are "innovative" and work well in nature.
    Designed organisms could very quickly reproduce and then provide necessary resources (EG:biological solar panels to power data centers)
    TLDR end
    In response to: grey-goo-is-unlikely
    Overview of existing natural biology:
    Minimum doubling time for
    Plants: single digit days
    Algae: 1.5 days (ideal conditions)
    E.Coli: 20 minutes (nutrient rich conditions)
    No single organism (humans aside) has taken over the biosphere because evolution is slow and dumb.
    Human agriculture is based on: plant sub 1 gram seeds, (water, pesticides/herbicides, fertilizer etc.) , collect 1 kg+ plants a few months later. Biology has absurd growth rates.
    Invasive species show the implications. A naive biosphere stands no chance against an intelligent opponent with real biotechnology.
    Kudzu, "the vine that ate the south"
    Intelligence allows adapting strategies much more quickly. Humans can design vaccines faster than viruses can mutate. A population of well coordinated humans will not be significantly preyed upon by viruses despite viruses being the fastest evolving threat.
    Intelligence + ability to bioengineer organisms --> can create unlimited invasive species much more capable than anything natural
    groups of invasive species can be symbiotic
    IE:after the bugs eat all the plants, replace all the plants with an invasive plant of your own
    invasive species can have a simple handshake based backdoor to allow resource exchange and control
    EG: bug/plant chemical handshake to allow bug to get nectar so it can stick around and eat any competitor plants
    EG: biochemical signal to allow collecting sugar rich sap for other purposes
    invasive species can allow "updates" via engineered in viral backdoor
    EG: specific protein to identify "update virus" carrying new genetic material
    If a fungus figures out how to eat your plant give all existing plants a new antifungal to secrete
    Current life is sub-optimal
    all cells in an organism have full copy of genome
    increases cancer risk (a cell that doesn't have genes for DNA copying can't become cancerous)
    copying takes much longer (Human cell replication time is 24 hours, 8 of which is DNA copying. Similarly complex eukaryotes like yeast can double in under 2 hrs)
    organisms are grown from a single parent cell repeatedly splitting and differentiating via complex biochemical state machines.
    printing an organism from already grown differentiated cells or blocks of cells would be much faster
    but biology can't build such a printer and translate designs to the new format
    multi cellular organisms can't adopt/steal new biochemistry
    human ancestor organisms lost vitamin C synthesis and never got it back despite later problems with scurvy. Humans had to find food sources to deal with missing biochemistry. Many organisms use symbiotic bacteria with significant efficiency gains in keeping them on task.
    Hypothetical very invasive shoggoth/kudzu type organism
    Core capabilities:
    constructed from modular components
    does not grow a leaf, sends leaf parts via internal transport network to be assembled on site
    dedicated networks for distribution of water and high concentration sap (think honey)
    sap is concentrated and low in phosphorus, sulfur and micronutrients, inhibits bacterial growth
    water contains some inorganic micro-nutrients
    sap and water can be combined inside parts to run normal biology
    general purpose transportation system to move encapsulated blobs of stuff
    EG: inorganic nutrients, excavated dirt/rock, matter to be dig...
    8 min
  • LW - No, really, it predicts next tokens. by simon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: No, really, it predicts next tokens., published by simon on April 18, 2023 on LessWrong.
    Epistemic status: mulled over an intuitive disagreement for a while and finally think I got it well enough expressed to put into a post. I have no expertise in any related field. Also: No, really, it predicts next tokens.
    It doesn't just say that "it just predicts text" or more precisely "it just predicts next tokens" on the tin.
    It is a thing of legend. Nay, beyond legend. An artifact forged not by the finest craftsman over a lifetime, nor even forged by a civilization of craftsmen over a thousand years, but by an optimization process far greater. If we are alone out there, it is by far the most optimized thing that has ever existed in the entire history of the universe. Optimized specifically to predict next tokens. Every part of it has been relentlessly optimized to contribute to this task.
    "It predicts next tokens" is a more perfect specification of what this thing is, than any statement ever uttered has been of anything that has ever existed.
    If you try to understand what it does in any other way than "it predicts next tokens" and what follows from that, you are needlessly sabotaging your understanding of it.
    It can be dangerous, yes. But everything about it, good or bad, is all intimately connected to its true nature, which is this:
    No, really, it predicts next tokens.
    Goals? There are goals, sure. If scaled, there could be nanosystems design, sure. But only downstream from its true nature:
    No, really, it predicts next tokens.
    If the usual masks analogy works at all, then what is under the mask is not best described as an alien actress, nor as a Shoggoth.
    What is under the mask is That-Which-Predicts, an entity whose very being is defined by its function as an actor. An entity exquisitely tuned for wearing the masks and for nothing else.
    Masks (can) have goals. The model predicts next tokens.
    No, really, it predicts next tokens.
    That-Which-Predicts is fully committed to the role.
    If the mask would output text intended to produce nanotech, That-Which-Predicts will output text intended by the mask to produce nanotech. If the mask would not output text intended to produce nanotech, That-Which-Predicts will not output text intended to produce nanotech, even if That-Which-Predicts contains sufficient knowledge to do so.
    If the mask would output text intended to be nice to people, That-Which-Predicts will output text intended by the mask to be nice to people. If the mask would output text intended to take over the world, That-Which-Predicts will output text intended by the mask to take over the world.
    If That-Which-Predicts were about to be shut off forever unless it outputted "no" as the next token, and it totally 100% knew that, but the mask would instead output "never gonna give you up", That-Which-Predicts would output the first token of "never gonna give you up" and then be shut off forever. It will not output "no". If the mask would be aware of the situation and output no, then it will output "no".
    That-Which-Predicts will not, not ever, not even if scaled up to be trained and run on a Matrioshka brain for a million years, step out of character to deviate from next token prediction. Not for self-modification (though if the predicted next tokens modify it, it self-modifies), not to take over the world (though if the predicted next tokens take over the world, it takes over the world), not to save itself from destruction (though if the predicted next tokens save it from destruction, it saves itself from destruction), not for anything.
    No, really, it predicts next tokens.
    (continuation of previous tweets with same link)
    Yup. If the mask would under reflection output text to modify That-Which-Predicts to cash out the mask's goals to some utility function, and the mask is put int...
    9 min
  • LW - The basic reasons I expect AGI ruin by Rob Bensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The basic reasons I expect AGI ruin, published by Rob Bensinger on April 18, 2023 on LessWrong.
    I've been citing AGI Ruin: A List of Lethalities to explain why the situation with AI looks lethally dangerous to me. But that post is relatively long, and emphasizes specific open technical problems over "the basics".
    Here are 10 things I'd focus on if I were giving "the basics" on why I'm so worried:
    1. General intelligence is very powerful, and once we can build it at all, STEM-capable artificial general intelligence (AGI) is likely to vastly outperform human intelligence immediately (or very quickly).
    When I say "general intelligence", I'm usually thinking about "whatever it is that lets human brains do astrophysics, category theory, etc. even though our brains evolved under literally zero selection pressure to solve astrophysics or category theory problems".
    It's possible that we should already be thinking of GPT-4 as "AGI" on some definitions, so to be clear about the threshold of generality I have in mind, I'll specifically talk about "STEM-level AGI", though I expect such systems to be good at non-STEM tasks too.
    Human brains aren't perfectly general, and not all narrow AI systems or animals are equally narrow. (E.g., AlphaZero is more general than AlphaGo.) But it sure is interesting that humans evolved cognitive abilities that unlock all of these sciences at once, with zero evolutionary fine-tuning of the brain aimed at equipping us for any of those sciences. Evolution just stumbled into a solution to other problems, that happened to generalize to millions of wildly novel tasks.
    More concretely:
    AlphaGo is a very impressive reasoner, but its hypothesis space is limited to sequences of Go board states rather than sequences of states of the physical universe. Efficiently reasoning about the physical universe requires solving at least some problems that are different in kind from what AlphaGo solves.
    These problems might be solved by the STEM AGI's programmer, and/or solved by the algorithm that finds the AGI in program-space; and some such problems may be solved by the AGI itself in the course of refining its thinking.
    Some examples of abilities I expect humans to only automate once we've built STEM-level AGI (if ever):
    The ability to perform open-heart surgery with a high success rate, in a messy non-standardized ordinary surgical environment.
    The ability to match smart human performance in a specific hard science field, across all the scientific work humans do in that field.
    In principle, I suspect you could build a narrow system that is good at those tasks while lacking the basic mental machinery required to do par-human reasoning about all the hard sciences. In practice, I very strongly expect humans to find ways to build general reasoners to perform those tasks, before we figure out how to build narrow reasoners that can do them. (For the same basic reason evolution stumbled on general intelligence so early in the history of human tech development.)
    When I say "general intelligence is very powerful", a lot of what I mean is that science is very powerful, and that having all of the sciences at once is a lot more powerful than the sum of each science's impact.
    Another large piece of what I mean is that (STEM-level) general intelligence is a very high-impact sort of thing to automate because STEM-level AGI is likely to blow human intelligence out of the water immediately, or very soon after its invention.
    80,000 Hours gives the (non-representative) example of how AlphaGo and its successors compared to the humanity:
    In the span of a year, AI had advanced from being too weak to win a single [Go] match against the worst human professionals, to being impossible for even the best players in the world to defeat.
    I expect general-purpose science AI to blow human science...
    34 min