The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Open Philanthropy is hiring for (lots of) operations roles! by maura
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open Philanthropy is hiring for (lots of) operations roles!, published by maura on December 14, 2022 on The Effective Altruism Forum.
    It’s been a busy year at Open Philanthropy; our ambitious growth goals have had us expanding our team rapidly this year (we roughly doubled over the course of 2022). We recently posted 7 operations roles, which are now live on our jobs page and linked below. Each role will be vital to scaling our operations and executing on our grantmaking goals:
    A Business Operations Lead, who will manage multiple Business Operations Generalists on our team which provides organization-wide tech and office infrastructure, helps with event planning, and handles all forms of time-saving support.
    Note: the generalist role is open to applicants without any previous experience, and we’re happy to consider candidates graduating in May.
    A Finance Operations Assistant to help manage accounts payable, contractors, and internal expense reporting.
    Multiple Grants Associates (both generalist and longtermist-focused) to join the team responsible for structuring, conducting due diligence on, and processing every grant we make.
    A People Operations Generalist to work on onboarding, retention, offboarding, payroll, benefits, and helping our team grow more international.
    A Recruiter to help us continue to grow.
    A Salesforce Administrator & Technical Project Manager to help manage Salesforce (our grantmaking platform) and support other tech needs.
    If you know someone who would be great for one of these roles, please refer them to us. We welcome external referrals, and have found them extremely helpful in the past. We also offer a $5,000 referral bonus; more information here.
    A few notes for prospective applicants
    We’re always adjusting our hiring process based on candidate feedback, and we wanted to highlight (or reiterate) a few things we’re trying for this round of roles.
    For most of the roles posted (all except the Business Operations Lead and Salesforce roles), you only need to apply once to opt into consideration for as many of these roles as you’re interested in.
    There will be a checkbox on the application form asking if you want to be considered for any other teams. Checking multiple boxes will generally not mean you will be asked to do additional work tests for each role you’re applying to. If you proceed to the end of the process, you may have interviews with multiple hiring managers if multiple teams are still interested in considering you as a candidate, and you may receive an offer for one or multiple of the roles you expressed interest in.
    Interested candidates can self-select into applying for a 3-month temporary two-way trial for these roles. If you select this track, this means you’ll:
    Potentially have the opportunity to go through a more streamlined application process (one brief work test, rather than two, with the option to consider alternatives to work tests), if you need a decision more quickly.
    If you seem like a very strong candidate for full-time employment at the end of that streamlined process, receive an offer of 3 months of employment, at the end of which we will mutually decide whether to continue a longer-term employment relationship (assuming no compliance issues arise, we plan to extend this option to both US and international candidates).
    Please don’t feel any pressure to select this option — we’re providing it as an experiment, and in case it’s actively preferable for some candidates (e.g. because they’re uncertain they want to transition into a career in ops and would like to try it out, don’t have the financial flexibility to go through a more involved application process, or don’t feel they can make a good longer-term career decision right now for whatever reason). If we get feedback from applicants that this isn’t useful, or is stres...
    7 min
  • LW - Trying to disambiguate different questions about whether RLHF is “good” by Buck
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trying to disambiguate different questions about whether RLHF is “good”, published by Buck on December 14, 2022 on LessWrong.
    (A few of the words in this post were written by Ryan Greenblatt and Ajeya Cotra. Thanks to Oliver Habryka and Max Nadeau for particularly helpful comments.)
    Sometimes people want to talk about whether RLHF is “a promising alignment strategy”, or whether it “won’t work” or “is just capabilities research”. I think that conversations on these topics are pretty muddled and equivocate between a bunch of different questions. In this doc, I’ll attempt to distinguish some of these questions, and as a bonus, I’ll give my opinions on them.
    I wrote this post kind of quickly, and I didn’t have time to justify all the claims I make; I hope that this post is net helpful anyway. I’m sympathetic to claims that alignment researchers should err more on the side of writing fewer but better posts; maybe I’ll regret making this one now instead of waiting.
    Is “make a powerful AGI by using RLHF, where the feedback is coming from unaided humans”, a promising strategy for building aligned AGI?
    IMO, this seems like the baseline, “we didn’t really try much at all” alignment scheme. Ajeya calls this a “naive safety effort” in her training game post, which lays out the basic case for pessimism about this strategy.
    Here’s how I’d quickly summarize my problems with this scheme:
    Oversight problems:
    Overseer doesn’t know: In cases where your unaided humans don’t know whether the AI action is good or bad, they won’t be able to produce feedback which selects for AIs that do good things. This is unfortunate, because we wanted to be able to make AIs that do complicated things that have good outcomes.
    Overseer is wrong: In cases where your unaided humans are actively wrong about whether the AI action is good or bad, their feedback will actively select for the AI to deceive the humans.
    Catastrophe problems:
    Even if the overseer’s feedback was perfect, a model whose strategy is to lie in wait until it has an opportunity to grab power will probably be able to successfully grab power.
    I don’t think we’re 100% doomed if we follow this plan, but it does seem pretty likely to go badly.
    RLHF with unaided humans is not literally the most doomed alignment scheme I’ve ever heard seriously proposed. For example, “train models with automated rewards (e.g. simulating evolution, or training models on a curriculum of math problems) and hope that the resulting models are aligned” might be a worse plan. (Though it’s pretty plausible to me that this kind of scheme would have such obvious alignment issues that people would quickly switch to the naive safety plan.)
    I don’t think that many alignment researchers are seriously proposing this naive plan. Many researchers who work on RLHF are sympathetic to the concerns that I listed here. For example, OpenAI’s alignment plan emphasizes the importance of using models to assist human evaluation.
    Is RLHF broadly construed (i.e. pretraining a model and then fine-tuning it based on some overseer’s evaluations of its actions) plausibly part of a not-completely-doomed alignment plan?
    IMO, yes, we are very likely to want to make powerful models by fine-tuning models based on overseer feedback, because fine-tuning models using preferences over their outputs is a broadly applicable strategy that can be used in lots of ways.
    I think that our best current alignment strategies (I basically agree with Holden here on the current best plan) involve fine-tuning a model based on feedback from human overseers who have AI and software assistance, on inputs including some that were chosen by a red team who is trying to find inputs where the model behaves very badly.
    If prosaic alignment researchers are able to come up with alignment schemes which look good on paper (i.e., do...
    11 min
  • EA - Neuron Count-Based Measures May Currently Underweight Suffering in Farmed Fish by MHR
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Neuron Count-Based Measures May Currently Underweight Suffering in Farmed Fish, published by MHR on December 13, 2022 on The Effective Altruism Forum.
    Epistemic status: layperson’s attempt to understand the relevant literature. I welcome corrections from anyone with a better understanding of fish biology.
    Summary
    Neuron counts have historically been used as a proxy for the moral weight of different animal species. While alternate systems have been proposed in response to criticisms of using neuron counts alone, neuron counts likely remain a useful input into these more holistic weighting processes.
    The only publicly-available empirical reports of fish neuron counts sample exclusively from very small species (<1 g bodyweight), but many farmed fish are of species at least 1000x more massive.
    Some sources apply neuron count estimates from small fish to larger farmed fish without correction, which seems very unlikely to be reasonable given the large size differences involved.
    Even among sources that try to account for these size differences, there is significant uncertainty in how to extrapolate neuron count data from small fish to larger fish.
    As a result, animal welfare advocates should be highly skeptical of current neuron count-based estimates of the moral weight of farmed fish, and should consider funding studies to empirically measure neuron counts in these species.
    The role of neuron counts in estimating moral significance
    As long as there are limited resources available for improving animal welfare, advocates will require a means of morally weighting the harm that occurs to different species. Historically, neuron counts have often been used as a proxy for moral weight in interspecies comparisons, sometimes as the sole factor used. Researchers have raised issues with the direct use of neuron counts in determining moral weights, with a prominent recent example being a segment of the Rethink Priorities Moral Weight Project, which argues against the validity of neuron counts alone as a proxy for moral importance. However, even the Rethink Priorities team does not argue that neuron counts are useless, but rather, that they should be considered as just one component among many in establishing moral weight:
    Overall, we suggest that neuron counts should not be used as a sole proxy for moral weight, but cannot be dismissed entirely. Rather, neuron counts should be combined with other metrics in an overall weighted score that includes information about whether different species have welfare-relevant capacities.
    Given this context, neuron counts will likely continue to be a meaningful factor in most plausible models for estimating interspecies relative moral importance; it is therefore important both morally and epistemically that neuron count estimates are accurate. This is especially true with respect to those species for which we can plausibly impact the welfare of a very large number of individual members. Farmed fish, of which more than 50 billion are slaughtered each year (probably more than the number of farmed chickens), are easily among those species for which it is most important to obtain a high-quality neuron count estimate.
    Current neuron count data for fish
    Invincible Wellbeing's Planetary Animal Welfare Survey (PAWS) spreadsheet is, to the best of my knowledge, the most comprehensive literature review of neuron count estimates across animal species. Despite this, only two fish species, both weighing less than one gram, appear in PAWS.
    Table 1. Fish neuron counts (PAWS)
    Guppy (Poecilia reticulata)0.41 g4.3 millionLinkZebrafish (Danio rerio)0.75 g10 millionLink
    Species
    Average Adult Body Weight
    Neuron Count
    Source
    A further independent literature review revealed only a single additional estimate of a fish species’s neuron count: in What We Owe the Future, Will ...
    12 min
  • EA - Announcing ERA: a spin-off from CERI by Nandini Shiralkar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing ERA: a spin-off from CERI, published by Nandini Shiralkar on December 13, 2022 on The Effective Altruism Forum.
    The Existential Risk Alliance (ERA) is a non-profit project working to mitigate existential risk by equipping the community of young researchers and entrepreneurs with the skills and the knowledge necessary to tackle existential risks via our summer research fellowship programme.
    Highlights
    The CERI Fellowship is now a spin-off from the Cambridge Existential Risks Initiative (CERI), and will be run by a new nonprofit project called ERA in 2023 and beyond.
    Join the ERA team! ERA is looking for focus area leads in our main cause areas (e.g., AI Safety, Biosecurity, etc.) who could support the core team and help us grow. We are accepting expressions of interest for certain positions that we haven't yet opened hiring rounds for.
    ERA Fellowship Mentors’ expression of interest form is now live. ERA mentors will advise our summer research fellows, an international cohort of top students and early career professionals who are dedicated to using their summers (and in many cases, their careers) to help address large-scale threats to humanity's future.
    ERA Cambridge Fellows’ expression of interest form is also live - we expect applications for 2023 summer to open in late January/early February.
    CERI’s evolution to ERA
    The Cambridge Existential Risks Initiative (CERI) was founded in April 2021 as a project at the University of Cambridge with the aim to reduce existential risk (x-risk) by raising awareness and promoting x-risk research, via initiatives such as our flagship Summer Research Fellowship programme.
    Since then, CERI has evolved into a meta organisation supporting many projects within this space; some projects have been run entirely by students (such as the Existential Risks Introductory Course), and some projects have been run by a mixture of students and full time community builders (such as the CERI Fellowship).
    Now, CERI will refocus to be a University of Cambridge student group, engaging with the local community via projects such as HackX. ERA will be running the ERA Fellowship (which was previously called the CERI Fellowship) in 2023 and beyond.
    Time for a new ERA!
    Our mission is to reduce the probability of an existential catastrophe. We believe that one of the key ways to reduce existential risk lies in fostering a community of dedicated and knowledgeable x-risk researchers. Through our summer research fellowship programme, we aim to identify and support aspiring researchers in this field, providing them with the resources and mentorship needed to succeed.
    The establishment of ERA will allow us to optimise our programming and support ecosystem for the Summer Research Fellowship programme. We plan to take an evidence-based approach to community building to improve our existing programme and to find promising new approaches in this space.
    Why rename CERI to ERA?
    The naming situation with the -ERIs is just very confusing. There is CHERI, CERI, SERI, BERI, and LERI (the ones I know of). We think that we could potentially reach a different subset of our target demographic by rebranding, and also reduce the amount of time we spend clarifying questions over exactly which -ERI we are. The acronym ERA also has very positive connotations, and is fairly short/catchy.
    Join the ERA team
    ERA is looking for passionate and driven focus area leads who could support the core ERA team and help us grow. We are accepting expressions of interest for certain positions that we haven't yet opened hiring rounds for.
    We have made the first step as easy as possible, and we recommend submitting an expression of interest even if you are not sure whether you are a good fit. If we think you could be a particularly good fit for our team, we will reach out to you.
    ERA Cambri...
    5 min
  • LW - Existential AI Safety is NOT separate from near-term applications by scasper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Existential AI Safety is NOT separate from near-term applications, published by scasper on December 13, 2022 on LessWrong.
    Stephen Casper, [email protected]. Thanks to Alex Lintz and Daniel Dewey for feedback.
    This is a reply but not an objection to a recent post from Paul Christiano titled AI alignment is distinct from its near term applications. The post is fairly brief and the key point is decently summed up by this excerpt.
    I worry that companies using alignment to help train extremely conservative and inoffensive systems could lead to backlash against the idea of AI alignment itself. If such systems are held up as key successes of alignment, then people who are frustrated with them may end up associating the whole problem of alignment with “making AI systems inoffensive.”
    I have no disagreements with this claim. But I would push back against the general notion that AI [existential] safety work is disjoint from near term applications. Paul seems to agree with this.
    We can develop and apply alignment techniques to these existing systems. This can help motivate and ground empirical research on alignment, which may end up helping avoid higher-stakes failures like an AI takeover.
    This post argues for strongly emphasizing this point.
    What do I mean by near-term applications? Any challenging problem involving consequential AI and society. Examples include:
    Self-driving cars
    Recommender systems
    Search engines
    AI weapons
    Cybersecurity
    Unemployment
    Bias/fairness/justice involving systems that work with humans or human data
    Misuses of media generators including text and image generators
    I argue that working on these problems probably matters a lot for three reasons. The second and third of which are potential matters of existential safety.
    Non X-risks from AI are still intrinsically important AI safety issues.
    There are many important non X-risks in this world, and any altruistic-minded person should care about them. For the same reason we should care about health, wealth, development, and animal welfare, we should also care about making important near-term applications of narrow AI go well for people.
    There are valuable lessons to learn from near-term applications.
    Imagine that we figured out ways to make near-term applications of AI go very well. I find it incredibly hard to imagine a world in which we did any of these things without developing a lot of useful technical tools and governance strategies that could be retooled or built on for higher-stakes problems later. Consider some examples.
    For self-driving cars to go well, we would need to effectively develop and iterate on techniques for robustness, reliability, and anomaly detection/handling.
    To make recommender systems go well, we would need to make a lot more progress on efficiently inferring what humans truly want in a way that is disentangled from what they seem to want.
    To minimize harms from AI weapons, we would need to introduce a lot of national and international laws and precedent for disincentivizing and responding to deadly AI.
    To minimize problems from discriminatory AI or the misuses of media generators (e.g. BLOOM or Stable Diffusion), we would need laws and precedent establishing definitions of harm and methods for recourse. Perhaps most importantly, establishing lots of laws and bureaucracy around AI systems like this may help to establish a legal regime that provides filters and obstacles to the deployment of risky systems. If auditing and slow timelines are good, so is this kind of bureaucracy.
    See also this post.
    Making allies and growing the AI safety field is useful
    AI safety and longtermism (AIS&L) have a lot of critics, and in the past year or so, they seem to have grown in number and profile. Many of whom are people who work on and care a lot about near-term applications of AI. To some extent t...
    6 min
  • EA - Improving EA events: start early & invest in content and stewardship by Vaidehi Agarwalla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Improving EA events: start early & invest in content and stewardship, published by Vaidehi Agarwalla on December 12, 2022 on The Effective Altruism Forum.
    This post is part of an ongoing series: Events in EA: Learnings and Critiques.
    TL:DR;
    The way many EA events are run is unsustainable for event organisers and does not leave sufficient slack for running excellent events and experimenting for learning.
    We share our models for better event planning and highlight some key challenges with current approaches to event operations (event ops). We make the case that hiring early, finding a venue and outsourcing logistics will create more capacity to invest in the most important (but relatively neglected) aspects of event ops - content, structure and stewardship. More capacity can also allow organisers to deviate from defaults and innovate with new event formats and styles, and engage in more resource-intensive programming.
    We aim to set realistic expectations and provide practical guidance for newer organisers to better prepare them. Good event ops people are not easily replaceable. We’ve observed a trend of event organisers being put under a lot of (unnecessary) pressure due to a lack of planning and capacity. We give a bunch of real (anonymised) examples throughout this post to illustrate our point.
    We don't go into specific suggestions because it's very event-dependent, but we'd estimate that adding ~20-40% of lead time and/or capacity would help achieve these goals.
    We’d love to hear if you're an event organiser and don’t find this helpful or actionable. Although we're only talking about events, we think it's possible there are similar trends in other areas -we’d love to hear about them if you've got observations!
    The Hierarchy of Events Planning
    In Maslow’s Hierarchy of Needs, needs lower down in the hierarchy must be satisfied before individuals can attend to needs higher up. Our pyramid is similar - components lower down in the hierarchy are more basic and are necessary for an event to exist. The exact configuration of each component affects the ones above it (e.g. if your venue only has 2 rooms, you probably can’t have an agenda with 3 tracks of events running simultaneously).
    However, influence is not strictly linear - so the structure of the event might influence which venue you choose (e.g. if you want to create an informal atmosphere, you might choose to host an event in a smaller, more casual venue instead of a formal one, or if you need a lot of stewardship for newer attendees, you might hire a larger team to manage those needs).
    Consider Leverage
    If you could invest 1-10% more resources (time) for 10-30% return, this seems worth considering. In the following, we’re going to try and point out the highest leverage points of each step of the pyramid - the places where spending that extra time could make a big difference and open up more possibilities.
    Work backwards from specific goals
    aka don’t start with “I want to run a conference”
    It may sound trite, but no event makes sense without specific, concrete, and measurable goals - you need to know the who and why, and then work backwards from there. Start with a specific goal like “I want to facilitate peer- to peer bonding for my local university group” and then list all the ways that could achieved, (host a social dinner, a retreat, a 1-day hike, a group project etc.), rather than do an event format because it's been done before and seems effective.
    Here are some goals an event might have - note that these goals are pretty different from each other, and it’s pretty hard to create events that could excel more than a couple at once:
    Networking - participants form important connections with others - either peers or mentors
    Learning/Skilling up - participants learn from each other or experts on a particular topic
    Id...
    24 min
  • LW - Applications open for AGI Safety Fundamentals: Alignment Course by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applications open for AGI Safety Fundamentals: Alignment Course, published by Richard Ngo on December 13, 2022 on LessWrong.
    The AGI Safety Fundamentals (AGISF): Alignment Course is designed to introduce the key ideas in AGI safety and alignment, and provide a space and support for participants to engage, evaluate and debate these arguments. Participants will meet others who are excited to help mitigate risks from future AI systems, and explore opportunities for their next steps in the field.
    The course is being run by the same team as for previous rounds, now under a new project called BlueDot Impact.
    Time commitment
    The course will run from February-April 2023. It comprises 8 weeks of reading and virtual small-group discussions, followed by a 4-week capstone project.
    The time commitment is around 4 hours per week, so participants can engage with the course alongside full-time work or study.
    Course structure
    Participants are provided with structured content to work through, alongside weekly, facilitated discussion groups. Participants will be grouped depending on their ML experience and background knowledge about AI safety. In these sessions, participants will engage in activities and discussions with other participants, guided by the facilitator. The facilitator will be knowledgeable about AI safety, and can help to answer participants’ questions.
    The course is followed by a capstone project, which is an opportunity for participants to synthesise their views on the field and start thinking through how to put these ideas into practice, or start getting relevant skills and experience that will help them with the next step in their career.
    The course content is designed by Richard Ngo (Governance team at OpenAI, previously a research engineer on the AGI safety team at DeepMind). You can read the curriculum content here.
    Target audience
    We are most excited about applicants who would be in strong position to pursue technical alignment research in their career, such as professional software engineers and students studying technical subjects (e.g. CS/maths/physics/engineering).
    That said, we consider all applicants and expect 25-50% of the course to consist of people with a variety of other backgrounds, so we encourage you to apply regardless. This includes community builders who would benefit from a deeper understanding of the concepts in AI alignment.
    We will be running another course on AI Governance in early 2023 and expect a different distribution of target participants.
    Apply now!
    If you would like to be considered for the next round of the courses, starting in February 2023, please apply here by Thursday 5th January 2023. More details can be found here. We will be evaluating applications on a rolling basis and we aim to let you know the outcome of your application by mid-January 2023.
    If you already have experience working on AI alignment and would be keen to join our community of facilitators, please apply to facilitate.
    Who is running the course?
    AGISF is now being run by BlueDot Impact - a new non-profit project running courses that support participants to develop the knowledge, community and network needed to pursue high-impact careers. BlueDot Impact spun out of Cambridge Effective Altruism, and was founded by the team who was primarily responsible for running previous rounds of AGISF. You can read more in our announcement post here.
    We’re really excited about the amount of interest in the courses and think they have great potential to build awesome communities around key issues. As such we have spent the last few months:
    Working with pedagogy experts to make discussion sessions more engaging
    Formalising our course design process with greater transparency for participants and facilitators
    Building systems to improve participant networking to create high-value ...
    5 min
  • LW - [Interim research report] Taking features out of superposition with sparse autoencoders by Lee Sharkey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Interim research report] Taking features out of superposition with sparse autoencoders, published by Lee Sharkey on December 13, 2022 on LessWrong.
    We're thankful for helpful comments from Trenton Bricken, Eric Winsor, Noa Nabeshima, and Sid Black.
    This post is part of the work done at Conjecture.
    TL;DR: Recent results from Anthropic suggest that neural networks represent features in superposition. This motivates the search for a method that can identify those features. Here, we construct a toy dataset of neural activations and see if we can recover the known ground truth features using sparse coding. We show that, contrary to some initial expectations, it turns out that an extremely simple method – training a single layer autoencoder to reconstruct neural activations with an L1 penalty on hidden activations – doesn’t just identify features that minimize the loss, but actually recovers the ground truth features that generated the data. We’re sharing these observations quickly so that others can begin to extract the features used by neural networks as early as possible. We also share some incomplete observations of what happens when we apply this method to a small language model and our reflections on further research directions.
    This post is not intended to be a polished research product. It’s a set of preliminary and incomplete results that we’d like to share early in case it benefits the broader mechanistic interpretability research community.
    Introduction
    Features in a neural network are represented in high dimensional neural activations. Recent work from Anthropic has demonstrated that neural networks probably use a phenomenon called superposition to represent more features than available dimensions. Superposition is a challenge for mechanistic interpretability because it leads to polysemanticity, which prevents us from telling simple, human-understandable stories about how individual representations in neural networks relate to each other. Superposition also introduces another, more practical problem: Many of the best known feature extraction methods are dimensionality reduction techniques that find at most N directions in an N dimensional space. But we need a method that finds more than N features in superposition (i.e. an ‘overcomplete’ set of features).
    If we could develop a method that decomposes a neural network’s activations into its component features, we would essentially be taking features out of superposition, thus resolving the problems that superposition causes for interpretability.
    Superposition is possible because not all features are active at any one time. In fact, we expect that only a few features are active at any one time; in other words, feature activations may often be sparse. The sparsity of feature activations suggests that a set of methods called sparse coding or sparse dictionary learning (Olshausen & Field, 1996; Chen et al. 2001; Donoho et al. 2003) may offer a way to identify the underlying features used by networks.
    Although there are many different sparse coding algorithms, one of the simplest and most scalable methods is a single-layer feedforward autoencoder with sparsity (L1 norm) regularization on its hidden activations. When training a sparse autoencoder to reconstruct toy data – a ‘fake’ dataset of neural signals where we generate the data from sparse linear combinations of known ground truth feature directions – the results we present here indicate (to our slight surprise) that the decoder weights learned by a sparse autoencoder actually approximate the underlying ground truth features used to generate the data. It wasn’t a priori obvious to us that it would do this, nor to a couple of other researchers whom we’ve talked to who were working on this problem. To our intuitions at least, it might plausibly have learned feature...
    35 min
  • LW - Okay, I feel it now by george
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Okay, I feel it now, published by george on December 13, 2022 on LessWrong.
    I’ve been coming to LessWrong for a while. I’ve read most of the arguments for how and why things might go wrong.
    I’ve been keeping across most developments. I’ve been following alignment efforts. I’ve done some thinking about the challenges involved.
    But now I feel it.
    Spending time observing ChatGPT – its abilities, its quirks, its flaws – has brought my feelings into step with my beliefs.
    I already appreciated why I should be concerned about AI. Like I say, I’d read the arguments, and I’d often agreed.
    But my appreciation took a detached, ‘I can’t fault the reasoning so I should accept the conclusion’ kind of form. I was concerned in the abstract, but I was never really worried. At least some of my concern was second-hand; people I respected seemed to care a lot, so I probably should too. It was forced.
    I spend a lot of time thinking about catastrophic risks. When I consider something like an engineered pandemic, I’ve always felt the danger. It comes naturally to me. It’s intuitive. The same goes for many other threats. But only in the last few weeks has that become true for AI.
    This is all a little embarrassing to admit. I know that ChatGPT isn’t an enormous leap from what existed previously. Yet it has significantly changed how I feel about AI risk. It’s made things click in a way that I can’t fully explain; all the arguments now hit with an added force.
    Those who’ve always 'felt it' are probably wondering what took me so long, and why now. I'm not sure. But I doubt my experience is unique. Hopefully some of you know exactly what I’m trying to convey here.
    Before I believed it. Now I feel it.
    And as much as I’d like to think otherwise, that makes a big difference.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Applications open for AGI Safety Fundamentals: Alignment Course by Jamie Bernardi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applications open for AGI Safety Fundamentals: Alignment Course, published by Jamie Bernardi on December 13, 2022 on The Effective Altruism Forum.
    The AGI Safety Fundamentals (AGISF): Alignment Course is designed to introduce the key ideas in AGI safety and alignment, and provide a space and support for participants to engage, evaluate and debate these arguments. Participants will meet others who are excited to help mitigate risks from future AI systems, and explore opportunities for their next steps in the field.
    The course is being run by the same team as for previous rounds, now under a new project called BlueDot Impact.
    Time commitment
    The course will run from February-April 2023. It comprises 8 weeks of reading and virtual small-group discussions, followed by a 4-week capstone project.
    The time commitment is around 4 hours per week, so participants can engage with the course alongside full-time work or study.
    Course structure
    Participants are provided with structured content to work through, alongside weekly, facilitated discussion groups. Participants will be grouped depending on their ML experience and background knowledge about AI safety. In these sessions, participants will engage in activities and discussions with other participants, guided by the facilitator. The facilitator will be knowledgeable about AI safety, and can help to answer participants’ questions.
    The course is followed by a capstone project, which is an opportunity for participants to synthesise their views on the field and start thinking through how to put these ideas into practice, or start getting relevant skills and experience that will help them with the next step in their career.
    The course content is designed by Richard Ngo (Governance team at OpenAI, previously a research engineer on the AGI safety team at DeepMind). You can read the curriculum content here.
    Target audience
    We are most excited about applicants who would be in strong position to pursue technical alignment research in their career, such as professional software engineers and students studying technical subjects (e.g. CS/maths/physics/engineering).
    That said, we consider all applicants and expect 25-50% of the course to consist of people with a variety of other backgrounds, so we encourage you to apply regardless. This includes community builders who would benefit from a deeper understanding of the concepts in AI alignment.
    We will be running another course on AI Governance in early 2023 and expect a different distribution of target participants.
    Apply now!
    If you would like to be considered for the next round of the courses, starting in February 2023, please apply here by Thursday 5th January 2023. More details can be found here. We will be evaluating applications on a rolling basis and we aim to let you know the outcome of your application by mid-January 2023.
    If you already have experience working on AI alignment and would be keen to join our community of facilitators, please apply to facilitate.
    Who is running the course?
    AGISF is now being run by BlueDot Impact - a new non-profit project running courses that support participants to develop the knowledge, community and network needed to pursue high-impact careers. BlueDot Impact spun out of Cambridge Effective Altruism, and was founded by the team who was primarily responsible for running previous rounds of AGISF. You can read more in our announcement post here.
    We’re really excited about the amount of interest in the courses and think they have great potential to build awesome communities around key issues. As such we have spent the last few months:
    Working with pedagogy experts to make discussion sessions more engaging
    Formalising our course design process with greater transparency for participants and facilitators
    Building systems to improve participant networking...
    5 min