The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Broken Benchmark: MMLU, published by awg on August 29, 2023 on LessWrong.

Phillip over at the AI Explained channel has been running some experiments on his SmartGPT framework against the MMLU benchmark and discovered a not-insignificant amount of issues with the problem set.

Among them:

Crucial context missing from questions (apparently copy-paste errors?)

Ambiguous sets of answers

Wrong sets of answers

He highlights a growing need for a proper benchmarking organization that can research and create accurate, robust, sensible benchmarking suites for evaluating SOTA models.

I found this video to be super interesting and the findings to be very important, so I wanted to spread this here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper Walkthrough: Automated Circuit Discovery with Arthur Conmy, published by Neel Nanda on August 29, 2023 on The AI Alignment Forum.

Arthur Conmy's Automated Circuit Discovery is a great paper that makes initial forays into automating parts of mechanistic interpretability (specifically, automatically finding a sparse subgraph for a circuit). In this three part series of Youtube videos, I interview him about the paper, and we walk through it and discuss the key results and takeaways. We discuss the high-level point of the paper and what researchers should takeaway from it, the ACDC algorithm and its key nuances, existing baselines and how they adapted them to be relevant to circuit discovery, how well the algorithm works, and how you can even evaluate how well an interpretability method works.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I am hiring an Executive Research Assistant - improve RP while receiving mentorship, published by Peter Wildeford on August 29, 2023 on The Effective Altruism Forum.

I'm Peter Wildeford, co-CEO at Rethink Priorities (RP). RP is a research and implementation group that identifies pressing opportunities to make the world better. We act upon these opportunities by developing and implementing strategies, projects and solutions to key issues. We do this work in close partnership with foundations and impact-focused non-profits.

I'm excited to be hiring a new Executive Research Assistant (ERA) role to work directly with me to help fill important gaps in RP's needs, and keep me and my fellow co-CEO Marcus consistently informed and prepared. See more detail + apply here.

The work is incredibly varied and will give you a diversity of experience. Past ERA projects have included varied tasks such as creating a risk management framework for RP, summarizing relevant research papers, interviewing different relevant experts about how to tackle a particular problem, producing EA Forum summaries, responding to co-CEO inbound email correspondence in a timely and organized manner, and ensuring meetings with co-CEOs and others are scheduled.

Project selection is adaptable to the candidate's interests and can fit more junior candidates (eg. more scheduling) and more senior candidates (eg. more research, project management, or stakeholder outreach).

The role also comes with significant career mentoring. In fact, the reason we are hiring for this role is that past people in this position have already moved on to significant more senior roles at RP. As your manager I will meet with you regularly to understand your career goals, help you test fit for various roles, and put you on a path to success. The work itself will also build varied skills and connections within the organization and potentially some of our external stakeholders.

Some fast facts about the role:

Deadline to apply - September 17

Role is permanent, full or part time with a minimum of 20hrs/week

The role is remote and we are able to hire in most countries

The compensation is $69,000 - $87,000 USD/year depending on experience (the amount is calculated using RP's salary algorithm and is dependent on prior relevant experience and corresponding title level)

See more detail + apply here.

A pitch for applying:

You could make an immediate impact on a large organization at the CEO level

Work at RP influences the decision-making of major foundations spending hundreds of millions of dollars

You will receive useful career mentorship and the opportunity to test fit and build skills for a variety of future roles based on your interests

Good work-life balance, generous paid time off leave, and generous benefits including parental leave of up to 6 months during the first 2 years after, or split before and after post a child's birth or adoption for parents of all genders

Prior people holding this role have said:

"I can say from first-hand experience that this is a great job - You get to do important, interdisciplinary work, and the Co-CEOs are super supportive of you finding and developing your own strengths"

"This role was fantastic for helping me build up relevant subject matter expertise and test fit for roles that would have been difficult to enter directly. There's also a lot of chances for direct impact - your ideas are listened to, and there's lots of autonomy such that the way you approach things and the specific skills you bring will make a real difference to outcomes of the projects you take on."

This role would be ideal for candidates who demonstrate:

Strong writing skills, especially the ability to write concisely yet accurately

Ability to accurately and quickly interpret research from a wide variety of fields, including fields yo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An OV-Coherent Toy Model of Attention Head Superposition, published by LaurenGreenspan on August 29, 2023 on The AI Alignment Forum.

Background

This project was inspired by Anthropic's post on attention head superposition, which constructed a toy model trained to learn a circuit to identify skip-trigrams that are OV-incoherent (attending from multiple destination tokens to a single source token) as a way to ensure that superposition would occur. Since the OV circuit only sees half of the information - the source tokens - the OV circuit of a single head cannot distinguish between multiple possible skip-trigrams. As long as there are more skip-trigrams with the same source-token to represent than heads, the model cannot represent them in the naive way, and may resort to superposition.

In a more recent update post, they found that the underlying algorithm for OV-incoherent skip-trigrams in a simpler 2-head model implemented a conditional on the source token. One head predicts the output for the skip trigram [current token] . [current token] -> [ground truth([0]...[current token])], one of which will yield the right answer. The second head destructively interferes with this result by writing out the negative logit contribution of the first head if the source token is not the one common to all skip-trigrams (in this case, [0]). Because their example cleanly separated tasks between the two attention heads, the authors argued that it was more like the building of high-level features out of low-level ones than a feature superimposed across multiple attention heads.

OV-coherent Superposition

Instead, we claim there is an analogous force pushing the model toward adopting a distributed representation/head superposition whenever the model must learn patterns that require implementing nonlinear functions of multiple source tokens given a fixed destination token. We call this "OV-coherent" superposition: despite of the information at the destination position being fixed, the information copied from an attended-to token depends on the information at source tokens to which it is not attending. This pushes the model to form interference patterns between heads attending to different tokens.

To test this, we implemented a 1-layer, attention-only toy model with one-hot (un)embeddings trained to solve a problem requiring attention to multiple source tokens, described below. Here, we focus on a 2-head model which solves the task with perfect accuracy, and lay out some interesting motifs for further investigation.

Key Takeaways:

Heads in our model seem to implement nested conditional statements that exploit the if-else nature of the QK circuits. This means they can learn to write more specific information conditional on attending to certain tokens, given that it can implicitly rule out the existence of other tokens elsewhere in the context. The heads furthermore implement these nested conditionals in such a way that they distribute important source tokens between them, and constructively interfere to produce the correct answer.

Most of the time, we found that this "conditional dependence" relies on heads implementing an "all or nothing" approach to attention. Heads do not generally spread their attention across multiple interesting tokens, but instead move through the hierarchy of features in their QK circuits and attend to the most "interesting" (still a poorly defined term!) one present. This seems to be a common property of attention patterns in real-model heads as well.

When there are multiple important source tokens to attend to in the context, heads implementing interference schema will tend to learn QK circuits such that they distribute tokens amongst themselves and don't leave crucial information unattended to. In 2-head models, this manifests are reversed "preference orderings" ov...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dating Roundup #1: This is Why You're Single, published by Zvi on August 29, 2023 on LessWrong.

Developments around relationships and dating have a relatively small speed premium, so I figured I would wait until I had a full post worth of them.

Indeed I now present such a post, in which I present several theories as to why so many of you might still be single.

While I am my usual opinionated self, I am not going to be offering a section of my list of related Good Advice. That would be its own project, which may or may not happen at some time in the future. There is still much in the way of practical implications or implied advice throughout.

You're Single Because You're Not Even Trying

A 2022 sample of singles is out, and charts are available, so that seems like a good place to start. None of this is properly representative or anything. It's still good data.

It is reasonable for a quarter of singles to not want a relationship for whatever reason. What is not so reasonable is for the vast majority of those who do want one to not be making any attempt at finding one.

It is not always the case that if you want a relationship and you don't have one, you should be actively looking for one. It is definitely not he case two-thirds of the time that you want a relationship that it is not worth actively looking.

The situation is getting worse.

This is usually not a question you want to leave to fate. If you want, go seek and you might find. If you do not want, or do not seek, you probably don't get.

This is what not dating looks like.

Assuming as Alexander does that the No Response are the 0s above, this says that almost no currently single people, less than 20%, go on multiple first dates in a year.

I am not saying that dating is easy or that I found it to be easy. I will go ahead and say it is not once-a-year level hard for most people to find worthwhile first dates.

What I especially find curious is that one is the most popular response rather than zero. It would make sense to me that the answer is frequently zero dates, because you are not trying and aren't 'date ready' in various senses. What's super weird is that the vast majority did go on the one first date, but mostly they didn't go on a second, and only half of those went on a third. It is as if people are capable of getting a date, then they go on one and recoil in 'oh no not that again' horror for about a year, then repeat the cycle? Or their friends set them up every year or so because it's been too long, or something? None of that makes sense to me.

Alternatively, what the data is also saying is that getting a first date is indeed the primary barrier to finding a relationship. If you went on four or more first dates in the past year, which is one every three months or ~1% of nights, then it is highly unlikely you are single.

There is the stereotype of the person (usually but not always a woman) who goes on dates constantly, finding an endless string of losers. The data here suggests that this essentially is not a thing, or that if you do that it works.

Dating app use is surprisingly small even now.

The above seems very much like a world of people who are not trying. The 17% rate of using dating apps roughly corresponds to the percentage of people trying at all.

Polyamory is not a popular ideal sexual relationship (note this adds to 71%).

There is more analysis but the big lesson seems very clear - people who are single have mostly opted out or at least are very much not trying.

Causation could however go either way. If no one single is trying, that could be because everyone who tries will succeed. Or it could be because a lot of people are doomed to failure if they try, and they have learned this so they stopped trying.

Alexander also has another thread with great data, and this jumped out there.

That is rather i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Giving gladly, giving publicly, published by HenryStanley on August 29, 2023 on The Effective Altruism Forum.

A short post on my blog detailing a recent donation - and my intention to be a bit more public about my giving in future.

I'm never sure of the expected value of these sorts of posts; they feel a bit awkward and braggy to me. But I do want to get more people into philanthropy and the mindset that they can make a difference even with (relatively) small sums of money,

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Preliminary Analysis of Intervention to Reduce Lead Exposure from Adulterated Turmeric in Bangladesh Shows Cost Benefit of About US$1 per DALY, published by Kate Porterfield on August 29, 2023 on The Effective Altruism Forum.

Pure Earth is a GiveWell Grantee dedicated to reducing lead exposure in low- and middle-income countries. In collaboration with Stanford University and the International Centre for Diarrhoeal Disease Research, Bangladesh (icddr,b), a preliminary cost-effectiveness analysis (CEA) was performed to assess the effectiveness of an intervention in Bangladesh. The CEA presents an encouraging outlook, with a cost per disability-adjusted life year (DALY)-equivalent averted estimated at just under US$1. As this assessment is preliminary, it may contain methodological inconsistencies with GiveWell's. As such, we welcome any comments and corrections.

In 2019, after investigations concluded that turmeric was the primary source of lead exposure among residents of rural Bangladesh, Stanford University and Bangladeshi non-profit icddr,b embarked on a mission to eliminate lead poisoning from turmeric. Stanford and icddr,b's investigations had revealed that lead chromate (an industrial pigment sometimes called "School Bus Yellow") was being added to turmeric roots to make them more attractive for sale. Armed with this evidence, the team coordinated with the Bangladeshi Food Safety Authority to conduct a crack-down of the adulteration by enforcing policies at the markets and raising awareness among businesspeople and the public nationwide. These efforts successfully halted the practice of adding lead chromate to turmeric: the prevalence of lead in turmeric dropped from 47% in 2019 to 0% in 2021.

In collaboration with Pure Earth, icddr,b continues to monitor turmeric and other spices and coordinate with government agencies to maintain the safety of these and other food products.

To gauge the effectiveness of this program in advancing the mission of reducing lead exposures globally, it is important to assess both impact and cost-effectiveness. To approach this task, Pure Earth and Stanford have completed a back-of-the-envelope cost-effectiveness analysis (CEA), incorporating preliminary data from blood lead level assessments and various assumptions. This model is built off of previous models created by LEEP and Rethink Priorities.

The preliminary findings are that this program can avert an equivalent DALY for just under $1. This result is extraordinary, albeit deserving of further scrutiny. It indicates that certain interventions in the lead space could be enormously cost-effective. The body of work to reduce lead exposures in LMICs is nascent, and not all interventions are likely to be as cost-effective as spices. But clearly, more work on spices is called for, and Pure Earth, Stanford, icddr,b, and others are pursuing funding to expand these programs into other countries.

Program Implementation Costs

To establish a framework for cost-effectiveness assessment, it is essential to define the terms "cost" and "effectiveness" within the context of the Stanford-led mission. The concept of "cost" encompasses the resources utilized by the project team and those expended by the Bangladesh government as a direct result of the project's activities. Specifically, we consider monetary expenses incurred by the program, which we estimate to be upfront costs of $360,000. These expenses include both the costs to identify the sources of lead exposure and implement the program, as well as continuing costs of $100,000 to monitor and continue the program after the initial implementation. Additionally, the Government of Bangladesh is expected to spend $100,000 over the course of the intervention.

To facilitate comparisons with other global health interventions, we define the project's"...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Barriers to Mechanistic Interpretability for AGI Safety, published by Connor Leahy on August 29, 2023 on The AI Alignment Forum.

I gave a talk at MIT in March earlier this year on barriers to mechanistic interpretability being helpful to AGI/ASI safety, and why by default it will likely be net dangerous. Several people seem to be coming to similar conclusions recently (e.g., this recent post).

I discuss two major points (by no means exhaustive), one technical and one political, that present barriers to MI addressing AGI risk:

AGI cognition is interactive. AGI systems interact with their environment, learn online and will externalize massive parts of their cognition into the environment. If you want to reason about such a system, you also need a model of the environment. Worse still, AGI cognition is reflective, and you will also need a model of cognition/learning.

(Most) MI will lead to capabilities, not oversight. Institutions are not set up and do not have the incentives to resist using capabilities gains and submit to monitoring and control.

This being said, there are more nuances to this opinion, and a lot of it is downstream of lack of coordination and the downsides of publishing in an adversarial environment like we are in right now. I still endorse the work done by e.g. Chris Olah's team as brilliant, but extremely early, scientific work that has a lot of steep epistemological hurdles to overcome, but I unfortunately also believe that on net work such as Olah's is at the moment more useful as a safety-washing tool for AGI labs like Anthropic than actually making a dent on existential risk concerns.

Here are the slides from my talk, and you can find the video here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Information warfare historically revolved around human conduits, published by trevor on August 29, 2023 on LessWrong.

Epistemic status: This is a central gear in any model of propaganda, public opinion, information warfare, PR, hybrid warfare, and any other adversarial information environment. Due to fundamental mathematical laws, like power law distribution, not all targets in the battlespace are created equal.

Generally, when people think about propaganda, censorship, public opinion, PR, and social psychology, they tend to think of the human receivers/viewers/readers/listeners as each being the ultimate goal - either the message is accepted, ignored, or rejected, either consciously or subconsciously, and each person either gains your side a single a point or loses you a point.

This is actually a bad model of large-scale influence. People with better models here are usually more inclined to use phrases like "steering public opinion", because the actual goal is to create human conduits, who internalize the message and spread it to their friends in personal conversations.

Messages that spread in that way are not only personalized influence by-default, even if initiated by early 20th century technology like radio or propaganda posters, but messages are also perceived as much more trustworthy when heard directly from the mouth of a friend than from one-to-many communication like broadcasting or posters, which were unambiguously spread by at least one elite with the resources to facilitate that and who you are not allowed to meet, even if they seem like a person just like you (although itiots might still fall for the "I'm an idiot like you" persona such as Donald Trump, Tucker Carlson, and particularly Alex Jones). Propaganda posters totally fell out of style, because it was plain as day that they were there to influence you; radio and television survived, including as a tool of state power, because they did not stick out so badly.

Censorship, on the other hand, is the prevention of this dynamic from emerging in the first place, which is likely a factor explaining why censorship is so widely accepted or tolerated by elites in authoritarian countries.

The battlespace

It's important to note that not all human conduits are created equal. This dynamic ultimately results in intelligent minds not merely relaying the message, but also using their human intelligence to add additional optimization power to the message's spread. On social media, this creates humans who write substantially smarter, more charismatic, eloquent, and persuasive one-liner posts and comments than those produceable by last-generation LLMs.

Furthermore, as an idea becomes more mainstream (and/or as the Overton window shrinks for rejecting the message), the conduits optimizing for the message's spread not only become greater in number, but also gain access to smarter speakers and writers. The NYT claims that Russian intelligence agencies deliberately send agents to the West to facilitate this process such as creating recruiting smart young people and creating grassroots movements that serve Russian interests; I've also previously had conversations with people who made strong arguments for and against the case that a large portion of the Vietnam antiwar movement was caused by Soviet agents orchestrating grassroots movements in-person in an almost identical way.

Slow takeoff

Social media and botnets, particularly acting autonomously due to increasingly powerful integrated AI, might be capable of navigating the information space as a battlespace, autonomously locating and focusing processing power on high-value targets who are predicted to be the best at adding a human touch to any particular message; a human touch that current or future LLMs might not be able to reliably generate on their own.

The technologic...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ACX Meetups Everywhere 2023: Times & Places, published by Scott Alexander on August 29, 2023 on LessWrong.

Thanks to everyone who responded to my request for ACX meetup organizers. Volunteers have arranged meetups in 169 cities around the world, from Baghdad to Bangalore to Buenos Aires.

You can find the list below, in the following order:

Africa & Middle East

Asia-Pacific

Europe

North America

South America

You can see a map of all the events on the LessWrong community page. You can also see a searchable sheet at this Airtable link.

Within each region, it's alphabetized first by country, then by city. For instance, the first entry in Europe is Sofia, Bulgaria, and the first entry for Germany is Bremen. Each region and country has its own header. The USA is the exception where it is additionally sorted by state, with states having their own subheaders. Hopefully this is clear. You can also just have your web browser search for your city by pressing ctrl+f and typing it if you're on Windows, or command+f and typing if you're on Mac. If you're on Linux, I assume you can figure this out.

Scott will provisionally be attending the meetup in Berkeley. ACX meetups coordinator Skyler will provisionally be attending Boston, Cavendish, Burlington, Berlin, Bremen, Amsterdam, Cardiff, London, and Berkeley. Some of the biggest ones might be announced on the blog, regardless of whether or not Scott or Skyler attends.

Extra Info For Potential Attendees

1. If you're reading this, you're invited. Please don't feel like you "won't be welcome" just because you're new to the blog, demographically different from the average reader, don't want to buy anything at the cafe or restaurant where it's held, or hate ACX and everything it stands for. You'll be fine! 2. You don't have to RSVP or contact the organizer to be able to attend (unless the event description says otherwise); RSVPs are mostly to give organizers a better sense of how many people might show up, and let them tell you if there are last-second changes. I've also given email addresses for all organizers in case you have a question.

Extra Info For Meetup Organizers:1. If you're the host, bring a sign that says "ACX MEETUP" and prop it up somewhere (or otherwise be identifiable).2. Bring blank labels and pens for nametags.3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier.4. If it's the first meetup, people are probably just going to want to talk, and if you try to organize some kind of "fun" "event" it'll probably just be annoying.5. It's easier to schedule a followup meetup while you're having the first, compared to trying to do it later on by email.6. In case people want to get to know each other better outside the meetup, you might want to mention reciprocity.io, the rationalist friend-finder/dating site.7.

If you didn't make a LessWrong event for your meetup (or if you did but Skyler didn't know about it) the LessWrong team did it for you using the email address you gave here. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).

If you need to change a meetup date or you have any other questions, please email skyler[at]rationalitymeetups[dot]org.

Africa & Middle East

Iraq

BAGHDAD, IRAQContact: Mustafa AhmedContact Info: tofiahmed117[at]gmail[dot]comTime: Friday, September 8th, 10:00 AMLocation: Grinders cafe, ZayounaCoordinates:+92Group Link:

Israel

TEL AVIV, ISRAELContact: InbarContact Info: inbar192[at]gmail[dot]comTime: Thursday, September 21st, 7:00 PMLocation: The grass area next to Max Brenner in Sarona park. I'll have an ACX signCoordinates:+MPGroup Lin...