Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Update on Harvard AI Safety Team and MIT AI Alignment, published by Xander Davies on December 2, 2022 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Update on Harvard AI Safety Team and MIT AI Alignment, published by Xander Davies on December 2, 2022 on LessWrong.

We help organize the Harvard AI Safety Team (HAIST) and MIT AI Alignment (MAIA), and are excited about our groups and the progress we’ve made over the last semester.

In this post, we’ve attempted to think through what worked (and didn’t work!) for HAIST and MAIA, along with more details about what we’ve done and what our future plans are. We hope this is useful for the many other AI safety groups that exist or may soon exist, as well as for others thinking about how best to build community and excitement around working to reduce risks from advanced AI.

Important things that worked:

Well-targeted outreach, which (1) focused on the technically interesting parts of alignment (rather than its altruistic importance), and (2) leveraged informal connections with networks and friend groups.

HAIST office space, which was well-located and very useful for running HAIST’s programming and co-working.

Well-contextualized leadership, with many of the people involved in running HAIST/MAIA programming having experience with AI safety research (including nearly all of the facilitators for our reading groups).

High-quality, scalable weekly reading groups, including 13 sections of introductory reading groups, 2 science of deep learning reading groups, 2 policy reading groups, and general member reading groups for HAIST and MAIA.

Significant time expenditure, including mostly full-time attention from several organizers.

Important things we got wrong:

Poor retention for MAIA programming, perhaps due to starting this programming too late in the semester.

Excessive focus on intro programming, which cut against ML engineering programming and advanced reading groups for more seasoned members.

If you’re interested in supporting the alignment community in our area, the Cambridge Boston Alignment Initiative is currently hiring.

What we’ve been doing

HAIST and MAIA are concluding a 3-month period during which we expanded from one group of about 15 Harvard and MIT students who read AI alignment papers together once a week to two large student organizations that:

Ran a large AI safety intro fellowship (organized by Sam Marks and adapted from the AGI Safety Fundamentals program) that attracted over 230 applicants and enrolled about 130 in 13 weekly reading groups, all facilitated by people with experience in AI safety research (some of whom are students). About 60 participants have continued attending as of this post, including undergraduates, grad students, and postdocs in math and computer science.

This wound up taking up most of our focus, which was not the original plan (we planned to spend more time on ML up-skilling and supporting research). This pivot was mostly intentional (we got a higher number of great applicants than expected), but we are worried about continually prioritizing our introductory program in the future (which we discuss further below).

Opened the HAIST office (with the significant help of Kaleem Ahmid, Frances Lorenz, and Madhu Sriram), which has become a vibrant coworking space for alignment research and related work. We plan to open a MAIA office, officially starting in February 2023.

Launched the MAIA/HAIST Research Fellows program (organized by Oam Patel), which paired 20 undergraduate and graduate students with AI safety and governance research mentors.

Started a Science of Deep Learning reading group at MAIA (organized by Eric Michaud), with around 8 active participants (more information about the reading group here). This program ended up being good for experienced member engagement and generating research ideas, but didn’t perform as well as an outreach mechanism (initial intention).

Ran two retreats (organized by Trevor Levin and Kuhan Jeyapragasan), w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How does the collapse of the FTX Future Fund change the picture for individual donors?, published by vipulnaik on December 1, 2022 on The Effective Altruism Forum.

In a post in

October, I outlined my thoughts at the time regarding where I'll make my end-of-year donation. In there, I noted that while I think that many cause areas that fall under "longtermism" are likely very important, the significant funding being directed at them from FTX Future Fund and Survival and Flourishing Fund makes these causes

non-neglected. That was one of the reasons I was leaning toward making my end-of-year donation to the Animal Welfare Fund instead of the Long-Term Future Fund.

This was before the November 2022 collapse of FTX and of the FTX

Future Fund. I'm wondering how to think about the effect of the collapse of the FTX Future Fund on the funding available for

"longtermist" projects in general, and how this should affect individual donors such as myself.

In particular, some of the things I've been wondering are:

How effectively does the Long-Term Future Fund (the main donation option for individual donors) funge with FTX Future Fund in terms of the projects and organizations funded? How much does the collapse of the FTX Future Fund increase the room for more funding for the

Long-Term Future Fund?

Are there specific other donees that have become particularly relevant for individual donors in light of the collapse of the FTX

Future Fund? An example (that I don't know much about) is the

Nonlinear Emergency

Fund intended to help FTX grantees, that seemed to indicate that they have room for more funding.

Does the continually developing nature of the FTX collapse situation make it more important to hold funds for now and donate them a little later once the ramifications are clearer? Or is it the opposite, namely, that there is a more urgent need for funds and therefore it's more important to donate more now?

Are there other related questions and considerations that I'm missing?

I don't know how the answers to the above questions will affect my donation decision (it's possible that my donation decision will ultimately be influenced by personal factors specific to me). I hope that any answers or comments generated by this post will be helpful not just to me but to other potential donors wondering how the FTX collapse situation affects them.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I’m a 22-year-old woman involved in Effective Altruism. I’m sad, disappointed, and scared., published by Maya D on December 2, 2022 on The Effective Altruism Forum.

Before I get to the heart of what I want to share, a couple of disclaimers:

I am one person and, as such, I recognize that my perspective is both informed and limited by my experience and identity. I would like to share my perspective in the hope that it may interact with and broaden yours.

In writing this post, my aim is not to be combative nor divisive. The values of Effective Altruism are my values (for the most part - I likely value rationality less than many), and its goals are my goals. I do not, therefore, aim to “take down” or harm the Effective Altruism community. Rather, I hope to challenge us all to think about what it means to be in community.

Who am I and why am I writing this?

I’m a 22-year-old college senior, set to graduate with a degree in Human and Organizational Development in May. I learned about Effective Altruism last fall, when I transferred to a new university and started attending my school’s EA events and fellowships. I have become increasingly involved in the Effective Altruism community over the past 14 months - participating in intro and in-depth fellowships, taking on a leadership role in my school’s EA club, and attending an EAGx and an EAG.

So why am I writing this? Because I am at a point in my life where I have to make a lot of choices: where I want to live; what type of work I want to engage in; the people whom I want to surround myself with. When I found Effective Altruism, it seemed as though I had stumbled across a movement and a community that would provide me with guidance in all three of these areas. However, as my social and intellectual circles became increasingly entangled with EA, I grew hesitant, then skeptical, then downright sad as I observed behavior (both in-person and online) from those involved in the EA community. I’m writing this because I want to be able to feel proud when I tell people that I am involved in Effective Altruism. I want to feel as if I can encourage others to join without an added list of disclaimers about the type of behavior they may encounter.

Lastly, I want the Effective Altruism community to revisit and continuously strive towards what the Centre for Effective Altruism calls the core principles of EA: commitment to others, scientific mindset, openness, integrity, and collaborative spirit.

Gender and Culture

According to the EA Survey 2020 (the latest year for which I could find data), the makeup of people involved in EA was very similar to that of 2019: 76% white and 71% male. Lack of diversity within movements, organizations, and “intellectual projects” is incredibly damaging for many important reasons. CEA writes several of these reasons on their Diversity and Inclusion page, but the one I would like to highlight is We don’t want to miss important perspectives. As the website reads, “if the community isn’t able to welcome and encourage members who don’t resemble the existing community, we will consistently miss out on the perspectives of underrepresented groups.” I agree with this statement and that is why I am concerned - I’m unconvinced that this community is effective in “welcoming and encouraging” people who don’t fit the majority white-and-man mold.

I question the welcoming-ness of the EA community because, despite fitting the EA mold in many ways - I’m white, American, in my twenties, and will soon graduate from a highly-ranked college - I still often feel as if the EA community is not something I want to be a part of. I can imagine that those with even less of the predominant EA demographic characteristics face exponentially increased barriers to entry. Several (yet not all) instances in which I’ve felt this way.

A friend ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FYI: CC-BY license for all new Forum content from today (Dec 1), published by Will Bradshaw on December 1, 2022 on The Effective Altruism Forum.

From today, all new Forum content (including comments and shortform posts) will be published under CC-BY:

Therefore, as of December 1, 2022, we are requiring that all content posted to the Forum be available under a CC BY 4.0 license.

There's not currently much on the Forum making people aware of this change, especially for minor content like comments and shortforms. Even though I support the change, I think it would be pretty bad if a user accidentally licensed their content under CC-BY due to lack of information, so I'm posting this as a transitional heads-up.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Re-Examining LayerNorm, published by Eric Winsor on December 1, 2022 on LessWrong.

Please check out the colab notebook for interactive figures and more detailed technical explanations.

This post is part of the work done at Conjecture.

Special thanks to Sid Black, Dan Braun, Carlos Ramón Guevara, Beren Millidge, Chris Scammell, Lee Sharkey, and Lucas Teixeira for feedback on early drafts.

There's a lot of non-linearities floating around in neural networks these days, but one that often gets overlooked is LayerNorm. This is understandable because it's not "supposed" to be doing anything; it was originally introduced to stabilize training. Contemporary attitudes about LayerNorm's computational power range from "it's just normalizing a vector" to "it can do division apparently". And theories of mechanistic interpretability such as features as directions and polytopes are unhelpful, or even harmful, in understanding normalization's impact on a network's representations. After all, normalization doesn't alter the direction of vectors, but it still bends lines and planes (the boundaries of polytopes) out of shape. As it turns out, LayerNorm can be used as a general purpose activation function (you can solve MNIST with a LayerNorm MLP, for example). Concretely, it can do things like this:

and this:

We will explain what's going on in these animations later, but the point is that to develop a strong, principled theory of mechanistic interpretability, we need to grapple with this non-linearity.

In this interactive notebook, we study LayerNorm systematically using math and geometric intuition to characterize the ways in which it can manipulate data. We show that the core non-linearity of LayerNorm can be understood via simple geometric primitives. We explain how these basic primitives may perform semantic operations. For example, folding can be viewed as extracting extremal features (e.g. separating extreme temperatures from normal ones). We leverage these primitives to understand more complex low-dimensional classification tasks with multiple layers of non-linearities. The methods and intuition developed here extend to non-linearities beyond LayerNorm, and we plan to extend them in future work. Below is an interactive summary of the content of the notebook. (If you find the summary interesting, you really should check out the notebook: manipulating the figures helps build geometric intuition.)

Summary

The formula for LayerNorm is something messy like

But it turns out the core non-linear operation is (almost) normalizing a vector:

Graphically, this function has the iconic sigmoid shape in one dimension (note that in 1D the norm is simply the absolute value).

Interesting things start happening when we precompose this normalization function with affine transformations (such as scaling and shifting). Below, we start with a collection of points, (x,y), distributed uniformly on the sphere. Then we compute uϵ(tx,y) (stretch/shrink by a factor of t in the x direction and then normalize). Since normalization guarantees points end up on the circle, this operation stretches the distribution along the circle. The right hand panel illustrates how this stretching is captured in the shape of a 1D activation function (the x coordinate of the input against the x coordinate of the output). For example, when t is close to 5, we see that any points with x coordinates not near 0 get compressed towards x=−1 or x=1. This matches the picture in the circle where we see the points bunch up at the left and right sides of the circle.

If we make the same plots with a shifting operation, uϵ(x+t,y), the operation folds the input data.

We can envision possible "semantic" uses for these geometric operations. Stretching can be used to perform an approximate "sign" operation (as in + or - sign). For example, it ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Cambridge Boston Alignment Initiative [Hiring!], published by kuhanj on December 2, 2022 on The Effective Altruism Forum.

TLDR: The Cambridge Boston Alignment Initiative (CBAI) is a new organization aimed at supporting and accelerating Cambridge and Boston students interested in pursuing careers in AI safety. We’re excited about our ongoing work, including running a winter ML bootcamp, and are hiring for Cambridge-based roles (rolling applications, priority deadline Dec. 14 to work with us next year).

We think that reducing risks from advanced AI systems is one of the most important issues of our time, and that undergraduate and graduate students can quickly start doing valuable work that mitigates these risks.

We (Kuhan, Trevor, Xander and Alexandra) formed the Cambridge Boston Alignment Initiative (CBAI) to increase the number of talented researchers working to mitigate risks from AI by supporting Boston-area infrastructure, research and outreach related to AI alignment and governance. Our current programming involves working with groups like the Harvard AI Safety Team (HAIST) and MIT AI Alignment (MAIA), as well as organizing a winter ML bootcamp based on Redwood Research’s MLAB curriculum.

We think that the Boston and Cambridge area is a particularly important place to foster a strong community of AI safety-interested students and researchers. The AI alignment community and infrastructure in the Boston/Cambridge area has also grown rapidly in recent months (see updates from HAIST and MAIA for more context), and has many opportunities for improvement: office spaces, advanced programming, research, community events, and internship/job opportunities to name a few.

If you’d like to work with us to make this happen, we’re hiring for full-time generalist roles in Boston. Depending on personal fit, this work might take the form of co-director, technical director/program lead, operations director, or operations associate. We will respond to applications submitted by December 14 by the end of the year. For more information, see our website. For questions, email [email protected].

We’ll also be at EAGxBerkeley, and are excited to talk to people there.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The LessWrong 2021 Review (Intellectual Circle Expansion), published by Ruby on December 1, 2022 on LessWrong.

If you're already familiar with the LessWrong annual review, feel free to skip this opening and jump to What's new in 2021? or Unveiling the 2020 Annual Review books

The tradition continues! We will do for 2021 the same as we did for 2018, 2019, and 2020: approximately one year after the year is complete, we look back with the benefit of some distance and ask how good were the posts written then? After the hype has died down, do their claims still seem valid? Were they useful or interesting in a lasting way? Etc.

On LessWrong we're trying to make intellectual progress on problems that matter, but it can be hard to tell whether we're making progress. The Annual Review is the once-yearly Schelling time to reflect, debate, discuss and vote. It's a poke to say "remember that idea you and everyone else were obsessed with in 20xx? Did it continue to matter or was it just a fad?" It's also the time when we can determine which contributions most affected people in the long-term, and reward those authors and inspire others to follow suit. I think it's pretty great that when an author who invested hours into a post and greatly benefits the community, the community eventually recognizes and rewards them in some fashion.

The full technical details of how the Annual Review works is described in the final section of this post [link], but the core of it is the same as the past few years. There are three phases:

Preliminary Voting Phase (Dec 1- 14): Site members start identifying posts especially worthy of consideration in the review. Posts with 2 preliminary votes move into the Discussion Phase.

Discussion Phase (Dec 14 - Jan 11): Site members begin reviewing and debating posts in earnest. This discussion will inform members on how to cast their final votes. Posts that receive at least one written review move to the Final Voting phase.

Final Voting (Jan 11 to Jan 25): Site members allocate their vote budget (quadratic voting) amount nominated posts. The outcome of the final determines the Annual Review results.

For more of the philosophy of the Annual Review, see the previous announcement posts here and here and here.

Getting Started

If you are super eager to get started participating in the review, you can start preliminary voting on posts from the All 2021 Posts page, or the View Your Past Upvotes page. Note: only users with accounts registered before January 2021 are eligible to vote.

New in 2021: Offsite Content

Largely, the system is the same. With one exception–and this might be a terrible idea because it was a last-minute decision and it isn't fully thought out yet:

This year, we are allowing people to submit off-site, non-LessWrong content to the Review. If you want something posted on another website, e.g. ACX or arXiv, to be considered for the review, make a linkpost for it and then let the LessWrong mod team know, we will backdate and it can be included. (Let us know by writing a comment on this post or contacting us).

What kind of external resources are worth including?

There are two kinds of content that seem particularly worth including in my mind:

Content that was broadly part of the community's intellectual diet even if it wasn't posted on LessWrong. For example, suppose we were doing the Annual Review of 2014 and Meditations on Moloch hadn't been crossposted to LessWrong. Since it was a major contribution to the community's intellectual commons, I think it would have been good to include it in that year's review.

Highly impactful Alignment content that's not on LessWrong/Alignment Forum. AI Alignment is the most notable specific intellectual problem our community is trying to tackle, and I'd like to see our work evaluated alongside major contributions on t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA needs more humor, published by SWK on December 1, 2022 on The Effective Altruism Forum.

In the wake of the FTX collapse, much ink has been spilled on EA reform by people smarter and more experienced in this space than I am. However, as someone who has been engaging with EA over the past few years and who has become increasingly connected with the community, I have a modest proposal I’d like to share: EA needs more humor.

Criticism of EA has roots back in the “earning to give” era. This Stanford Social Innovation Review editorial from 2013 describes EA as “cold and hyper-rationalistic,” deeming the idea of numerically judging charities as “defective altruism.” This piece in Aeon from 2014 essentially argues that the EA utilitarian worldview opposes art, aesthetic beauty, and creativity in general.

Criticism of EA has only heightened in recent years with the rise of longtermism. Another Aeon editorial from 2021 characterizes the “apocalypticism” of longtermist thought as “profoundly dangerous” while also lampooning EA organizations like the “grandiosely named Future of Humanity Institute” and “the even more grandiosely named Future of Life Institute.”

In the last few months before the FTX situation, criticism was directed at Will MacAskill’s longtermist manifesto, What We Owe the Future. A Wall Street Journal review concludes that “‘What We Owe the Future’ is a preposterous book” and that it is “replete with highfalutin truisms, cockamamie analogies and complex discussions leading nowhere.” A Current Affairs article once again evokes the phrase “defective altruism” and asserts that MacAskill’s book shows how EA as a whole “is self-righteous in the most literal sense.”

The above example are, of course, just a small snapshot of the criticism EA has faced. However, I think these examples capture a common theme in EA critiques. Overall, it seems that critics tend to characterize EA as a community of cold, calculating, imperious, pretentious people who take themselves and their ostensible mission to “save humanity” far too seriously.

To be honest, a lot of EA criticism seems like it’s coming from cynical, jaded adults who relish in the opportunity to crush young people’s ambitious dreams about changing the world. I also think many critics don’t really understand what EA is about and extrapolate based on a glance at the most radical ideas or make unfair assumptions based on a list of EA’s high-profile Silicon Valley supporters.

However, there is a lot of truth to what critics are saying: EA’s aims are incredibly ambitious, its ideas frequently radical, and its organizations often graced with grandiose names. I also agree that the FTX/SBF situation has exposed glaring holes in EA philosophy and shortcomings in the organization of the EA community.

However, my personal experience in this community has been that the majority of EAs are not cold, calculating, imperious, pretentious people but warm, intelligent, honest, and altruistic individuals who wholeheartedly want to “do good better.”

I think one thing the EA community could do moving forward to improve its external image and internal function is to embrace a bit more humor. EA could stand to acknowledge and make fun of the craziness of comparing the effectiveness of charities as disparate as a deworming campaign and a policy advocacy group, or the absurdity of a outlining a superintelligent extinction event.

I say these ideas are absurd not because I don’t believe in them; I have the utmost respect for rigorous charity evaluators like GiveWell and am convinced that AI is indeed the most important problem facing humanity. But I think that acknowledging the external optics of these ideas and, to a degree, joking about how crazy they may seem could make EA less disagreeable for many people on the outside looking in.

There ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Biases are engines of cognition, published by remember on November 30, 2022 on LessWrong.

This work was done while at Conjecture.

Humans are not perfect reasoners. Cognitive biases are not biases; they are engines of cognition, each exploiting regularities of the universe. Combining enough of these “cognitive biases” was enough to create a general intelligence but doesn’t even approximate perfect reasoners.

These cognitive biases initially arose through the directed search process of evolution. In evolution, it is far easier to identify regularities in the universe and build up biases that leverage these local regularities, than to figure out how to reason apriori. So we are built to rely on our intuition and vibes in social settings rather than explicitly calculate who knows what. Once enough biases have accumulated, and enough regularities have been taken advantage of, these biases combine to form human general intelligence.

Human general intelligence then launched a more directed search process in the form of culture and transfer of knowledge and practices. These were found in a fairly ad-hoc fashion and constructed on top of previous biases that form human intelligence. So we rely on biased intuitions to decide who to trust, who to emulate, and then we emulate and learn from them, allowing us to pass knowledge and know-how between generations. This takes the form of learning strategies suited to our environment, both natural and social.

Even science, which seeks deeper mechanistic understanding, is still a directed search process that targets what regularities we can find in the universe. We don't reason from quantum mechanics to a unified theory of everything. Instead, we often isolate and attack problems as we come across them, build a fragmentary understanding, and only integrate later, if we do at all. Despite the importance of molecular mechanisms in understanding evolution, Darwin first constructed his theory based on the similarities between species and a vague understanding that the world was very old. We didn’t understand the underlying mechanism at the time; all we could do was construct scaffolds of understanding around these local regularities and push deeper when more of the terrain was revealed.

So biases are engines of cognition that can be accumulated through directed search. The end result is interlocking biases tied to understanding and exploiting specific regularities. This is not perfect reasoning, nor even an approximation thereof. But somehow, it was enough to add up to general intelligence.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.