The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - EU AI Act passed vote in Plenary meeting by Ariel G.
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EU AI Act passed vote in Plenary meeting, published by Ariel G. on June 16, 2023 on The Effective Altruism Forum.
    Mainly discussed in the linked article - keeping the post brief.
    The newest EU AI Act parliament version passed an important vote, despite recent political uncertainty.
    I found it fascinating to watch the the Plenary session (from June 13th, the vote was on the 14th), where the Act was discussed by various EU parties. A few things that stood out to me:
    I was surprised that many EU country representatives mentioned the Open Letters and Existential Risk as a real concern, even though the EU AI Act was not originally intended to address it (though, it now has GPAI/foundation model bits added). Transparency and Fairness took a back seat, to some extent.
    Real-time Biometric monitoring was a big debate topic - whether to giving an exemption for law enforcement or not, for national security. Currently it looks like it will not be allowed, other than post-incident with special approval. This may be a useful lever to keep in mind for policy work
    Others that watched the stream, feel free to mention insights in the comments.
    Linked here (relevant timestamp 12:39 - 14:33)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • EA - UN Secretary-General recognises existential threat from AI by Greg Colbourn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: UN Secretary-General recognises existential threat from AI, published by Greg Colbourn on June 16, 2023 on The Effective Altruism Forum.
    At the Digital Platforms policy brief press conference on Monday, UN Secretary-General António Guterres started his speech with:
    Distinguish members of our press corps. New technology is moving at warp speed, and so are the threats that come with it. Alarm bells over the latest form of Artificial Intelligence - generative AI - are deafening. And they are loudest from the developers who designed it. These scientists and experts have called on the World to act, declaring AI an existential threat to humanity on a par with the risk of nuclear war. We must take those warnings seriously. Our proposal, Global Digital Compact New Agenda for Peace and Accord on the Global Governance of AI, will offer multilateral solutions based on human rights.
    (Video here.)Guterres went on to discuss current damage from digital technology ("but the advent of generative AI must not distract us from the damaged digital technology is already doing to our world").
    The opening mention of existential threat from AI is a very welcome development in terms of the possibility of global coordination on the issue.
    It seems likely that the CAIS Statement on AI Risk - "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." - was instrumental in prompting this, given the mention of nuclear war.
    In terms of extinction risk, remember that the right to life is first and foremost!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • LW - Looking Back On Ads by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Looking Back On Ads, published by jefftk on June 15, 2023 on LessWrong.
    It is difficult to get a man to understand something, when his salary depends on his not understanding it.
    Talking about ads online I would often get a response that as someone working in ads I was bound to support my employment. I'm now a year out from working in ads and very unlikely to return to the industry: with the economic bias removed but still knowing the industry reasonably well, what do I think of it all now?
    My overall views haven't changed very much. I still think the only serious options are paywalls ("pay with your money") and ads ("pay with your attention"), and in the case of the web the value of moving fluidly from site to site pushes strongly for ads. And for sites that are supported by ads it's worth working to make them, to the extent possible, not-bad for visitors and lucrative for the site.
    Sometimes technical work can improve ads on the web on both axes simultaneously. The introduction of intersection observer and ping both took something ads really wanted (logging which ads were within the user's viewport or that an ad had been clicked on) and was already possible (with the scroll event or a redirect) and made the implementation much more efficient. These were already so valuable for ads that ~all ads did them, so increasing efficiency didn't increase how often the tracking happened, it just made the web a bit more responsive. I interpret Chrome's Privacy Sandbox work as another attempt in this direction, trying to retain the economic effects of adtech being able to track users from site to site (predicting which ads users will best respond to, detecting abuse, etc) while dramatically improving the privacy situation.
    In other cases, though, there's really just a fight over whether ads should be more or less intrusive, numerous, etc. Users want to see the stuff they came to the site for, publishers want to make more money. This is held somewhat in check by how if a publisher makes their site too unpleasant users will leave (or install adblockers) but only a bit: if I have a few tasteful text-only static ads for relevant things that only goes so far if other sites have flashing hovering noisy moving ads. There's a commons problem where each annoying site makes things worse for all the sites.
    Ideally users could credibly say "if you make your site too unpleasant we're not going to visit", as some sort of collective, and hold publishers to a tight standard. Google actually does a bit of this, through Ad
    Experience Reviews. The idea is there's the Better Ads Standard for what counts as a decent experience, and if I make a site that's more annoying than allowed Chrome will block my ads when you visit my site. Another attempt here is AdBlock Plus' Acceptable Ads Program, where they'll allow ads through if they meet the criteria and you pay their fees, though I'm not keen on the non-transparent way the fees are determined.
    One place, close to my former work, where my views have changed, though, is that I think it would be better if
    Google's display ads business (putting ads on non-Google websites) weren't part of Google. In its role as a search engine and browser manufacturer Google is in a strong position to advocate for what's best for their users, including what leads to a thriving web ecosystem. But because they also operate an adtech business, any serious efforts in this direction (something stronger than the Better
    Ads Standard or a simpler and faster version of Privacy Sandbox that's better for users but makes less money for sites) raises massive competition questions. Are they really doing it because they care about users, or is it a ploy to advantage their own adtech? When I worked in this area my experience was that me and my coworkers were doing things for the right re...
    5 min
  • LW - I still think it's very unlikely we're observing alien aircraft by dynomight
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I still think it's very unlikely we're observing alien aircraft, published by dynomight on June 15, 2023 on LessWrong.
    Some suggest there might be alien aircraft on Earth now. The argument goes something like this:
    (1) A priori, there’s no reason there shouldn’t be alien aircraft. Earth is 4.54 billion years old, but the universe is 13.7 billion years old, and within a billion light years of Earth there are something like 5 × 10¹⁴ stars. Most of those stars have planets, and if an alien civilization arose anywhere and built a von Neumann probe, those probes would spread everywhere.
    (2) We have tons of observations that would be more likely if there were alien aircraft around than if there weren’t. These include:
    Vast numbers of anecdotal reports from pilots.
    Videos that appear to show objects with flight characteristics far beyond known human capabilities.
    Senators—with access to classified information—raising concerns about unknown craft near military installations.
    Rumors from multiple sources that the government has captured, intact alien craft.
    (3) So if we agree that:
    the prior probability P[aliens] is pretty high; and
    P[gimbal video | aliens] > P[gimbal | no aliens]; and
    P[Pheonix lights | aliens] > P[Pheonix | no aliens]; and
    P[Trans-en-Provence | aliens] > P[Trans-en-Provence | no aliens]; and
    P[2021 Pentagon UAP Task Force report | aliens] > P[report | no aliens]; and
    P[UAP Task Force member whistleblower alleges US government has intact non-human vehicles | aliens] > P[whisleblower | no aliens]; and
    P[Former intelligence official says four people confirm secret US government program to analyze off-world craft | aliens] > P[official | no aliens],
    then don’t we have to conclude that the posterior probability P[aliens | everything] is pretty high?
    No.
    What I like about this argument
    First, I agree the prior probability should be pretty high. Say you just told me:
    Okay, say there’s a universe with at least 10²⁴ stars, and most stars have planets, and on one planet somewhere life has evolved and started sending stuff to other planets and has an ever-accelerating pace of technological development.
    Then I would agree—the odds that there would be alien aircraft on my planet seem good. It seems like we—or our artificial descendants—might well send aircraft to other planets. So why shouldn’t alien aircraft be here, now?
    Second, it takes eyewitness reports seriously. There are a huge number of reports by pilots seeing objects accelerating at insane rates with no obvious wings, control surfaces, or signs of propulsion. It’s a mistake to dismiss these as a product of diseased minds or attention-seeking. There are just too many reports—both civilian and military—sometimes from pilots in different planes at the same time, sometimes invisible to radar and sometimes confirmed by radar from multiple sources. Usually, these people make little effort to draw attention to themselves—we only hear about their observations secondhand. They seem serious and well-intentioned.
    (I hesitate to mention this, but I even have family members that report seeing something very strange years ago. I respect their intelligence and I don’t think they have an agenda—they barely talk about it and don’t consider it very important. The story goes that I myself saw it, but I was very young and remember nothing.)
    Third, I agree that lots of these observations are very hard to explain. Now, of the 510 reports investigated for the 2022 UAP task force report, more than half had mundane explanations—usually balloons or drones. And some of the leaked videos could plausibly be explained in terms of stuff like rotating glare. But other incidents have apparently been measured by different sensors (e.g. vision, radar) or from multiple locations (e.g. a plane and a ship) simultaneously, and just don’t have a cl...
    10 min
  • EA - Why EA Community building by Rob Gledhill
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why EA Community building, published by Rob Gledhill on June 15, 2023 on The Effective Altruism Forum.
    I have heard people who are uncertain about whether EA community building is the right move for them, given the increased prominence of AI Safety. I think that EA community building is the right choice for a significant number of people, and wanted to lay out why I believe this.
    AI Safety Community building seems important
    I’m excited to see AI Safety specific community building and I hope it continues to grow. This piece is not intended to claim that no-one should be working on AIS community building. Although CEA’s groups team is at an EA organisation, not an AI-safety organisation. I hope we can collaborate with AI Safety groups, as:
    It would likely benefit both parties to synch on issues like data collection
    I think there are lessons learned from EA community building that would be relevant and valuable to share
    The reasons that I think the case for AI Safety community building is strong, are:
    If we want people to work in AI Safety, directly talking about AI Safety seems the most straightforward way to do this
    There are talented people who will find the AI Safety framing attractive, but would not like the EA framing
    Early AIS community building efforts have managed to attract significant numbers of talented individuals (although I don’t think it’s inevitable that these early wins will scale, or successfully avoid causing accidental harm)
    EA community building is also important
    I think EA community building is still very valuable, for five reasons
    EA groups have been successful
    In the 2020 EA and Longtermist survey, local groups were mentioned by 42% of respondents
    I care about EA values in decision makers during crunch time. E.g., I think people in the EA movement have thought unusually deeply about what catastrophes would and wouldn’t lead to the loss of humanity’s future potential
    Having a compelling answer to the question “how do I do the most good” or “how do I live a good life” has been something that has historically attracted a lot of talent, and talent that would not necessarily have been attracted by AI Safety (conversely I expect AI Safety groups to attract people who wouldn’t be drawn to discussions of “how do I do the most good”)
    Specific, talented, organisers can be a better fit for either AIS or EA, and I want both options to exist
    I think for both options to exist, both options need to have great organisers. If all of the best organisers went for a single option, I think the other option would either become irrelevant, or cease to exist.
    It seems important to note that it is still possible that the risks from AI don’t manifest in the way that EAs widely expect, in which case we’ll be glad that we have a network of people that care about EA ideas
    I want to see collaboration between EA and AIS community building
    Although this isn’t the reason I’d like to see AIS CB, there is an extra benefit: I think work on the most pressing problems could go better if EAs did not form a super-majority (but still a large enough faction that EA principles are strongly weighted in decision making)
    Since the FTX crisis there has been increasing discussion about trustingness amongst EAs. Although I think the FTX crisis could have happened in less trusting communities (e.g., Many VCs also lost money in FTX) - I think it is true that there are areas where high trust is harmful. I think operating in an environment where EAs aren’t a super-majority would improve certain processes that currently overly rely on trust. Additionally I think having EA form a part of your identity can cause in-group effects, where ideas from the outgroup aren’t taken seriously enough. I suspect this would be lessened if people identifying as EA didn’t form a majority
    Based on the above, EA commu...
    8 min
  • EA - Mindmap with overview of EA organisations via tinyurl.com/eamindmap (and many other lists of orgs) by mariekedev
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mindmap with overview of EA organisations via tinyurl.com/eamindmap (and many other lists of orgs), published by mariekedev on June 15, 2023 on The Effective Altruism Forum.
    TLDR: I've assembled a mindmap of all EA (-related) entities I could find. You can access it at tinyurl.com/eamindmap. If I missed or misrepresented anything: leave a note (in the mindmap) and I'll add it or correct it. I've also added other existing lists of orgs to this post.
    Context
    Starting my position at EA Netherlands as a co-director, I was advised by EA Pathfinder (now Successif) to make an overview of the EA space. In this way it would be easier to onboard, support community members and follow conversations.
    I made this mindmap of EA(-related) entities and this quickly got out of hand. I asked around, whether such overviews already exists. There were a few:
    Overview of EA organizations
    EA Org list (EA Opportunity Board) - From this list, I use this overview most often as it seems to be very comprehensive and includes meta data such as 'involvement with EA'.
    EA Org wiki (Jamie Gittins) - Introduced via a Forum post. I'd share this link if someone is new, or looking for an EA aligned career, as it has clear descriptions for every org
    EA/Rationalist-Related Orgs (Arden P. B. W.) - Also contains fellowships, courses and YouTube channels
    Database of orgs relevant to longtermist/x-risk work (Michael Aird) - Very comprehensive list
    Overview of effective giving organisations ->Overview of the public effective giving ecosystem (supply and demand) [shared] (Sjir Hoeijmakers) - Amazing to see all these giving orgs, I had no idea
    Let's advertise EA infrastructure projects, Feb 2023 - A list of EA-related projects and orgs that offer free and useful services
    EASE: directory of independent agencies and freelancers offering expertise to EA-aligned organizations
    AISafety.world is a map of the AIS ecosystem - Very hard to compete with this :)
    Other somewhat related lists:
    OpenBook: New EA Grants Database
    List of EA Slack workspaces (Alex Berezhnoi)
    List of EA-aligned research training programs (Michael Aird)
    Effective Altruism Data
    Historical EA funding data
    Let me know if I missed an important list of orgs here.
    Somehow I still ended up sharing my mindmap with others, as it gives a clear visual overview of the orgs per main category or cause area. I know it is not perfect, but it is easy to maintain and people seem to like it. I was asked multiple times to add it to the forum, so here it is.
    Disclaimer
    Organisation inclusion doesn't imply my endorsement.
    The branches (the categories or cause areas) are a bit arbitrary, as some organisations serve multiple categories or cause areas. For me the aim was to get an overview, not a perfect mindmap.
    It's incomplete. I'm still adding orgs weekly. Feel free to add notes (in the mindmap) where this overview can be corrected or orgs can be added or deleted.
    I use the words orgs, organisations and entities in this post. That is because 'orgs' is more accessible and clear as a word, but the entities aren't all official organisations.
    I changed the title of the mindmap from EA(-aligned) to EA(-related), due to this comment. I'm still a bit uncomfortable with it, because there are many more EA-related orgs which won't fit all into this mindmap (see 80k jobboard), but it seems more right (for example: I added the AI labs..). I try to add only the orgs that self-identify as EA-aligned or the ones that many EAs refer to.
    How to use this
    However you like :) the left side contains more meta entities and the right side contains more object-level (cause area related) orgs.
    I heard people use it in order to quickly get an overview of existing entities in one category (like EA career support or biorisk).
    Sometimes I use it in an intro presentation to make people more aware of ...
    5 min
  • LW - Why libertarians are advocating for regulation on AI by RobertM
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why libertarians are advocating for regulation on AI, published by RobertM on June 14, 2023 on LessWrong.
    Motivation: some people on the internet seem confused about why libertarians (and others generally suspicious of government intervention) are advocating for regulation to help prevent x-risk from AI. The arguments here aren't novel and generally seem pretty obvious to me, but I wanted a quick reference.
    Who is advocating for regulations?
    This is obviously the first question: is the premise correct? Are libertarians uncharacteristically advocating for regulations?
    They probably aren't the loudest or most numerous among the set of people who are advocating for regulations while being motivated by reducing AI x-risk, but I'm one such person and know multiple others.
    I tend to bucket those advocating for regulations into three buckets.
    Non-libertarians
    Probably the biggest group, though I have wide error bars here. Most of the people I know working on the policy & comms side are either liberal, apolitical, or not usefully described by a label like that. I think people in this group tend to have somewhat more optimistic views on how likely things are to go well, but this isn't a super strong correlation.
    Libertarians
    Although some of them are focusing primarily on policy & comms, those I know more often dual-class with a technical primary & comms secondary. Tend to have more pessimistic views on our likelihood of making it through.
    Eliezer Yudkowsky
    I could have put him in the "libertarian" group, but he has a distinct and specific policy "ask". Also often misquoted/misunderstood.
    The basic argument
    There are many arguments for why various regulations in this domain might seem compelling from a non-libertarian point of view, and people generally object to those on different grounds (i.e. accusations of regulatory capture, corruption, tribalism, etc.), so I'll skip trying to make their case for them.
    Why might libertarians advocate for (certain kinds of) regulation on AI, given their general distrust of government?
    Straightforwardly, the government is not a malicious genie optimizing for the inverse of your utility function, should you happen to ask it for a favor. Government interventions tend to come in familiar shapes, with a relatively well-understood distribution of likely first- and second-order effects.
    If you're mostly a deontological libertarian, you probably oppose government interventions because they violate people's sovereignty; nevermind the other consequences.
    If you're mostly a consequentialist libertarian, you probably oppose government interventions because you observe that they tend to have undesirable second-order effects, often causing more harm than any good from the first-order effects.
    But this is a contingent fact about the intersection between the effects of regulations, and the values of libertarians. Many regulations often have the effect of slowing down technological progress and economic growth, by making it more expensive to operate a business, do R&D, etc. Libertarians usually aren't fans, since, you know, technological progress and economic growth are good things.
    Unless you expect a specific kind of technological progress to kill everyone, possibly in the next decade or few. A libertarian who believes that the default course of AI development will end with us creating an unaligned ASI that kills us and eats the accessible lightcone is not going to object to government regulations on AI because "regulations bad".
    Now, a libertarian who believes this, and is thinking sensibly about the subject, will have specific models about which regulations seem like they might help reduce the chance of that outcome, and which might hurt. This libertarian is not going to make basic mistakes like thinking that the intent of the regulation will be stron...
    8 min
  • LW - On the Apple Vision Pro by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On the Apple Vision Pro, published by Zvi on June 14, 2023 on LessWrong.
    Apple is offering a VR/AR/XR headset, Vision Pro, for the low, low price of $3,500.
    I kid. Also I am deadly serious.
    The value of this headset to a middle class American or someone richer than that is almost certainly either vastly more than $3,500, or at best very close to $0.
    This type of technology is a threshold effect. Once it gets good enough, if it gets good enough, it will feel essential to our lives and our productivity. Until then, it’s a trifle.
    Thus, like Divia Eden, I am bullish on using the Tesla strategy of offering a premium product at a premium price, then later either people decide they need it and pay up or you scale enough to lower costs – if the tech delivers.
    Gaming could be a modest benefit. Mark Zuckerberg points out on the Lex Fridman podcast that with no native controller this could be a poor VR/AR gaming platform. Mark suggested this could drive demand for the more reasonably priced Occulus.
    This doesn’t apply to traditional gaming with the VR used to improve the screen and mobility, assuming you can get connected to devices that allow real gaming (e.g. console devices or PCs, not iPhones and Macs.) Apple lets you hook up a PS5 controller or mouse and keyboard if you like, but only directly integrates with Apple devices.
    My current impression of existing VR/AR is that it is in the ‘not worth much’ section of the curve. The games and activities are fun to try out, but not worth sustained engagement. Productivity wasn’t there at all.
    Can Apple do better? Are we there? There’s definitely a bunch of new tech here.
    Your New Computer Interface?
    That’s how their presentation pitch starts. ‘All your favorite apps,’ controlled by your eyes and finger scrolls and taps.
    Do we want to use a phone or computer in AR or VR? It makes sense to go use case by use case.
    Watching movies, shows and other media? Sounds good if quality is there. Automatically darkening the rules and resizing the screen seem like awesome features. Looks like experience quality is there. The question is, what does this offer that’s ten times better than a television screen, other than mobility? Otherwise, cost plus the trivial inconveniences involved will be big problems.
    Looking at your old videos and photos, and also taking photos and videos? Experience quality here seems fantastic. Banking memories and vistas for later viewing seems killer, and the amount you’re taken ‘out of the moment’ seems vastly lower, much better than ‘hey let me take some photos.’ The obvious catch is, how much does wearing the headset take you out of the moment inherently?
    Playing traditional games? I mean, hell yeah, let’s go, if they’ll let us. Battery life on the go is questionable but you can plug in or presumably buy and carry multiple batteries. Which games can we hook up? For now you will have to go through a Mac or iPhone. The question is how big a barrier that is in practice.
    Do video calls from anywhere and have no one be able to tell, thanks to your realistic digital avatar? Shut up and take my money. TechCrunch confirms that this works. Ars Technica and others warn that the digital avatars are currently a bit uncanny valley, and standard Zoom may still end up getting used anyway until things get sufficiently better. I expect the tech here to advance rapidly. This comes down to the social question. Will others be down for you using it?
    Would I want to read this way, either books or the web? Hard to know without a demo, both in terms of screen and interface. Is this friendly to your eyes? Is the scrolling experience better than a phone or mouse, is it precise enough? What about using those fancy eye monitors, seeing where you were reading and scroll along? How else can we enhance the experience?
    The mindfulness application...
    17 min
  • LW - Anthropic | Charting a Path to AI Accountability by Gabriel Mukobi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic | Charting a Path to AI Accountability, published by Gabriel Mukobi on June 14, 2023 on LessWrong.
    Below is the text copied from the linked post (since it's short). I'm not affiliated with Anthropic but wanted to link-post this here for discussion.
    This week, Anthropic submitted a response to the National Telecommunications and Information Administration’s (NTIA) Request for Comment on AI Accountability. Today, we want to share our recommendations as they capture some of Anthropic’s core AI policy proposals.
    There is currently no robust and comprehensive process for evaluating today’s advanced artificial intelligence (AI) systems, let alone the more capable systems of the future. Our submission presents our perspective on the processes and infrastructure needed to ensure AI accountability. Our recommendations consider the NTIA’s potential role as a coordinating body that sets standards in collaboration with other government agencies like the National Institute of Standards and Technology (NIST).
    In our recommendations, we focus on accountability mechanisms suitable for highly capable and general-purpose AI models. Specifically, we recommend:
    Fund research to build better evaluations
    Increase funding for AI model evaluation research. Developing rigorous, standardized evaluations is difficult and time-consuming work that requires significant resources. Increased funding, especially from government agencies, could help drive progress in this critical area.
    Require companies in the near-term to disclose evaluation methods and results. Companies deploying AI systems should be mandated to satisfy some disclosure requirements with regard to their evaluations, though these requirements need not be made public if doing so would compromise intellectual property (IP) or confidential information. This transparency could help researchers and policymakers better understand where existing evaluations may be lacking.
    Develop in the long term a set of industry evaluation standards and best practices. Government agencies like NIST could work to establish standards and benchmarks for evaluating AI models’ capabilities, limitations, and risks that companies would comply with.
    Create risk-responsive assessments based on model capabilities
    Develop standard capabilities evaluations for AI systems. Governments should fund and participate in the development of rigorous capability and safety evaluations targeted at critical risks from advanced AI, such as deception and autonomy. These evaluations can provide an evidence-based foundation for proportionate, risk-responsive regulation.
    Develop a risk threshold through more research and funding into safety evaluations. Once a risk threshold has been established, we can mandate evaluations for all models against this threshold.
    If a model falls below this risk threshold, existing safety standards are likely sufficient. Verify compliance and deploy.
    If a model exceeds the risk threshold and safety assessments and mitigations are insufficient, halt deployment, significantly strengthen oversight, and notify regulators. Determine appropriate safeguards before allowing deployment.
    Establish pre-registration for large AI training runs
    Establish a process for AI developers to report large training runs ensuring that regulators are aware of potential risks. This involves determining the appropriate recipient, required information, and appropriate cybersecurity, confidentiality, IP, and privacy safeguards.
    Establish a confidential registry for AI developers conducting large training runs to pre-register model details with their home country’s national government (e.g., model specifications, model type, compute infrastructure, intended training completion date, and safety plans) before training commences. Aggregated registry data should be protect...
    7 min
  • EA - Some talent needs in AI governance by Sam Clarke
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some talent needs in AI governance, published by Sam Clarke on June 14, 2023 on The Effective Altruism Forum.
    I carried out a short project to better understand talent needs in AI governance. This post reports on my findings.
    How this post could be helpful:
    If you’re trying to upskill in AI governance, this post could help you to understand the kinds of work and skills that are in demand.
    If you’re a field-builder trying to find or upskill people to work in AI governance, this post could help you to understand what talent search/development efforts are especially valuable.
    Key takeaways
    I talked with a small number of people hiring in AI governance—in research organisations, policy think tanks and AI labs—about the kinds of people they’re looking for. Those hiring needs can be summarised as follows:
    All the organisations/teams I talked to are interested in hiring people to do policy development work—i.e. developing concrete proposals about what key actors (e.g. governments, AI labs) should do to make AI go well.
    There’s currently high demand for this kind of work, because windows of opportunity to implement useful policies have begun arising more frequently.
    There’s also a limited supply of people who can do it, partly because it requires the ability to do both (a) high-level strategising about the net value of different policies and (b) tactical implementation analysis about what, concretely should be done by people at the government/AI lab/etc. to implement the policy. This is an unusual combination of skills, but one which is highly valuable to develop.
    AI governance research organisations (specifically, GovAI and Rethink Priorities) are also interested in hiring people to do other kinds of AI governance research—e.g. carrying out research projects in compute governance or corporate governance, or writing touchstone pieces explaining important ideas.
    AI governance teams at policy think tanks and AI labs are interested in hiring people whose work would substantially involve engaging with people to do stakeholder management, consensus building and other activities to help with the implementation of policy actions.
    Also, there is a lot of work requiring technical expertise (e.g. hardware engineering, information security, machine learning) that would be valuable for AI governance. Especially undersupplied are technical researchers who can answer questions that are not yet well-scoped (i.e. where the questions require additional clarifying before they are crisp and well-specified). Doing this well requires an aptitude for high-level strategic thinking, along with technical expertise.
    Method
    I conducted semi-structured interviews with a small number of people hiring in AI governance—in research organisations, policy think tanks and AI labs—about the kinds of people they’re looking for.
    I also talked with two people about talent needs in technical work for AI governance.
    Findings
    Talent needs
    I report on the kinds of work that people I interviewed are looking to hire for, and outline some useful skills for doing this work.
    Note: when I say things like “organisation X is interested in hiring people to do such-and-such,” this doesn’t imply that they are definitely soon going to be hiring for exactly these roles. It should instead be read as a claim about the broad kind of talent they are likely to be looking for when they next open a hiring round.
    AI governance research organisations
    Currently, GovAI is especially interested in research agendas that contribute to policy development work—i.e. developing concrete proposals about what key actors (e.g. governments, AI labs) should do to make AI go well. There’s high demand for this kind of work and very few people who can do it.
    Researchers who can write touchstone pieces explaining, clarifying, and justifying important ideas...
    15 min