The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - What is epigenetics? by Metacelsus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is epigenetics?, published by Metacelsus on November 6, 2022 on LessWrong.
    Among all areas of biology related to my research, epigenetics is the one that is most commonly misunderstood, not only by the general public but even by other scientists. After being irritated one too many times, I’ve decided to make a series of posts to explain what epigenetics really is, why it’s important, and how it’s misunderstood. I will also explain how epigenetics is important for my own research on making gametes from stem cells.
    This first post covers the definition of epigenetics, and the basic biology of epigenetic marks.
    What is genetics?
    Before defining epigenetics, let’s start with a definition of genetics. Genetics is the study of genes, which are sequences of genetic material that encode functional products.
    Let’s take the IGF2 gene as an example.
    Depicted above is a region of human chromosome 11 containing the IGF2 gene, which encodes the IGF2 protein, an important growth factor for fetal development. The boxes represent exons and lines represent introns. The darker green color is the protein-coding sequence, and non-coding (i.e. untranslated) regions are shown in lighter green. Arrows represent the direction of transcription.
    The bottom of this image shows the location of common genetic variants (present at >1% frequency). If you look closely, you might notice that none of them are in the protein-coding sequence (the dark green boxes). This is not a coincidence, because nothing is ever a coincidence most mutations to essential proteins (including IGF2) are harmful and thus selected out of the population. However, there are several common mutations in non-coding regions of this gene.
    To recap, genetics is the study of genes (such as IGF2) and the effects of genetic variation on their functions.
    What is epigenetics?
    Epigenetics is the study of epigenetic marks, which are changes to genetic material that alter gene expression, but do not change the genetic sequence. A decent analogy for epigenetic marks is CAPITALIZATION, bolding, or strikethroughs in text.
    DNA methylation and histone modifications are the two kinds of epigenetic marks. Some people also consider long noncoding RNAs (such as those involved in X-chromosome inactivation) to be epigenetic marks. Although these RNAs are undoubtedly important for regulating gene expression, I would not classify them as epigenetic marks since they are not direct modifications to genetic material.
    In vertebrate animals, the cytosine in CG sequences often has a methyl group attached, forming 5-methylcytosine. A CG sequence is also CG on the opposite strand, so the cytosines on both strands can be methylated.
    To make things confusing, methylation at CG sequences is termed CpG methylation, the lowercase p standing for phosphate. 5-methylcytosine will pair with guanine just like normal cytosine, but it is not equivalent to cytosine in its interactions with DNA-binding proteins. Generally, CpG methylation suppresses the expression of nearby genes. CpG sites often cluster together to form “CpG islands” in important regulatory regions. Other organisms (invertebrates, plants, fungi, bacteria) have different ways of methylating DNA. I won’t get into them in this post series, but you should know that CpG methylation is not universal.
    Modifications to histones are another important set of epigenetic marks. Histones are DNA packaging proteins, which form complexes called nucleosomes. DNA winds around nucleosomes sort of like thread around spools. The overall assembly of DNA and histones is known as chromatin.
    Chemical modifications to histones are important epigenetic marks that can have drastic changes on gene expression. For example, trimethylation of lysine 4 on histone H3 (known as H3K4me3) marks promoters of actively transcribed genes. ...
    13 min
  • EA - Instead of technical research, more people should focus on buying time by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instead of technical research, more people should focus on buying time, published by Akash on November 5, 2022 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Takeaways from a survey on AI alignment resources by DanielFilan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Takeaways from a survey on AI alignment resources, published by DanielFilan on November 5, 2022 on LessWrong.
    What am I talking about?
    In June and July of this year, I ran a survey to ask a lot of people how useful they found a variety of resources on AI alignment. I was particularly interested in “secondary resources”: that is, not primary resource outputs, but resources that summarize, discuss, analyze, or propose concrete research efforts. I had many people promote the survey in an attempt to make it not obvious that I was running it (so that it would not affect what people said about AXRP, the podcast that I run). CEA helped a great deal with the shaping and promotion of the survey.
    The goal of the survey was initially to figure out how useful AXRP was, but I decided that it would be useful to get a broader look at the space of these secondary resources. My hope is that the results give people a better sense of what secondary resources might be worth checking out, as well as gaps that could be filled.
    Participants were shown a list of resources, select those they’d engaged with for >30 min, and for each they selected, rate on a scale from 0 to 4 how useful they’d found it, how likely they’d be to recommend to a friend getting into the field who hadn’t read widely, and how likely they’d be to recommend to someone paid to do AI alignment research. You can do a test run of the survey at this link.
    My summary of the results
    AXRP, my podcast, is highly rated among people paid to work on technical AI alignment resources, but less highly rated in other cohorts.
    On a personal note, I find this a bit disappointing: I had hoped it could be useful for people orienting to research directions that they had not read widely about.
    Rob Miles videos are highly rated among everyone, more than I would have guessed.
    People really liked the AI Safety Camp, the AGI Safety Fundamentals Course, and conversations with AI alignment researchers.
    People trying to get into alignment really liked the above and also MLAB. That said, they recommend Rob Miles videos higher than the AI Safety Camp and conversations with AI alignment researchers (but lower than MLAB and the AGI Safety Fundamentals Course).
    Basic stats
    Entries with demographic info: 139
    Entries that rate various resources: 99
    Number that say ‘I have heard of AI alignment’: 95
    Number that say ‘I am interested in AI alignment research’: 109
    Number that say ‘I am trying to move into a technical AI alignment career’: 68
    Number that say ‘I spend some of my time solving technical problems related to AI alignment’: 51
    Number that say ‘I spend some of my time doing AI alignment field/community-building’: 37
    Number that say ‘I spend some of my time facilitating technical AI alignment research in ways other than doing it directly’: 35
    Number that say ‘I spend some of my time publicly communicating about AI alignment’: 36
    Number that say ‘I am paid to work on technical AI alignment research’: 30
    Number that say ‘I help run an organization with an AI alignment mission (e.g. CHAI, MIRI, Anthropic)’: 11
    Context for questions
    When sorting things by ratings, I’ve included the top 5, and anything just below the top 5 if that was a small number. I also included ratings for AXRP, the podcast I make. Ratings are paired with the standard error of the mean (total ratings have this standard error multiplied by the number of people in the sample). Only things that at least 2 people engaged with were included.
    Ratings were generally rounded to two significant figures, and standard errors were reported to the same precision.
    Usefulness ratings
    Among all respondents:
    Total usefulness (multiplying average rating by reach):
    80k podcast: 167 +/- 8
    Superintelligence: 166 +/- 8
    Talks by AI alignment researchers: 134 +/- 6
    Rob Miles videos: 131 +/- 7
    AI alignment ...
    12 min
  • EA - The Slippery Slope from DALLE-2 to Deepfake Anarchy by stecas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Slippery Slope from DALLE-2 to Deepfake Anarchy, published by stecas on November 5, 2022 on The Effective Altruism Forum.
    OpenAI developed DALLE-2. Then StabilityAI made an open source copycat. This is a dangerous dynamic.
    Stephen Casper ([email protected])
    Phillip Christoffersen ([email protected])
    Rui-Jie Yew ([email protected])
    Thanks to Tan Zhi-Xuan and Dylan Hadfield-Menell for feedback.
    This post talks about NSFW content but does not contain any. All links from this post are SFW.
    Abstract
    Since OpenAI published their work on DALLE-2 (an AI system that produces images from text prompts) in April, several copycat text-to-image models have been developed including StabilityAI’s Stable Diffusion. Stable Diffusion is open-source and can be easily misused, including for the almost-effortless development of NSFW images of specific people for blackmail or harassment. We argue that OpenAI and StabilityAI’s efforts to avoid misuse have foreseeably failed and that both share responsibility for harms from these models. And even if one is not concerned about issues specific to text-to-image models, this case study raises concerns about how copycatting and open-sourcing could lead to abuses of more dangerous systems in the future.
    To reduce risks, we discuss three design principles that developers should abide by when designing advanced AI systems. Finally we conclude that (1) the AI research community should curtail work on risky capabilities–or at the very least more substantially vet released models (2) the AI governance community should work to quickly adapt to heightened harms posed by copycatting in general and text-to-image models in particular, and (3) public opinion should ideally not only be critical of perpetrators for harms that they cause with AI systems, but also originators, copycatters, distributors, etc. who enable them.
    What’s wrong?
    Recent developments in AI image generation have made text-to-image models very effective at producing highly realistic images from captions. For some examples, see the paper from OpenAI on their DALLE-2 model or the release from Stability AI of their Stable Diffusion model. Deep neural image generators like StyleGan and manual image editing tools like Photoshop have been on the scene for years. But today, DALLE-2 and Stable Diffusion (which is open source) are uniquely effective at rapidly producing highly-realistic images from open-ended prompts.
    There are a number of risks posed by these models, and OpenAI acknowledges this. Unlike conventional art and Photoshop, today’s text-to-image models can produce images from open-ended prompts by a user in seconds. Concerns include (1) copyright and intellectual property issues (2) sensitive data being collected and learned (3) demographic biases, e.g. producing images of women when given the input, “an image of a nurse” (4) using these models for disinformation by creating images of fake events, and (5) using these models for producing non-consensual, intimate deepfakes.
    These are all important, but producing intimate deepfakes is where abuse of these models seems to be the most striking and possibly where we are least equipped to effectively regulate misuse. Stable Diffusion is already being used to produce realistic pornography. Reddit recently banned several subreddits dedicated to AI-generated porn including r/stablediffusionnsfw, r/unstablediffusion, and r/porndiffusion for a violation of Reddit’s rules against non-consensual intimate media.
    This is not to say that violations of sexual and intimate privacy are new. Before the introduction of models such as DALLE-2 and Stable Diffusion, individuals have been victims of non-consensual deepfakes. Perpetrators often make this content to discredit or humiliate people from marginalized groups, taking advantage of the negative sociocultural ...
    30 min
  • LW - Spectrum of Independence by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spectrum of Independence, published by jefftk on November 5, 2022 on LessWrong.
    When I compare our parenting decisions to peers and neighbors' we tend to be near the far end of thinking our kids are ready to do things
    on their own. I don't see other first graders (6y) unaccompanied at the park or
    crossing the street, and it was only this year that our third grader's (8y) friends started ringing the doorbell to ask if she could go out and play, while she's been going over and ringing theirs for a couple years. Last year when we gave permission for our second grader (7y) and kindergartner (5y) to walk home together the school said yes, but somewhat reluctantly and they didn't remember this situation with a kindergartner before. [1]
    It's reasonably common for people to point at the past for examples of this level of independence being normal. Reading older children's books the kids wander everywhere, my dad walked to school and back alone in kindergarten, and articles comparing child walksheds by generation are evergreen viral content. The phenomenon of declining children's independence over time is well documented. But what I hadn't internalized until seeing it recently was that even in the US there is still much more variation than I see in my daily life.
    A few weeks ago I brought the kids along to a dance I was playing, and we stayed with a couple who lived adjacent to a farm. They're on good terms with the farmers, and one afternoon they brought us over to look around. While I knew it was common for farm kids to have more independence and responsibility, it was different seeing it in person.
    The farmers have several kids, and we ran into their 9yo moving soybeans for storage. They drove past us in a large tractor pulling a short train of wagons, and lined it up carefully with a hopper. They had another tractor set up with an unshrouded power take-off for an auger to get the beans up into the bin. After transferring the beans in the first trailer, they moved the train up to unload the second. All of this was completely on their own, and they moved like someone who had done this many times.
    I'm not saying I'd let my kids operate serious machinery unsupervised at the same age—this is dangerous work even for adults—but it illustrates how wide a range there is in what different people consider appropriate and normal levels of independence and makes me feel less like I have extreme views.
    [1] The school is about 120 years old so they definitely have had many kindergartners walk home before, but apparently not within institutional memory.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - The Slippery Slope from DALLE-2 to Deepfake Anarchy by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Slippery Slope from DALLE-2 to Deepfake Anarchy, published by Stephen Casper on November 5, 2022 on The AI Alignment Forum.
    OpenAI developed DALLE-2. Then StabilityAI made an open source copycat. This is a concerning case study for AI alignment.
    Stephen Casper ([email protected])
    Phillip Christoffersen ([email protected])
    Rui-Jie Yew ([email protected])
    Thanks to Tan Zhi-Xuan and Dylan Hadfield-Menell for feedback.
    A different version of post has been posted in the EA forum. It is somewhat longer, focuses more on policy, and is written for a slightly more general audience.
    This post talks about NSFW content but does not contain any. All links from this post are SFW.
    Abstract
    Since OpenAI published their work on DALLE-2 (an AI system that produces images from text prompts) in April, several copycat text-to-image models have been developed including StabilityAI’s Stable Diffusion. Stable Diffusion is open-source and can be easily misused, including for the almost-effortless development of NSFW images of specific people for blackmail or harassment. We argue that OpenAI and StabilityAI’s efforts to avoid misuse have foreseeably failed and that both share responsibility for harms from these models. And even if one is not concerned about issues specific to text-to-image models, this case study raises concerns about how copycatting and open-sourcing could lead to abuses of more dangerous systems in the future. We discuss design principles that developers should abide by when designing advanced AI systems to reduce risks.
    We conclude that this case study highlights issues with working on risky capabilities and repudiates attempts to achieve AI alignment via racing to advance capabilities.
    What’s wrong?
    Recent developments in AI image generation have made text-to-image models very effective at producing highly realistic images from captions. For some examples, see the paper from OpenAI on their DALLE-2 model or the release from Stability AI of their Stable Diffusion model. Deep neural image generators like StyleGan and manual image editing tools like Photoshop have been on the scene for years. But today, DALLE-2 and Stable Diffusion (which is open source) are uniquely effective at rapidly producing highly-realistic images from open-ended prompts.
    There are a number of risks posed by these models, and OpenAI acknowledges this. Unlike conventional art and Photoshop, today’s text-to-image models can produce images from open-ended prompts by a user in seconds. Concerns include (1) copyright and intellectual property issues (2) sensitive data being collected and learned (3) demographic biases, e.g. producing images of women when given the input, “an image of a nurse” (4) using these models for disinformation by creating images of fake events, and (5) using these models for producing non-consensual, intimate deepfakes.
    These are all important, but producing intimate deepfakes is where abuse of these models seems to be the most striking and possibly where we are least equipped to effectively regulate misuse. Stable Diffusion is already being used to produce realistic pornography. Reddit recently banned several subreddits dedicated to AI-generated porn including r/stablediffusionnsfw, r/unstablediffusion, and r/porndiffusion for a violation of Reddit’s rules against non-consensual intimate media.
    This is not to say that violations of sexual and intimate privacy are new. Before the introduction of models such as DALLE-2 and Stable Diffusion, individuals have been victims of non-consensual deepfakes. Perpetrators often make this content to discredit or humiliate people from marginalized groups, taking advantage of the negative sociocultural attitudes that already surround them. An estimated 96% of deepfake videos online are porn, almost all featuring women. In one case, when a video of a jo...
    20 min
  • LW - Weekly Roundup #4 by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Weekly Roundup #4, published by Zvi on November 4, 2022 on LessWrong.
    Relatively light roundup this week, likely due to Elon Musk buying Twitter plus the incoming midterms sucking all the oxygen from everything else. I do have a Twitter/Musk post in progress tentatively scheduled for Monday.
    Prediction Markets Would Probably Be Great
    Journalist Ben Collins says ‘very good example of how illogical betting markets are’ and gives classic example of prediction markets being completely logical and doing good probabilistic reasoning.
    Metaculus timelines for AI winning a gold medal in the International Math Olympiad keep compressing, down this year from 2036 to 2026. This is certainly super scary and exciting on the AI front. It is also disappointing as a former aspirational math competitor, as it reveals that the IMO problems are far more formulaic and hackable than we realized. When AI makes super rapid progress in some areas (e.g. math competitions) while stalling in others (e.g. self-driving cars) relative to expectations, a lot of that is learning about the underlying task space.
    People Are Trying To Destroy the Internet
    Joe Manchin taking his turn? Proposing to force filing of ‘suspicious activity’ reports for essentially everything. From a few weeks ago, forgot to include before.
    Sports Go Sports
    It is real, and it is spectacular. Worth watching.
    Pat McAfee @PatMcAfeeShow
    Is this real?
    9:11 PM ∙ Oct 30, 2022
    112,354Likes8,984Retweets
    He learned this from playing NASCAR on Game Cube growing up. It works in real life.
    Competitors mostly thought it was super cool, which is correct.
    The question is, it was super cool that time. What happens when it happens on a regular basis? Looks like there is currently no appetite for a rules change until the off season. I think this is a mistake. We honor such crazy one-time moves by saying ‘yes, that was super cool and super legal, with great execution. Kudos. Now let us make sure that never happens again.’ If it happens a few more times before the season is over, that lessons the moment, and also seems dangerous and like it would rapidly get kind of dumb.
    Bad News
    On China stocks last week, I noticed my confusion that stocks were down after the party congress. This shows how long it has been since I have been trading. Back in the day, I would not have missed what a reader reminded me about, which is that stocks are not permitted to fall during a party congress, so one should expect them to fall after a party congress. Also, others said that while it was expected the congress would be bad for stocks, it did go worse than one would have expected, which I did not realize.
    The New York Times remains loosely banned here. It is no longer a strict ban, where I will not use a source a link to things like a Twitter post that leads to a NYT article. However, to the extent feasible, they are not welcome here, and this week provided a reminder of why during discussion of crazy reactions to Elon’s Twitter moves (which mostly I have put in their own post.)
    Kelsey tries her best to be nice about it.
    I would be less nice. They set out to use take position as America’s most trusted newspaper and use it to go after America’s technology industry while pretending to be doing objective reporting. They did their best to sabotage the people looking to give us nice things out of some combination of political motives, an attempt to protect their business model, a hatred for technology, and spite.
    The Intercept on DHS’s attempts to ‘fight misinformation.’ Report of which TechDirt reports is absolute garbage. Kids, let’s not fight, both the DHS and the Intercept can be up to no good at the same time.
    Uber Eats fees have always been ridiculous when added up, here in New York I find it absurd that anyone would use them over some combination of Caviar (for my p...
    10 min
  • EA - The Letten Prize - An opportunity for young EA researchers? by Håkon Sandbakken
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Letten Prize - An opportunity for young EA researchers?, published by Håkon Sandbakken on November 4, 2022 on The Effective Altruism Forum.
    Hi,
    Are you a young researcher who conducts research aimed at addressing global challenges within the fields of health, development, environment and equality in all aspects of human life?
    Or do you know of someone who might fit this description?
    If so, you/they can apply for The Letten Prize.
    Prize money at 2,5 MNOK (~235 000 USD).
    Interested? View the criteria for applying here:/
    Disclamers: I've been involved with the EA community since 2017, eg. established EA Oslo and served on the board of EA Norway. I am currently working freelance for The Young Academy of Norway, as well as serving on the board of the Letten Foundation. This research prize are a result of a collaboration by the two parties. I see why this may be seen as "spam". However, I truly believe that we have several great candidates within our community who could contend for the prize. I will serve as the secretary of the prize committee, but have no influence on the committees decision.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - EA orgs should accept Summer 2023 interns in January by Abby Hoskin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA orgs should accept Summer 2023 interns in January, published by Abby Hoskin on November 5, 2022 on The Effective Altruism Forum.
    TLDR; EA organizations hiring graduating seniors or hosting undergraduates as 2023 summer interns should send program acceptances in January 2023 so that students don't have to accept a corporate offer before they apply to EA programs.
    The Problem
    Multiple undergrads have complained that you can't apply to corporate internships and EA summer programs at the same time. Instead, non-EA orgs have very early deadlines (October to January), allowing them to get decisions back to students very early in the year. They typically want responses from students within a few weeks of sending out offers. Meanwhile, many EA orgs don't send out acceptances until much later, like April, well past the deadline for responding to the corporate internship offers.This puts students in an awkward position: either accept the corporate offers while planning to renege if they get a better EA internship, or gamble on the EA jobs, and potentially not get any. It's bad to miss out on anything for the summer, but lying violates most people's moral principles and going back on an offer that you previously accepted burns bridges at the organization you lied to.
    As a result of this application deadline system, many talented students are being funneled away from EA summer opportunities.
    The Solution
    Ideally, students could apply for corporate programs and EA fellowships at the same time, see where they are accepted, and take the best offer without having to lie or burn bridges.
    Moving EA program deadlines up to compete with industry might be challenging for orgs that aren't 100% sure what kind of funding they'll have for summer programs. In this case, consider what kind of guidelines you can provide to prospective candidates to give them a better sense of how likely they will be accepted for a summer program at your org. Also, consider doing early acceptances if an extremely talented student contacts you with an exploding offer from another organization, and publicizing this policy if you decide to enact it.
    The Competition
    Here are some summer 2023 internship application deadlines for corporations competing with us for top talent (ie this is how early applications need to open in order to get offers back to students on a competitive timeframe):
    Technology
    Google: October 31st, 2022Palantir: Sometime in October 2022
    Consulting
    Bain: November 29, 2022McKinsey: January 19th, 2023
    Finance
    Goldman Sachs: November 20th, 2022JP Morgan: November 27th, 2022
    NB: Many of these organizations have multiple summer programs which all have different deadlines. I've tried to select programs with representative deadlines, but bizarrely, some companies even have different deadlines for students from different universities! If you actually want to apply to one of these places, please go to the org's website to double check the deadline for the program that best suits you.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Applications are now open for Intro to ML Safety Spring 2023 by Joshc
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applications are now open for Intro to ML Safety Spring 2023, published by Joshc on November 4, 2022 on The Effective Altruism Forum.
    The Center for AI Safety is running another iteration of Intro to ML Safety this Spring for people who want to learn about empirical AI safety research topics.
    Apply to be a participant by January 29th, 2023.
    Apply to be a facilitator by December 30th.
    Website: mlsafety.org/intro-to-ml-safety
    About the Course
    Introduction to ML Safety is an 8-week course that aims to introduce students with a deep learning background to empirical AI Safety research. The program is designed and taught by Dan Hendrycks, a UC Berkeley ML PhD and director of the Center for AI Safety, and provides an introduction to robustness, alignment, monitoring, systemic safety, and conceptual foundations for existential risk.
    Each week, participants will be assigned readings, lecture videos, and required homework assignments. The materials are publicly available at course.mlsafety.org.
    There are two tracks:
    The introductory track: for people who are new to AI Safety. This track aims to familiarize students with the AI X-risk discussion alongside empirical research directions.
    The advanced track: for people who already have a conceptual understanding of AI X-risk and want to learn more about existing empirical safety research so they can start contributing.
    The course will be virtual by default, though in-person sections may be offered at some universities.
    How is this program different from AGISF?
    Intro to ML Safety is generally more focused on empirical topics rather than conceptual work. Participants are required to watch recorded lectures and complete homework assignments that test their understanding of the technical material. If you’ve already taken AGISF and are interested in empirical research, then you are the target audience for the advanced track.
    Intro to ML Safety also emphasizes different ideas and research directions than AGISF does. Examples include:
    Detecting trojans: this is a current security issue but also a potential microcosm for detecting deception and testing monitoring tools.
    Adversarial robustness: it is helpful for reward models to be adversary robust. Otherwise, the models they are used to train can ‘overoptimize’ them and exploit their deficiencies instead of performing as intended. This applies whenever an AI system is used to evaluate another AI system. For example, an ELK reporter must also be highly robust if its output is used as a training signal.
    Power averseness: Arguments for taking AI seriously as an existential risk often focus on power-seeking behavior. Can we train language models to avoid power-seeking actions in text-based games?
    You can read about more examples in Open Problems in AI X-risk.
    Time Commitment
    The program will last 8 weeks, beginning on February 20th and ending on April 14th. Participants are expected to commit at least 5 hours per week. This includes ~1 hour of recorded lectures (which will take more than one hour to digest), ~1-2 hours of readings, ~1-2 hours of written assignments, and 1 hour of discussion.
    We understand that 5 hours is a large time commitment, so to make our program more inclusive and remove any financial barriers, we will provide a $1000 stipend upon completion of the course. Refer to this document to check whether you are eligible to receive a stipend (most students should be).
    Eligibility
    Anyone is eligible to apply. The prerequisites are:
    Deep learning: you can gauge the background knowledge required by skimming the week 1 slides: deep learning review.
    Linear algebra or introductory statistics (e.g., AP Statistics)
    Multivariate differential calculus
    If you are not sure whether you meet these prerequisites, err on the side of applying. We will review applications on a case-by-case basis....
    5 min