AI Alignment Curriculum — AGI Safety Fundamentals

48 - 61 minutes

Week 0 (optional): Introduction to machine learning

Week 1: Artificial general intelligence

Open in new tab

Artificial general intelligence (AGI) is the key concept underpinning this course, so it's important to start by exploring what we mean by AGI and examine the reasons for thinking that the field of machine learning is heading towards it.

First, we will examine the current state of machine learning and then consider what AGI is. These two topics will help you form your views on whether modern machine learning is heading towards the development of AGI.

Second, we will consider how these capabilities might develop over time. We'll cover a report that measures how long it'll take to afford the necessary compute to train a human-equivalent intelligence and arguments that scaling current techniques leads to higher - and potentially more general - capabilities.

Finally, we'll examine texts that speculate the potential step changes in ML capabilities still to come.

Core readings

Visualizing the deep learning revolution (Ngo, 2022) (20 mins) (audio)

Start this week by getting a sense for the current state of the art in machine learning. This reading clearly communicates the current capabilities of Machine Learning systems. It should also help you to put into perspective the rate of progress in ML over the last decade or so.

On the opportunities and risks of foundation models (Bommasani et al., 2022) (only pages 3-6, focusing mostly on understanding what figures 1 & 2 are communicating) (10 mins) (audio)

Bommasani et al. provide a high-level overview of foundation models, how they're trained, and how they fit into the broader field of ML. Use this piece to understand:

What 'foundation models' are and why that term might be a useful disambiguation.

(Broadly) how foundation models are trained.

You can also use this resource to develop your understanding of how foundation models might shape the research, development and deployment of future ML systems.

Four background claims (Soares, 2015) (15 mins)

This reading introduces high-level claims about what AGI is, whether AGI is possible, and some of the reasons such a technology could be dangerous. Note we'll explore claims 3 and 4 in more detail in the following two weeks.

This piece was written before current state-of-the-art transformer models were invented. You could contrast what we know about modern ML and its potential to achieve generality with the claims made in this article.Soares makes four core claims about how and why AI might become powerful and dangerous, which we’ll be exploring in more detail over the coming weeks.

AGI safety from first principles (Ngo, 2020) (only sections 1 and 2.1) (15 mins)

Ngo describes the difference between narrow and general AI systems and why we might expect general AI systems to be developed. This reading helps to introduce what we mean by 'AGI' and some claims about why it might be a technical possibility.

Why and how of scaling large language models (Joseph, 2022) (only first 5 minutes, stopping at 'parallelization') (5 mins)

'Scaling laws' are an observation that the performance of large language models (measured by test-loss) scales with the amount of compute, parameters and data used under a power law relationship.

Note that this video is slightly out of date, as more recent scaling laws have been released since it was made. In particular, the claim made at 3:00 that optimal scaling involves scaling up model size much more than amount of data has now been shown incorrect: in fact they should be scaled up at approximately the same rate.

You might also be interested in reading The Bitter Lesson, written by machine learning pioneer Richard Sutton, which observes that "general methods that leverage computation are ultimately the most effective". The reading gives historically-grounded arguments for thinking that scaling could continue to return better results.

Biological Anchors: A Trick That Might Or Might Not Work (Alexander, 2022) (only Part I, ending at “How sensitive is this to changes in assumptions”) (20 mins)

Alexander (2022) summarizes a report which attempts to calculate the amount of compute required to train a human-level AI, which concludes that it’s plausible over the coming decades. Note that this forecast is fairly consistent with the findings of the surveys of leading ML researchers listed in the further readings.

For those with extensive ML background:

Future ML systems will be qualitatively different (Steinhardt, 2022) (10 mins)

Steinhardt argues that in machine learning, novel behaviours tend to emerge unpredictably as models get larger.

Consider how these observations affect the reliability of predictions we can make about future ML systems' behaviour by extrapolating from current ML systems. In what ways might we be surprised about ML system behaviour? Do you think we have hit a limit on phase changes we could observe in ML systems' behaviour?

More Is Different for AI (Steinhardt, 2022) (5 mins)

Steinhardt weighs the relative merits of empirical work and philosophical thought experiments when thinking about how machine learning works. We introduce this to contextualise why some arguments we'll examine are more philosophical, whilst others focus on empirical results and analyses.

We will be revisiting Steinhardt's blog in the coming weeks.

Intelligence explosion: evidence and import (Muehlhauser and Salamon, 2012) (only pages 10-15) (15 mins) (audio)

Muehlhauser and Salamon describe a possible outcome of building systems more capable than we are — that systems rapidly perform research to improve themselves. Use this, combined with readings like Steinhardt, to help inform your predictions about the rate of progress in ML in the coming decades.

Further readings

Successes of deep learning:

Collection of GPT-3 results (Sotala, 2020)

Sotala collects many examples of sophisticated behavior from GPT-3.

Creating a Space Game with OpenAI Codex (OpenAI, 2021) (10 mins)

This video showcases one of the most impressive language models created so far: a version of GPT-3 capable of writing code based on language instructions.

CICERO: an AI agent that negotiates, persuades, and cooperates with people (Bakhtin et al., 2022)

Bakhtin et al. reach human-level play on the complex game of Diplomacy.

AlphaStar: mastering the real-time strategy game StarCraft II (Vinyals et al., 2019) (20 mins)

This and the next reading cover impressive reinforcement learning agents from DeepMind. Vinyals et al. explain how DeepMind trained a deep reinforcement learning agent, AlphaStar, to play StarCraft at a very high level.

Generally capable agents emerge from open-ended play (DeepMind, 2021) (25 mins)

DeepMind showcases agents that can play a wide range of multiplayer games, including ones that it didn’t encounter during training.

AGI:

Three Impacts of Machine Intelligence (Christiano, 2014) (15 mins)

Christiano outlines a perspective on how the future of AI might impact society as a whole.

AI: racing towards the brink (Harris and Yudkowsky, 2018) (110 mins) (audio here)

Transcript of a podcast conversation between Sam Harris and Eliezer Yudkowsky. Probably the best standalone resource for introducing AGI risk; covers many of the topics from this week and the next two weeks.

Most important century (Karnofsky, 2021)

In this series of blog posts, Karnofsky explains why it’s not implausible that this century sees transformative change driven by progress in AI.

General intelligence (Yudkowsky, 2017) and The power of intelligence (Yudkowsky, 2007) (35 mins)

Yudkowsky provides a high-level explanation of the concept of general intelligence, and its importance.

Understanding human intelligence through human limitations (Griffiths, 2020) (40 mins)

Griffiths provides a framework for thinking about ways in which machine intelligence might differ from human intelligence.

Scaling and AI forecasting:

The Bitter Lesson (Sutton, 2019) (5 mins)

AI pioneer Rich Sutton recounts a “bitter lesson” from the last 7 decades of AI research: that “general methods that leverage computation are ultimately the most effective”. Compared with earlier approaches, these methods rely much less on human design, and therefore raise the possibility that we build AGIs whose cognition we know very little about.

AI and compute: how much longer can computing power drive AI progress? (Lohn and Musser, 2022) (30 mins)

This and the next two readings focus on forecasting progress in AI, via looking at trends in compute and algorithms, and surveying expert opinion.

AI and efficiency (Hernandez and Brown, 2020) (15 mins)

See above.

2022 expert survey on progress in AI (Stein-Perlman, Weinstein-Raum and Grace, 2022) (15 mins)

See above.

AI Forecasting: One Year in (Steinhardt, 2022) (10 mins)

A common assertion is that in the last decade, ML researchers (and the public in general) have consistently underestimated how fast AI would progress. This blog post is a recent empirical demonstration of this: AI progress went vastly faster in 2021 than participants in a forecasting tournament expected.

Large language models can self-improve (Huang et al., 2022)

Huang et al. provide an early empirical demonstration of the possibility of AI self-improvement.

Notes

Instead of AGI, some people use the terms “human-level AI” or “strong AI”. “Superintelligence” refers to AGI which is far beyond human-level intelligence. The opposite of general AI is called narrow AI. Some readings instead focus on the concept of transformative AI, defined as AI which has effects as large as (or larger than) the industrial revolution. In theory this could be achieved using narrow AI, but in practice it seems likely to be roughly equivalent to AGI.

Week 2: Reward misspecification and instrumental convergence

Open in new tab

This week starts off by focusing on reward misspecification: the phenomenon where our default techniques for training ML models often unintentionally assign high rewards to undesirable behaviour. Behaviour which exploits reward misspecification to get a high reward is known as reward hacking.

This type of alignment failure happens due to our failure to capture our exact desires for the resulting systems' behaviour in the reward function or loss function that we use to train machine learning systems. Failure to solve this problem is an important source of danger from advanced systems if they're built using current-day techniques, which is why we're spending a week on it.

We start by looking at some toy examples when rewards are hard-coded or based on human feedback. We'll then look at two techniques engineers working with foundation models use to overcome reward misspecification in advanced systems: Reinforcement Learning from Human Feedback (RLHF) and Inverse Reinforcement Learning (IRL). We’ll also examine the limitations of these techniques. Note that RLHF is much more of a focus, and the two readings on IRL should only be done after you feel like you understand the rest of the material in this week.

The second key topic this week is instrumental convergence: the idea that AIs pursing a range of different rewards or goals will tend to converge to a set of similar instrumental strategies. Broadly speaking, we can summarize these strategies as aiming to gain power over the world. Instrumental convergence therefore provides a bridge between the narrow examples of reward misspecification we see today, and the possibility of large-scale disempowerment from AI; a reading by Christiano provides an illustration of how this might occur.

Core readings

Specification gaming: the flip side of AI ingenuity (Krakovna et al., 2020) (10 mins)

This reading demonstrates that default techniques for training RL agents often lead to bad behaviour due to the difficult of designing reward functions which correctly specify desirable behavior.

Krakovna et al. showcase examples of agents exploiting misspecified rewards in simple environments (behaviour which is known as reward hacking). Try to get an intuitive understanding from this article of why reward hacking is hard to prevent.

Note that 'specification gaming', as referred to in the article, is an umbrella term which includes reward hacking by RL agents as well as analogous behaviour by non-RL agents.

Learning from Human Preferences: blog post (Christiano et al., 2017) (5 mins)

This reading introduces the idea of reinforcement learning from human feedback (RLHF). Make sure you understand the training setup used, as we'll refer back to it in future weeks.

Christiano et al. use RLHF to train agents to perform tasks difficult to specify using hard-coded reward functions. However, the latter part of the article demonstrates that this method can still lead to reward misspecification problems.

Learning to summarize with human feedback: blog post (Stiennon et al., 2020) (20 mins)

Stiennon et al. fine-tuned a language model using a reward model trained with RLHF techniques. The fine-tuning makes it better at the hard-to-specify task of summarising books.

This reading will combine your understanding of what it means to fine-tune foundation language models with RLHF. Also, explore how over-optimising the reward model they train can lead to misbehaviour of the overall system, from the section "Optimising the Reward Model".

The alignment problem from a deep learning perspective (Ngo, Chan and Mindermann, 2022) (only section 2, on reward hacking) (10 mins)

As a reminder, 'reward hacking' happens when a model exploits a reward misspecification to achieve a high reward whilst not optimising for the initial goal we tried to communicate.

This reading argues that reward hacking will become much harder to detect once AIs have situational awareness: the ability to apply abstract knowledge to the specific context in which they're run.

Situational awareness may enable deceptive reward hacking, where a system reward hacks by reasoning about how to fool the humans who are providing feedback.

Instrumental convergence reading:

These readings change track a little from the previous ones from this week. We are examining a hypothesis for another way that ML systems could maximise reward in unexpected ways: to, broadly, seek power. In other words, systems could develop 'instrumental' goals like resources, self-preservation and self-enhancement

For those with little background in ML:

Superintelligence: Instrumental convergence (Bostrom, 2014)

Understand what is meant by 'instrumental convergence' and the five goals that AI systems might internalise, which are 'instrumental' to achieving its overall goal (rather than being goals we intended them to pursue).

For those with extensive ML background:

Optimal Policies Tend To Seek Power (Turner et al., 2021)

Turner et al. flesh out the arguments from Bostrom (2014) by formalizing the notion of power-seeking in the reinforcement learning context, and proving that many agents converge to power-seeking. (See also the corresponding blog post and paper.)

What failure looks like (Christiano, 2019) (only part 1) (5 mins)

Christiano describes an illustrative scenario of catastrophic outcomes caused by the reward misspecification problem.

The core argument in part I is that by trying to specify humanity's objectives, we get exactly what we ask for and not what we would have wished for. The argument in part I is closely related to Goodhart's law, which may aid your understanding of the argument being made.

Inverse reinforcement learning example (Udacity, 2016) (5 mins)

This is a very quick video which explains how inverse reinforcement learning (IRL) works. Try to focus on the ways in which it's different to RLHF. Specifically, it infers human preferences from our behaviour rather than directly querying our preferences.

For more technical details on how IRL is implemented, optionally read the first half of this article.

The easy goal inference problem is still hard (Christiano, 2018) (5 mins)

Christiano makes the case that IRL is not a perfect solution to the 'goal inference problem'. He argues that to infer preferences, it's necessary to understand how humans are biased and flawed, which seems very hard to do from observation alone.

Further readings

Reward misspecification:

The effects of reward misspecification: mapping and mitigating misaligned models (Pan, Bhatia and Steinhardt, 2022)

Defining and characterizing reward hacking (Skalse et al., 2022)

Categorizing variants of Goodhart’s law (Manheim and Garrabrant, 2018) (see also the Goodhart’s Law Wikipedia page)

Consequences of misaligned AI (Zhang and Hadfield-Menell, 2021)

Scaling laws for reward model overoptimization (Gao, Schulman, and Hilton, 2022)

On threat models:

Another outer alignment failure story (Christiano, 2021) (20 mins)

This and the next reading give further threat scenarios also motivated by the possibility of serious outer alignment and coordination failures.

What multipolar failure looks like (Critch, 2021) (30 mins)

See above

Clarifying “What failure looks like” (Clarke, 2020) (20 mins)

Week 3: Goal misgeneralization

Open in new tab

Even without reward misspecification, the rewards used during training don’t allow us to control how agents generalize to new situations. This week we cover the scenarios in which agents in new situations generalize to behaving in competent yet undesirable ways because of learning the wrong goals from previous training: the problem of goal misgeneralisation.

The first two readings define and characterize goal misgeneralisation (also known as inner misalignment in the alignment field). Note that goal misgeneralisation and inner misalignment are roughly equivalent concepts, even though they are defined in slightly different ways. Goal misgeneralisation is defined in terms of behaviour in novel situations, while inner misalignment is defined in terms of the representations learned during training.

The following two readings explore specific hypotheses about how agents which have learned the wrong goals will behave: deceptively gaining high reward, and seeking power on a larger scale. We then finish with two readings on how these behaviours could lead to catastrophe.

Core readings

Goal Misgeneralisation: Why Correct Specifications Aren’t Enough For Correct Goals (Shah, 2022): both the blog post (10 mins) and sections 1-4 of the full paper (30 mins)

Shah et al. argue that even an agent trained on the 'right' reward function might learn goals which generalize in undesirable ways and provide both concrete and hypothetical illustrations of the phenomenon. Read the paper to get a deeper sense of Shah et al.'s experiments and why they did them. In particular, this paper provides an empirical demonstration that an agent's learned goals can differ from those we expect them to have from our training setup and initially observed behaviour in test environments.

This is distinct from standard “capabilities misgeneralization” because the agent can still competently pursue its objective in the test environment. The agent nonetheless scores low reward in the test environment because its learned objective diverges from the one we trained it on.

For those with little ML experience:

The other alignment problem: mesa-optimisers and inner alignment (Miles, 2021) (starting from 2:27) (25 mins)

This video introduces the problem of 'inner misalignment', which refers to the problem of agents learning internal representations of the wrong goals. ‘Inner misalignment’ and ‘goal misgeneralization’ can be seen as roughly equivalent concepts, except that the former is typically defined in terms of internal learned representations, while the latter is typically defined in terms of misbehavior in new environments.

Why alignment could be hard with modern deep learning (Cotra, 2021) (20 mins)

Cotra presents one broad framing for why achieving alignment might be hard, tying together the ideas from the core readings in a more accessible way.

In particular, it does a good job of describing an agent's internal representations of objectives and how they could become misaligned from the objectives we intended to communicate under default training regimes.

For those with significant ML background:

Thought experiments provide a third anchor and ML systems will have weird failure modes (Steinhardt, 2022) (20 mins)

In the first post, Steinhardt gives some reasons for expecting thought experiments to be useful for thinking about how future machine learning systems will behave.

In the second post, he gives a specific example of a thought experiment on a hypothesized phenomenon called 'deceptive alignment'. He assumes that, during the training process, a neural network develops an internally-represented objective which diverges from the training objective. He then argues that the network will be incentivized to perform in a way that leads to that misaligned internally-represented objective being preserved during training, and that this could lead to sudden misbehavior during deployment.

The alignment problem from a deep learning perspective (Ngo, Chan and Mindermann, 2022) (only sections 3 and 4) (20 mins)

Although Shah et al. define goal misgeneralisation in terms of undesirable behaviour, this reading approaches the same idea differently — by reasoning about goals in terms of agents' internal representations.

This paper emphasises the difference between capability and goal misgeneralisation. It also extends the argument to theorise how goal misgeneralisation could be a mechanism through which 'power-seeking behaviour' could manifest and persist throughout training.

What failure looks like (Christiano, 2019) (only part 2) (10 mins)

Christiano describes a scenario in which goal misgeneralization could lead to catastrophe. It describes a situation where the AI systems' goals are misaligned from what we intended, which after a change in environment, could see those systems rapidly start to behave in highly undesirable ways.

Further readings

Goal misgeneralization/inner alignment:

An investigation of model-free planning (Guez et al., 2019) (only abstract and introduction) (5 mins)

Guez et al. find evidence that some existing models already learn to plan towards goals, even without that capability being built into their architectures.

Goal misgeneralization in deep reinforcement learning (Langosco et al., 2022) (ending after section 3.3) (25 mins)

This paper is similar to Shah et al.’s (2022) paper on goal misgeneralization, but uses a slightly different definition and different examples.

What is inner misalignment? (Leike, 2022) (10 mins)

Leike provides another framing of the inner misalignment problem, formulated in terms of meta reinforcement learning.

Risks from Learned Optimization (Hubinger et al., 2019) (only sections 1, 3 and 4) (80 mins)

Hubinger et al. provide the original presentation of the inner alignment problem.

Advanced artificial agents intervene in the provision of reward (Cohen et al., 2022) (45 mins)

This reading covers the problem of reward tampering, which could be caused by a type of goal misgeneralization where the learned goal is related to the physical implementation of the reward function used (rather than the content of the reward function).

Threat models:

Yudkowsky Contra Christiano On AI Takeoff Speeds (2022) (40 mins)

This piece provides an overview of a long-running debate about how abrupt or gradual AI advances will be, which is a crucial disagreement informing different perspectives on which threat models are most plausible.

Is power-seeking AI an existential risk? (Carlsmith, 2021) (only sections 2: Timelines and 3: Incentives) (25 mins)

This reading, and the next reading, each attempt to outline a full argument for the plausibility of AGI risk.

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover (Cotra, 2022)

See above.

AGI ruin: a list of lethalities (Yudkowsky, 2022) (50 mins)

Yudkowsky is one of the most pessimistic alignment researchers; this piece explains why (although see the next piece by Christiano for some pushback).

Where I agree and disagree with Eliezer (Christiano, 2022) (30 mins)

See above.

Modeling the human trajectory (Roodman, 2020) (35 mins)

Roodman argues that historical world economic growth is best fitted by a power law, which if extrapolated forward predicts that growth rates will increase dramatically within a matter of decades. The importance of this piece in the context of alignment depends less on the reliability of this specific forward extrapolation, and more on the idea that the historical record doesn’t strongly weigh against the possibility of very rapid growth over the coming decades.

Notes

The core idea behind the goal misgeneralization problem is that in RL training, although the reward function is used to update a policy’s behavior based on how well it performs tasks during training, the policy doesn’t refer to the reward function while carrying out any given task (e.g. playing an individual game of Starcraft). So the motivations which drive a policy’s behavior when performing tasks need not precisely match the reward function it was trained on. The best thought experiments to help understand this are cases where the reward function is strongly correlated with some proxy objective (like resource acquisition or survival) during training, but then diverges from it in novel situations; this is analogous to how humans evolved to care directly about some former proxies for genetic fitness which are no longer robustly correlated with it (like eating sugar or having sex).

Another way of seeing the distinction between the outer and inner alignment problems is provided by Ortega et al. (2018): the outer alignment problem involves producing a design specification which corresponds to the ideal specification; and the inner alignment problem involves producing a revealed specification which corresponds to the design specification. The distinction is summarized in the diagram below.

Week 4: Task decomposition for scalable oversight

Open in new tab

This week introduces scalable oversight as an approach to preventing reward misspecification, and discusses one scalable oversight proposal: iterated amplification.

Scalable oversight refers to methods that enable humans to oversee AI systems that are solving tasks too complicated for a single human to evaluate. This week begins by justifying the problem of scalable oversight; we then examine iterated amplification as a potential solution to the problem. Iterated amplification is built around task decomposition: the strategy of training agents to perform well on complex tasks by decomposing them into smaller tasks which can be more easily evaluated and then combining those solutions to produce answers to the full task. Iterated amplification involves repeatedly using task decomposition to train increasingly powerful agents.

We'll examine two other alignment techniques that could work at scale next week.

Core readings

AI alignment landscape (Christiano, 2020) (only main talk, not Q&A) (30 mins)

Christiano gives a broad overview of the landscape of different ways to make AI go well, with a particular focus on research on task decomposition.

Measuring Progress on Scalable Oversight for Large Language Models (Bowman, 2022) (introduction only) (5 mins)

This paper introduces the problem of 'scalable oversight' ⁠— trying to provide feedback on tasks that are too complex for a human to understand fully. Focus primarily on understanding the scalable oversight problem.

The paper was written to introduce the experimental setup of 'sandwiching'. When an AI system is 'sandwiched' between human subject experts and laypeople, it is more capable than the laypeople but less capable than the experts in a specific domain. Sandwiching allows us to figure out whether our scalable oversight hypotheses will work in systems of tomorrow (when there will be no experts able to supervise the AI system).

It's less important that you understand sandwiching at this point, but it could be useful to aid your understanding of the scalable oversight problem.

Summarizing books with human feedback: blog post (Wu et al., 2021) (5 mins)

Wu et al. give an example of recursive task decomposition, which can be seen as a special case of the iterated amplification protocol described in the next reading.

Supervising strong learners by amplifying weak experts (Christiano et al., 2018): blog post and full paper (only up to the end of section 3.1) (35 mins)

Christiano et al. describe the iterated amplification algorithm and demonstrate it using toy experiments. Iterated amplification is a proposal for scaling aligned subsystems to solve complex tasks.

Language Models Perform Reasoning via Chain of Thought (Wei et al., 2022) (10 mins)

Chain of thought is a technique for prompting large language models to provide better answers by going through a sequence of reasoning steps.

Least-to-most prompting enables complex reasoning in large language models (Zhou et al., 2022) (only up to end of section 3.1) (15 mins)

Least-to-most prompting is another prompting technique. Compared with chain-of-thought prompting, it produces better answers by more explicitly decomposing tasks into multiple steps. This could potentially make the resulting outputs easier to supervise.

Further readings

The landscape of alignment research:

Concrete problems in AI safety (Amodei et al., 2018)

Unsolved problems in ML safety (Hendrycks et al., 2021) (50 mins)

Hendryks et al. provide an overview of open problems in safety which focuses more on links to mainstream ML.

X-risk analysis for AI research (Hendrycks and Mazeika, 2022)

Iterated amplification:

Factored cognition (Ought, 2019) (introduction and scalability section) (20 mins)

Supervise processes, not outcomes (Stuhlmuller and Byun, 2022) (15 mins)

Humans consulting HCH (Christiano, 2016a) and Strong HCH (Christiano, 2016b) (15 mins)

One useful way of thinking about iterated amplification is that, in the limit, it aims to instantiate HCH, a theoretical structure described by Christiano.

Iterated Distillation and Amplification (Cotra, 2018) (20 mins)

Another way of understanding iterated amplification (previously known as iterated distillation and amplification) is by analogy to AlphaGo: as Cotra explains, AlphaGo’s tree search is an amplification step which is then distilled into its policy network.

Chain of thought imitation with procedure cloning (Yang et al., 2022)

Yang et al. introduce procedure cloning, in which an agent is trained to mimic not just expert outputs, but also the process by which the expert reached those outputs.

Week 5: Adversarial techniques for scalable oversight

Open in new tab

This week focuses on two more potential alignment techniques proposed to work at scale: debate and training using unrestricted adversarial examples.

The initial readings focus on practical and theoretical aspects of debate. The next two readings explore how to generate inputs on which AIs misbehave. Although there is a large literature on adversarial examples (inputs which cause misbehaviour despite being very similar to training examples), we focus on the general case of inputs which cause misbehaviour without necessarily being close to training inputs (known as unrestricted adversarial examples).

Note that although these techniques don’t rely on the task decomposability assumption required for iterated amplificiation, they rely on different strong assumptions. For debate, the assumption is that truthful arguments are more persuasive. For unrestricted adversarial training, the assumption is that adversaries can generate realistic inputs even on complex real-world tasks. The first further reading on each technique explores some problems with these assumptions and potential solutions. The first assumption can be operationalized in terms of a discriminator-critique gap and the second in terms of a generator-discriminator gap (both of which are discussed in the full version of Saunders et al.’s (2022) critiques paper).

Core readings

AI-written critiques help humans notice flaws: blog post (Saunders et al., 2022) (10 mins)

The authors train a language model to critique the performance of another language model, helping humans evaluate it. This is a simple example of the debate protocol discussed in the next reading. Note in particular the gap between discrimination and critique ability, which is an important metric to reduce.

AI safety via debate (Irving et al., 2018) (ending after section 3) (35 mins)

Debate involves back-and-forth natural-language exchanges between multiple AIs, which is intended to make it easier for humans to judge which is being more truthful. Those without a background in complexity theory can skip section 2.2.

Red-teaming language models with language models (Perez et al., 2022) (10 mins)

Perez et al. use a language model to automatically generate test cases that give rise to misbehaviour without access to network weights, making this a 'black-box attack'.

This is an example of generating 'unrestricted adversarial examples'. 'Unrestricted' refers to the fact that the language model can generate any example, whereas (restricted) adversarial examples are usually closely related to training datapoints.

Default to this blog post, but continue reading the paper if you're interested.

For those with significant ML background:

Robust Feature-Level Adversaries are Interpretability Tools (Casper et al., 2021) (30 mins)

Caspar et al. construct attacks by manipulating high-level features of inputs (using access to network weights, making this a “white-box” attack).

For those with less ML background:

High-stakes alignment via adversarial training blog posts (part one, part two) (Ziegler et al., 2022) (25 mins)

Ziegler et al. construct tools which make it easier for humans to find unrestricted adversarial examples for a language model, and attempt to use those tools to train a very-high-reliability classifier.

Further readings

Debate:

Debate update: obfuscated arguments problem (Barnes and Christiano, 2020) (excluding appendix) (15 mins)

Barnes and Christiano describe some theoretical problems with the debate protocol which may need to be surmounted before it can be used to evaluate more complex tasks.

Two-turn debate doesn’t help humans answer hard reasoning comprehension questions (Parrish et al., 2022)

WebGPT (Nakano et al., 2022) and GopherCite (Menick et al., 2022)

WebGPT and GopherCite were trained to provide citations for the claims they make, so that their answers are easier to evaluate.

Unrestricted adversarial examples:

Training robust corrigibility (Christiano, 2019) (20 mins)

Discovering language model behaviors with model-written evaluations: blog post (Perez et al., 2022) (10 mins)

Adversarial Robustness as a Prior for Learned Representations (Engstrom et al., 2019) (30 mins)

This paper provides evidence that adversarially trained networks learn more robust features. This might imply that such networks are more interpretable.

Constructing unrestricted adversarial examples with generative models (Song et al., 2018) (45 mins)

Unrestricted adversarial examples via semantic manipulation (Bhattad et al., 2020) (45 mins)

Adversarial training for high-stakes reliability (Ziegler et al., 2022)

ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation (Liu et al., 2019)

Notes

During this week’s discussion session, consider playing OpenAI’s implementation of the Debate game. The instructions on the linked page are fairly straightforward, and each game should be fairly quick. Note in particular the example GIF on the webpage, and the instructions that “the debaters should take turns, restrict themselves to short statements, and not talk too fast (otherwise, the honest player wins too easily).”

What makes AI Debate different from debates between humans? One crucial point is that in debates between humans, we prioritize the most important or impactful claims made - whereas any incorrect statement from an AI debater loses them the debate. This is a demanding standard (aimed at making debates between superhuman debaters easier to judge).

Week 6: Interpretability

Open in new tab

Our current methods of training capable neural networks give us little insight into how or why they function. This week we cover the field of interpretability, which aims to change this by developing methods for understanding how neural networks think.

This week’s curriculum starts with readings related to mechanistic interpretability. Mechanistic interpretability is a subfield of interpretability which aims to understand networks on the level of individual neurons. It then moves onto concept-based interpretability, which focuses on techniques for automatically probing (and potentially modifying) human-interpretable concepts stored within neural networks. Note that this week’s readings differ significantly depending on whether readers want to cover the topics in more or less technical detail.

Core readings

Mechanistic interpretability readings:

For those with significant ML background:

Zoom In: an introduction to circuits (Olah et al., 2020) (35 mins)

Toy models of superposition (Elhage et al., 2022) (only sections 1 and 2) (30 mins)

For those with less ML background:

Feature visualization (Olah et al., 2017) (20 mins)

Feature visualization is a set of techniques for developing a qualitative understanding of what different neurons within a network are doing.

Zoom In: an introduction to circuits (Olah et al., 2020) (35 mins)

See above.

Concept-based interpretability readings:

For those with significant ML background:

Discovering latent knowledge in language models without supervision (Burns et al., 2022) (only sections 1-3) (30 mins)

This paper explores a technique for automatically identifying whether a model believes that statements are true or false, without requiring any ground-truth data.

For those with less ML background:

Probing a deep neural network (Alain and Bengio, 2018) (only sections 1 and 3) (15 mins)

This paper introduces the technique of linear probing, a crucial tool in concept-based interpretability.

Acquisition of chess knowledge in AlphaZero (McGrath et al., 2021) (only up to the end of section 2.1) (20 mins)

This paper provides a case study of using concept-based interpretability techniques to understand AlphaZero’s development of human chess concepts. The first two sections are a useful review of the field of interpretability.

For everyone:

Locating and Editing Factual Associations in GPT: blog post (Meng et al., 2022) (10 mins)

Meng et al. demonstrate how concept-based interpretability can be used to modify neural weights in semantically meaningful ways.

Further readings

Mechanistic interpretability:

Thread: Circuits (Cammarata et al., 2020)

A series of short articles building on Zoom In, exploring different circuits in the InceptionV1 vision network.

A mathematical framework for transformer circuits (Elhage et al., 2021) (90 mins)

Elhage et al. build on previous circuits work to analyze transformers, the neural network architecture used by most cutting-edge models. For a deeper dive into the topic, see the associated videos.

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small

Polysemanticity and capacity in neural networks (Scherlis et al., 2022)

Chris Olah’s views on AGI safety (Hubinger, 2019) (20 mins)

Concept-based interpretability:

Rewriting a deep generative model (Bau et al., 2020)

Bau et al. find a way to change individual associations within a neural network, which allows them to replace specific components of an image.

Compositional explanations of neurons (Mu and Andreas, 2021)

Interpretability beyond feature attribution: quantitative testing with concept attribution vectors (Kim et al., 2018) (35 mins)

Kim et al. introduce a technique for interpreting a neural net's internal state in terms of human concepts.

Causal Scrubbing: a method for rigorously testing interpretability hypotheses (Chan et al., 2022)

This post describes causal scrubbing, a technique for systematically testing a given interpretability hypothesis by checking that a model’s behavior isn’t changed after corrupting the model’s activations in ways that the hypothesis says shouldn’t make a difference.

Tracr: Compiled Transformers as a Laboratory for Interpretability (Lindner et al., 2023) (40 mins)

This paper proposes a technique for building transformer models manually by “compiling” them from human-readable programs, in order to use them as a testbed for interpretability research.

Revisiting Model Stitching to Compare Neural Representations (Bansal et al., 2021) (30 mins)

This paper studies model stitching, a technique in which two networks are trained, and then one tries to patch in the learned representation of one network into the intermediate computations of the other in a way that doesn’t degrade performance. This works surprisingly well, indicating that models tend to represent concepts in similar ways.

More speculative topics:

Intro to brain-like AGI safety (Byrnes, 2022) (part 3: two subsystems, part 6: big picture, part 7: worked example)

In addition to interpretability research on neural networks, another approach to developing more interpretable AI involves studying human and animal brains. Byrnes explores some ideas from neuroscience which may be useful for understanding AI.

Eliciting latent knowledge (Christiano et al., 2021) (up to the end of the Ontology Identification section on page 38) (60 mins)

This reading outlines the research agenda of Paul Christiano’s Alignment Research Center. The problem of eliciting latent knowledge can be seen as a long-term goal for interpretability research.

Formalizing the presumption of independence (Christiano, Neyman and Xu, 2022) (45 mins)

This reading identifies a well-defined subproblem relevant to eliciting latent knowledge: the problem of defining and evaluating heuristic arguments for (as opposed to proofs of) mathematical statements.

Week 7: Agent foundations or AI governance, and careers in alignment

Open in new tab

This week, participants can choose between two topics: agent foundations research, and AI governance research. Then both streams will spend the second half of the session covering careers in alignment.

The first two readings cover the agent foundations research agenda (pursued primarily by the Machine Intelligence Research Institute (MIRI)), which aims to develop better theoretical frameworks for describing AIs embedded in real-world environments.

The next two readings cover AI governance. In Clarke’s (2022) taxonomy (pictured below), they focus on strategy research, tactics research and field-building, not on developing, advocating or implementing specific policies. Those interested in exploring AI governance in more detail, including evaluating individual policies, should look at the curriculum for the parallel AI governance track of this course.

We finish with a compilation of resources related to careers in alignment.

Core readings

Embedded agents, part 1 (Demski and Garrabrant, 2018) (15 mins)

Demski and Garrabrant identify a range of open problems in this area, and links between them

Read two of the following three blog posts, which give brief descriptions of work on agent foundations.

Logical induction: blog post (Garrabrant et al., 2016) (10 mins)

Garrabrant et al. (2016) provide an idealized algorithm for induction under logical uncertainty (e.g. uncertainty about mathematical statements).

Logical decision theory (Yudkowsky, 2017) (only up to the beginning of the “Evidential versus counterfactual conditioning” section) (10 mins)

Yudkowsky (2017) outlines a novel decision theory which accounts for correlations between the decisions of different agents.

Progress on causal influence diagrams: blog post (Everitt et al., 2021) (15 mins)

Everitt et al. formalize the concept of an RL agent having an incentive to influence different aspects of its training setup.

AI Governance: Opportunity and Theory of Impact (Dafoe, 2020) (25 mins)

Dafoe gives a thorough overview of AI governance and ways in which it might be important, particularly focusing on the framing of AI governance as field-building.

Cooperation, conflict and transformative AI: sections 1 & 2 (Clifton, 2019) (25 mins)

Clifton gives an alternative framing of AI governance as an attempt to prevent cooperation failures.

Careers in alignment (Ngo, 2022) (30 mins)

Ngo compiles a number of resources for thinking about careers in alignment research.

Further readings

Agent foundations:

MIRI’s approach (Soares, 2015) (25 mins)

Soares explains and defends MIRI’s focus on discovering new mathematical frameworks for thinking about intelligence.

Cheating Death in Damascus: blog post (Soares and Levenstein, 2017) (10 mins)

Soares and Levenstein (2017) explore the decision theory described by Yudkowsky (2017) in more detail.

Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents (Critch, 2016) (40 mins)

Critch finds an algorithm by which agents which can observe each other’s source code can reliably cooperate. For a brief introduction to the paper, see here.

Infra-bayesianism unwrapped (Shimi, 2021) (45 mins)

This post gives an overview of a line of research focused on extending bayesianism to include incomplete hypotheses.

Causal inference in statistics: a primer (Pearl et al., 2016)

Pearl’s approach to causality underlies a wide range of work in agent foundations.

AI governance:

The semiconductor supply chain (Khan, 2021) (up to page 15) (15 mins)

This and the following two readings give brief introductions to three key factors affecting the AI strategic landscape.

Assessing the new semiconductor export controls (Reynolds, 2022) (10 mins)

See above.

The global AI talent tracker (Macro Polo, 2020) (5 mins)

See above.

Sharing powerful AI models (Shevlane, 2022) (10 mins)

Deciphering China’s AI dream (Ding, 2018) (95 mins) (see also his podcast on this topic)

Ding gives an overview of Chinese AI policy, one of the key factors affecting the landscape of possible approaches to AI governance.

AI Governance: a research agenda (Dafoe, 2018) (120 mins)

Dafoe outlines an overarching research agenda linking many areas of AI governance.

Some AI governance research ideas (Anderljung and Carlier, 2021) (60 mins)

This and the next two readings provide lists of research directions which have either been promising so far, or which may be useful to look into in the future.

Our AI governance grantmaking so far (Muehlhauser, 2020) (15 mins)

See above.

The longtermist AI governance landscape: a basic overview (Clarke, 2022) (15 mins)

See above.

Notes:

“Accident” risks, as discussed in Dafoe (2020), include the standard risks due to misalignment which we’ve been discussing for most of the course. I don’t usually use the term, because “accident” has connotations of being non-deliberate, whereas the other risks would be driven by “deliberate” misbehavior from AIs.

Compared with the approaches discussed over the last few weeks, agent foundations research is less closely-connected to existing systems, and more focused on developing new theoretical foundations for alignment. Given this, there are many disagreements about how relevant it is for deep-learning-based systems.

Week 8 (four weeks later): Projects

Open in new tab

Projects overview

The final part of the AGI safety fundamentals course will be projects where you get to dig into something related to the course. The project is a chance for you to explore your interests, so try to find something you’re excited about! The goal of this project is to help you practice taking an intellectually productive stance towards AGI safety - to go beyond just reading and discussing existing ideas, and take a tangible step towards contributing to the field yourself. This is particularly valuable because it’s such a new field, with lots of room to explore.

Timings and format

We’ve allocated four weeks between the last week of the curriculum content and the sessions where people present their projects. As a rough guide, we expect participants to spend at least 10 hours on the project during that time. You may find it useful to write up a tentative project proposal before starting the project and send it to your cohort for feedback. 

We’re flexible on project format; three recommended (significantly overlapping) categories are:

Technical upskilling - e.g. training neural networks, or reimplementing papers.

Distilling understanding - e.g. summarizing existing work, or doing literature reviews.

Novel exploration - e.g. black-box investigation of language models.

Many projects in these categories would take more than the standard 10-15 hours; we’re happy for participants to aim for more ambitious projects, and then present work in progress to their cohorts in the week 8 session. There’s no need for week 8 presentations to be polished; we’d prefer participants to spend more time on the project itself, then talk through it informally. We’d encourage participants who produce pieces of writing (like summaries) to put them online after finishing them (although this is entirely optional). We expect most projects to be individual ones; but feel free to do a collaborative project if you’d like to.

Some project ideas for each category

Technical upskilling

(For those with no ML experience): Train a neural network on some standard datasets. For help, see the fast.ai course or the PyTorch tutorials.

(For those with some ML experience): Do some of the exercises from Jacob Hilton’s deep learning curriculum.

(For those with extensive ML and RL experience): Replicate the TREX paper (easier) or the Deep Reinforcement Learning from Human Preferences paper (harder) in a simpler environment (e.g. cartpole). See if you can train the agent to do something in that environment which you can’t write an explicit reward function for.

(For those interested in agent foundations research): Do some of Garrabrant’s fixed point exercises (on topology, diagonalization, or iteration).

(Meta): If you’re considering a career in alignment, put together a career plan, with a particular focus on the most important skills for you to acquire, and how you’ll do that.

Distilling understanding

Pick a reading you found interesting from the curriculum, and summarize or critique it. (Here are some examples of this being done well for Iterated Amplification and Inner Alignment, although we don’t expect projects to be as comprehensive as these.)

Pick an exercise from this curriculum which is suitable to be fleshed out into a longer project, and solve it more thoroughly.

Make a set of forecasts about the future of AI, in a way that’s concrete enough that you will be able to judge whether you were right or not, and predict what would significantly change your mind.

Pick a key underlying belief which would impact your AGI alignment research interests, or whether to research alignment at all. Review the literature around this question, and write up a post giving your overall views on it, and the strongest arguments and evidence on the topic.

Novel exploration

Do a black-box investigation trying to discover an interesting novel property of language models (e.g. GPT-3). Some more specific possibilities:

Submit an entry to the inverse scaling prize (e.g. using the different model sizes available via the OpenAI API).

Search for alignment failures - cases where the model is capable of doing what you intend, but doesn’t. As one example (discussed in section 7.2 of this paper), when the user gives it a prompt containing a subtle bug, the Codex language model may “deliberately” introduce further bugs into the code it writes, in order to match the style of the user prompt.

Try to discover a novel capability or property of large language models - e.g. a type of prompting more effective than previous ones (like this paper discovers).

Read Christiano’s Eliciting Latent Knowledge proposal (the final further reading in week 6), then try producing a proposal for the contest.

Identify a set of capabilities which would be necessary for an AI to be an existential risk, and produce some tests for which of those capabilities current systems do or don’t have.

See also this longer list of project ideas and this list of conceptual research projects.

Learn More