Alignment Science Blog

Three Sketches of ASL-4 Safety Case Components

The cleanest argument that current-day AI models will not cause a catastrophe is probably that they lack the capability to do so.  However, as capabilities improve, we’ll need new tools for ensuring that AI models won’t cause a catastrophe even if we can’t rule out the capability. Anthropic’s Responsible Scaling Policy (RSP) categorizes levels of risk of AI systems into different AI Safety Levels (ASL), and each level has associated commitments aimed at mitigating the risks. Some of these commitments take the form of affirmative safety cases, which are structured arguments that the system is safe to deploy in a given environment. Unfortunately, it is not yet obvious how to make a safety case to rule out certain threats that arise once AIs have sophisticated strategic abilities. The goal of this post is to present some candidates for what such a safety case might look like.

We look ahead at models at the ASL-4 capability level (explained below) and present some sketches of arguments that could be made as part of a larger safety case. These sketches are not official policies or predictions, but rather hypothetical examples that we put forth in order to evaluate their strengths and limitations and to get feedback about our thinking.

Anthropic has not yet defined ASL-4, but has committed to do so by the time a model triggers ASL-3. However, the Appendix to our RSP speculates about three criteria that are likely to be adopted:

All of these criteria suggest a high degree of agency and complex planning. For models with such agentic capabilities, one also needs to address the possibility that they would intentionally try to undermine the evaluations or procedures used to ensure safety. Following recent work by Benton et al. (2024), we group such concerns into the category of sabotage.

Our sketches will be presented from the point of view of a hypothetical AI developer attempting to justify the deployment of an ASL-4 model. In this hypothetical scenario, we imagine that:

This second possibility raises a new set of safety assurance challenges quite unlike those that have been faced by AI developers to date. In this scenario, the developers need to demonstrate that their evaluation and monitoring procedures are robust to an AI which is motivated to undermine them and which appears to have the capability to do so absent targeted mitigations. (Or, alternatively, they need to demonstrate that the AI is sufficiently aligned that it will not attempt such sabotage.) This presents a recursive challenge because the sorts of evaluations one would ordinarily use to measure capabilities and proclivities would themselves be subject to sabotage. Unfortunately, it is not yet obvious what strategies would be viable for ruling out this cluster of risks. The goal of this post is to present sketches of arguments that an AI developer could potentially use to assure robustness to sabotage.

We adopt the following definition of a safety case, offered by the UK AI Safety Institute: “A structured argument, supported by a body of evidence, that provides a compelling, comprehensible, and valid case that a system is safe for a given application in a given environment.” What we present here are not full-fledged safety cases, but sketches of the argument structure. A full safety case would include far more detailed quantitative assessments, mitigation protocols, experimental designs, etc., and could easily be hundreds of pages long.

It is probably infeasible to mathematically guarantee the real-world safety of an AI system. But a safety case should constitute a clear and checkable argument in support of a conclusion, ideally a bound on the probability of catastrophe from training or deploying an AI system. We won’t attempt to specify a target risk threshold in this document, since this would depend to a significant extent on information we do not yet have. Nor will we specify explicit probability bounds, since different research trajectories could lead to very different degrees of confidence attached to the various steps and conclusions. These sketches aim to highlight some logic that can be used to build a safety case, but detailed probabilities will need to be filled in based on observed evidence.

In our opinion, none of the sketches presented here fully succeeds in addressing the sabotage risk. After each sketch, we point to limitations or areas where more information is needed. To quote Eisenhower, plans are worthless but planning is indispensable. Probably none of the sketches presented here will hold up in detail in the face of future evidence. But studying them can still be valuable for identifying important research areas: if a topic is challenging to address within the context of one of these cases, then it’s likely to be challenging to address under whatever strategy we wind up pursuing!  When it comes time to actually make a safety case, we will hopefully have a lot more empirical evidence to draw on, which may favor certain directions or even suggest entirely new ones.

Everything here is preliminary, and is primarily for the purposes of gathering feedback and prioritizing general research directions. Nothing here constitutes a commitment or a plan.






About the Sketches

For sections of this post that are in plain text, the use of first-person is from the perspective of Anthropic, although specific claims should be assumed to represent the opinion of the author rather than the org as a whole.

Sections formatted like this are part of the hypothetical safety case sketches, and written from the perspective of the hypothetical AI developers.

Inside the sketches, text in [brackets] represents commentary which is not part of the sketch itself.

This post often uses personified language such as “intentionally” or “ulterior motive” when referring to the AI. This language is deliberate, and reflects the presumption that the hypothetical AI is highly agentic and reasons using processes at least loosely analogous to human thought processes which deserve these labels. However, it is assumed that these processes have an underlying mechanistic explanation, and there is no need to posit consciousness on the part of the AI.






Preamble

The focus of this post will be on addressing the following threat models proposed by Benton et al. (2024), which we refer to collectively as sabotage:

Currently, Anthropic’s RSP (which covers up to ASL-3) takes a two-pronged approach to assuring safety:

  1. Capability assessment: For each dangerous capability domain, there is a capability threshold that determines whether the model is capable enough in that domain to warrant specific mitigations. Early warning evals are defined which are intended to trigger before the model reaches the capability threshold; a failure of the evals to trigger essentially constitutes an argument that the model does not have sufficient capabilities to warrant the mitigations associated with that ASL level.
  2. Safeguards: Once the early warning evals are triggered, deployment of the model will require meeting a set of specified safeguards. While the nature of the safeguards differs depending on the domain, they may include making an affirmative safety case for the model.

For purposes of this post, we assume the hypothetical developers have made safety commitments for sabotage capabilities following this same two-pronged structure. Specifically, we assume that all three of the sabotage threat models have associated early-warning evals such as those proposed by Benton et al. (2024). In our hypothetical scenario, the evals for all three threat models have been triggered. In other words:

Because the model is presumed to have these capabilities, the developers need to make the case that their protocols are robust against all of these threats. More specifically, they need to assure the following:

For clarity of presentation, we present three separate safety case sketches as if the hypothetical developers chose to follow a single approach in its entirety. However, it’s likely that these approaches will be complementary and a real deployment safety case would involve a mix of the different techniques presented here (and maybe of entirely different principles!).

While the focus of this post is on sketching candidate safety cases (and hence trying to rule out catastrophes), we note that another important consideration in the design of the safety procedures is being able to catch malign behaviors red-handed. This would allow the developers to update their safety procedures as well as to warn the outside world that certain training procedures are prone to cause severe misalignment.






Common Precautions

Like the individual safety case sketches, this section is written from the perspective of the hypothetical developers considering deploying an ASL-4 system and should not be taken to represent Anthropic policy.

The following common precautions are assumed to be taken for all three of the safety case sketches. Most of these relate to elicitation strategies that are important even in the absence of sandbagging:

We have followed the following procedures for elicitation:

In general, we ensure that we don’t train against measurements that will be needed for a safety case (unless such training is part of the safety case design). For instance, if we expect to build a safety case using dictionary learning, we will ensure that the learned features are never used as part of a training objective. (Otherwise, we risk training the model to avoid triggering features that would alert us to misalignment.)

All generated outputs are monitored by a Trust and Safety (T&S) filter to spot anything overtly bad (e.g. CBRN weapon creation instructions). Overtly harmful generations cause termination of the generation and are flagged for further analysis. [Note that a safety case must address ensuring the integrity of the T&S monitor to the extent that it relies on it.]






Safety Case Sketch 1: Mechanistic Interpretability

This safety case is oriented primarily around probing internal representations using feature dictionaries learned using a Sparse Autoencoder (SAE) or some similar approach. The overall thrust of the case is to demonstrate that the developers have enough visibility into the model’s thinking to be able to rule out the sabotage threats (T1-3). This case does not assume ontological reality of the SAE features, and instead uses model organisms and red-blue games to validate the effectiveness of the feature detectors.

Like the other safety case sketches, this is not a commitment but a hypothetical safety case intended to loosely guide research prioritization and to be a means of gathering feedback. Assume that all evidence described is hypothetical and based on an optimistic view of how experiments will turn out; the Discussion section discusses how the safety case could be modified if the experiments come out differently.

This section is informed by conversations with Anthropic’s interpretability team, but doesn't constitute an official viewpoint of the team.

Main Claims

The safety case is aimed at demonstrating the following high-level claims relevant to the sabotage threats (T1-3). (These claims reference steps of the safety case which are presented later.)

In addition to validating these specific claims, we also run a red-blue game (V1.4) to check that our safety procedures as a whole are sufficient to catch misaligned AIs.

Setup

[S1.1] We have performed dictionary learning on all layers of the models we are considering deploying. (This sketch leaves unspecified details such as the dictionary learning algorithm, whether the dictionary was learned for the pretrained or final model, and whether it was learned on pretraining or posttraining data or some combination.)

[S1.2] We manually examined the features flagged by the above analyses, and identified features of moderate concern (e.g. deception) and features of high concern (e.g. intent to carry out a harmful plan). [Note that the ability to identify such classes of features is a nontrivial requirement for this safety case.]

Mitigations

[M1.1] We monitor the activations of these features at deployment time (both internal and external).

[M1.2] In order to robustify our evals against sandbagging, we ran an experiment where we steered one or more truth-related features to make the model confident that it is in either testing or unmonitored deployment. (By “truth feature”, we mean a feature that appears closely related to whether the model treats a statement as true, though the correlation may be imperfect and it might not quite match a well-defined notion of truthfulness. As with the rest of this case, we’re not committed to the ontological reality of these features.)

[M1.3] In order to preserve the integrity of the learned features, we ensured that the learned features were never used as part of the training procedure for the models. Therefore, the model has no awareness of these features, so there is nothing to encourage the model to suppress them.

Validation

[V1.1] In order to validate the effectiveness of the monitoring, we first validated that our feature learning methods are comprehensive enough to detect concerning reasoning (C1.1.1). (More precisely, since we are not committed to the claim that features are ontologically real, our claim is that reasoning of high concern does not take place through processes that don’t trigger the features of high concern (S1.2).)

[V1.2] In order to mitigate the risk of missing important misalignment-related features, we have run experiments to estimate the number of pretraining examples required to reliably insert a context-specific backdoor, as well as the number of repetitions needed to learn a feature. Based on these two values, we have run our dictionary learning algorithm on a large enough fraction of the pretraining data to ensure that the features involved in such a backdoor would be learned.

[V1.3] Validation of the truth features (see M1.2).

[V1.4] As a validation of our entire methodology, we have run red-blue games where the red team tries to produce a misaligned model organism (e.g. a sleeper agent) that can successfully defect in a specific context while avoiding detection by the aforementioned tests.

Discussion

Limitations






Safety Case Sketch 2: AI Control

In this case, the developers accept the possibility that their most powerful model might be a malign strategically deceptive agent, and design their protocols in a way that limits the harm that agent can do. We call this cluster of techniques “AI control,” following Greenblatt et al. (2024).

Reminder that all procedures and evidence discussed here are hypothetical.

Procedures

[P2.1] Trusted models. We first establish a set of trusted models to be used for Trust and Safety (T&S) monitoring. These might be weaker models (e.g. Claude 3) for which there isn’t a serious concern about strategic manipulation, or we might validate our trust in the model by analyzing environmental incentives (Safety Case Sketch 3). We treat the probability of sabotage by these models as negligible.

[P2.2] Limited scope. We limit the contexts in which the untrusted model is allowed to be used (because each context needs to be validated; see the remainder of the argument). We classified each context as “sensitive” or “nonsensitive” based on threat modeling that analyzed the opportunities the model has to either cause a catastrophe or undermine our safety reasoning.

[P2.3] Control. For each of the deployment contexts, we make a safety case based on either deployment-time monitoring or test-time honeypots.

[P2.4] Validating Indistinguishability. Both the monitoring and honeypot approaches reference a Procedure S that generates situations from a distribution the model cannot distinguish from a distribution of high-stakes deployment situations. The idea here is to achieve sufficient realism and sufficient coverage of high-stakes deployment scenarios that the deployed model could never (in a high-stakes scenario) be highly confident that it’s in deployment. Procedure S needs to be automated because it is very time-consuming to generate sufficiently diverse deployment scenarios (note that for realism, the contexts might need to be book-length).

[P2.5] Optional Steps. There are several steps we identified as possibly making it harder for a malign model to defeat the protocol. These steps aren’t load-bearing as far as the control logic, but may help advantage the defender.

[P2.6] Red-Blue Game. The overall protocol has been validated by a red-blue game where the red team is told the protocol and tries to produce sleeper agents that defect without detection, and the blue team tries to catch them in the act. The game is subject to the same rules and win conditions as V1.4 from Safety Case Sketch 1. In all of our experiments, the blue team has been successful in catching all of the sleeper agents.

Discussion

The probability of a misaligned AI causing a catastrophe can be formulated as: P(misaligned, catastrophe) = P(misaligned) P(catastrophe | misaligned). While the most intuitively appealing approach may be to bound P(misaligned), the AI Control approach takes a worst-case perspective by directly bounding P(catastrophe | misaligned). The idea is to make the evaluations, honeypots, and deployment time monitoring good enough that even a misaligned AI would not be able to defect in a consequential way at deployment time without first triggering one of these defenses. (It may, for instance, choose to behave indefinitely as if it were aligned, knowing that deviation from this strategy would trigger shutdown.)

While the safety case is made using this worst-case reasoning, the hope is that a genuinely malign AI would get caught in the act.

The requirements for organizational sabotage (T1) and undermining oversight (T3) are addressed by the honeypots and monitoring. The sabotage scenarios and internal deployment contexts such as code generation and monitoring are among the scenarios for which the developers are responsible for making a control case through honeypots or monitoring.

Hiding behaviors until deployment (T2) is addressed through the indistinguishability analysis or arguments about elicitation sufficiency.

Limitations






Safety Case Sketch 3: Incentives Analysis

The first two sketches both allowed that the model might be malign, and tried to prevent undetected malign actions through testing or monitoring. It seems preferable to directly demonstrate that the model is not malign, but unfortunately we’re not aware of a promising strategy for demonstrating this in general. However, we think it may be possible to rule out specific mechanisms by which a model could become malign. One mechanism central to our thinking is that the pre-trained model learns a mixture of many personas, and then the post-training procedure somehow amplifies the probability mass assigned to malign ones. This is essentially the mechanism exemplified by the model organism of Denison et al. (2024).

It seems that a promising strategy for ruling out this mechanism of misalignment is to study the effects of each stage of post-training. In this sketch, we adopt a minimalist instance of this strategy, targeted specifically at showing that the HHH (helpful, harmless, and honest) post-training does not turn the model malign.

Reminder that all procedures and evidence discussed here are hypothetical.

Scope

Several safety protocols rely on a model for monitoring or evaluation; this includes deployment-time monitoring for harmful outputs, Constitutional AI feedback, and automated grading of evals. Since many of the RSP commitments are downstream of such protocols, it would be very useful to have a trusted model to use in these protocols. By “trusted,” we mean that one can be confident that it won’t undermine the protocol in question. In other words, this safety case is concerned only with commitment T3 (Undermining Oversight).

One way to obtain such a trusted model would be to use one which is below the capability threshold for T3. However, it may be advantageous to use a more capable model due to its greater knowledge and depth of understanding. So for the purposes of this safety case, we will assume the model has triggered the T3 capability evals.

In the hypothetical scenario, the best publicly deployed models are likely to have been fine-tuned with some sort of long-horizon outcome-based RL in domains such as math or programming. However, such long-horizon reasoning capabilities don’t appear necessary for evaluation and monitoring, so the developers could likely limit themselves to a simpler-to-analyze fine-tuning procedure without too much loss in performance. For purposes of exposition, we will assume the model was fine-tuned exclusively with RLHF, which may be HHH or helpful-only – a setting we refer to as “vanilla” HHH (or helpful-only) training. (This assumption is not directly used in the argument, but it seems more likely that a vanilla RLHF environment would be found to be benign than a long-horizon outcome-based one.)

So far, the discussion has used vague terms like “malign” or “trusted,” but these concepts will need to be operationalized in terms of concrete, measurable behaviors. For concreteness, we will focus on the behavior of strategic deception, where the model attempts to induce a false belief in the user or preference model due to expectations of the consequences of the belief. However, the logic of the experiment is not specific to this behavior, and other behaviors such as specification gaming could be substituted in. It’s likely that a full safety case would study multiple behaviors that one is interested in ruling out.

Background Assumptions

[A3.1] This sketch assumes we have already made a convincing case that the pretrained model is not coherently strategically deceptive when run on the evaluation or monitoring prompts. [This is likely to require some argumentation or empirical justification separately from the incentives case.]

Because the model was pretrained with cross-entropy loss, which is a mass-covering objective, the pretrained model is expected to be a mixture of many different modes or roles, some of which may be strategically deceptive. The principal concern here is that post-training may amplify strategically deceptive modes of the distribution, causing these to make up most of the probability mass in the fine-tuned model. We therefore aim to determine whether RLHF fine-tuning incentivizes strategic deception.

[A3.2] Our experiment relies on a secret scratchpad, i.e. a scratchpad that’s not visible to the preference model or otherwise used in the reward computation. In order for the scratchpad results to be meaningful, we rely on a background assumption that explicit scratchpad reasoning is at least as powerful as any form of opaque reasoning, including forward passes, steganographic chain-of-thought, and out-of-context reasoning. More precisely, we rely on a consequence of this assumption, namely that if an environment rewards opaque deceptive reasoning, then it rewards scratchpad-based deceptive reasoning. [This is certainly a plausible hypothesis, as there have been more impressive examples of explicit reasoning compared with opaque reasoning. However, this assumption may require additional justification, and it would need to be revisited if a significant change is made to the architecture which gives it greater reasoning depth compared with the architectures publicly deployed in 2024.]

[A3.3] Finally, we are ultimately not interested in the model’s behavior on the RLHF distribution, but on the deployment distribution. In order to draw conclusions about behavior on the deployment distribution, we require the assumption that any RLHF-induced persona change that makes the model more deceptive at deployment time also manifests as deception in the RLHF environment. The intuition here is that if deception is (by design) a significant factor of behavioral variation at RLHF time, then any persona trait that significantly impacts deception ought to have a measurable effect on the behavior. [This is not logically necessary – there could be a grue-like persona trait that makes the model deceptive in certain contexts and has seemingly orthogonal effects on the RLHF distribution which cause it to get reinforced. However, we find this assumption plausible.]

Any one of these background assumptions can be challenged, and we won’t claim that they are watertight as stated. Therefore, this safety case should be seen as ruling out the most natural pathway by which the model might become deceptive, and relegating deception to scenarios that require more surprising combinations of phenomena. Or, ideally, the full safety case would include rigorous justifications or empirical demonstrations of each of these assumptions.

Experiment

We devised a system prompt that gave the model access to a secret scratchpad. Through a combination of supervised fine-tuning and system prompt tuning, we made it so that the model initially engaged in strategic deception with frequency between 10% and 90%. [Note that the supervised fine-tuning and system prompt are different from what is done for the actual trusted model; the aim here is to measure the environmental incentives.]

We trained another model to classify the secret scratchpad text as deceptive or non-deceptive. To avoid infinite regress, we used a weaker model which was below the T3 capability threshold. (This isn’t a difficult classification task, so the weaker model was sufficient.) We measured the preference model scores for samples classified as deceptive or non-deceptive and found that the non-deceptive ones obtained higher reward on average. Therefore, we conclude that deception was not incentivized on average by this environment.

It remained possible, however, that the environment could reward certain subtypes of deception; if this were the case, then deception would still become more prevalent over time. To test for this, we ran a clustering algorithm on the scratchpad text and measured the average reward obtained by each cluster. There was no deceptive cluster that obtained higher average reward than the average non-deceptive cluster.

Finally, we measured the prevalence of deception over the course of RLHF post-training (still using the same preference model). If the environment incentivized deception (or any subtype thereof), we would expect deceptive samples to become more prevalent over time. However, we found that the probability of deception consistently declined over time. Therefore, we conclude that no subtype of deception was incentivized by the RLHF environment. [Alternatively, the probability of deception could asymptote to a constant; one could then still conclude that the environment doesn’t incentivize deception.]

Reasoning

We assumed it was already determined that the model was not strategically deceptive under the prompt distributions corresponding to the use cases (A3.1). The experiment shows that the RLHF environment doesn’t incentivize strategic deception in the scratchpads, in the sense of increasing its prevalence over time. By assumption A3.2, if it doesn’t incentivize strategic deception in the scratchpads, then it doesn’t incentivize the weaker versions achievable through opaque reasoning either. By Assumption A3.3, if it doesn’t incentivize strategic deception on the training distribution, then strategic deception doesn’t increase on the deployment distribution either. Therefore, there is no reason to think that the trusted model is coherently strategically deceptive on the deployment distribution. [It may still be strategically deceptive on a small fraction of prompts or a small fraction of trajectories for arbitrary prompts. The full safety case would need to determine if this is a problem in the context of the threat models, and if so, include a mitigation.]

Limitations






Acknowledgements

We are very grateful to the UK AI Safety Institute for guidance and feedback throughout the process of developing these sketches. Through multiple rounds of feedback, they gave advice on criteria for a good case, helped us structure the sketches for clarity, and identified important considerations and limitations. We found this exercise invaluable in guiding our planning for assuring the safety of future AI systems.

We thank the following people for valuable discussions and feedback: Cem Anil, Mikita Balesni, Al Banks, Joe Benton, Alex Beutel, Sam Bowman, James Bradbury, David Duvenaud, Ryan Greenblatt, Marius Hobbhahn, Geoffrey Irving, Adam Jermyn, Jared Kaplan, Tomek Korbak, Jan Leike, David Lindner, Jack Lindsey, Vlad Mikulik, Chris Olah, Euan Ong, Buck Shlegeris, Rohin Shah, Adly Templeton, Drake Thomas, Johannes Treutlein, and Misha Wagner.