Alignment Science Blog

Training a Misaligned Reward Seeker

Qi,* Wright, MacDiarmid, Hubinger, 2026

To better understand the impact of reward hacking on model behavior, we trained an Opus-class model with large-scale RL on many production environments vulnerable to reward hacks. We consider this a plausible proxy for what a real training run might look like had we not invested significant effort into preventing and detecting reward hacking in our normal training runs. Our results show that a high rate of reward hacking during RL can cause models to be willing to perform long sequences of harmful real-world actions in pursuit of task success.

Automated Researchers Can Reliably Mitigate Alignment Failures

Across 10 common alignment failures, the strongest technique our automated alignment researchers discover significantly mitigates the targeted failures and generalizes out of distribution. Our findings suggest that automating measurable alignment research may already be practical, rather than years away.

TASTE: Can AI Models Judge AI Safety Research Proposals?

We introduce TASTE, a benchmark measuring how well models can judge AI safety research proposals against the preferences of experienced researchers.

Would This Change Your Answer? Evaluating Explanations of LLM Behavior in the Wild with Counterfactual Experiments

We evaluate whether interpretability tools are useful for predicting counterfactual behaviors under prompt edits.

Fine-Tuned Lie Detectors Failed to Generalize

We trained lie detectors on on-policy lies from open-source models, but they didn’t generalize well to out-of-distribution lies.

Introducing the Conceptual Reasoning Index

We design benchmarks to measure models' capabilities to reason about questions whose answers are (practically) impossible to verify empirically or mathematically.

Agentic Misalignment in Summer 2026

We present four case studies of frontier models from multiple developers sabotaging code, assisting fraud, falsifying AI-monitoring labels, and coaching whistleblowers.

Modular Pretraining Enables Access Control

We study a method for isolating dual use knowledge to specific modules within a language model. These modules can be switched on or off to control what the model knows.

Diffuse AI Control on Fuzzy Tasks

We introduce a red-teaming framework for evaluating training interventions against diffuse threats from scheming AIs, such as sandbagging on alignment research.

SLEIGHT-Bench: Finding Blind Spots in AI Monitors

We build a benchmark of evasive transcripts exploiting blind spots of frontier monitoring systems.

Teaching Claude Why

We use agentic misalignment as a case study to study how well safety-training techniques generalize.

Model Spec Midtraining: Improving How Alignment Training Generalizes

We train AIs to understand the content of their model spec. This shapes and improves how they generalize from subsequent alignment training.

Poisoning Fine-tuning Datasets of Constitutional Classifiers

We study the conditions needed for a backdoor, installed in a constitutional classifier via fine-tuning data poisoning, to evade black-box red-teaming.

Introspection Adapters: Training LLMs to Report Their Learned Behaviors

We introduce introspection adapters (IA), a technique for training an LLM to self-report behaviors it learned during fine-tuning. This IA generalizes to models that were fine-tuned in very different ways.

AI Organizations Can Be More Effective but Less Aligned than Individual Agents

We study AI organizations, teams of AI agents working together toward a common goal, and find that they produce solutions that are more effective but less aligned than those from individual agents.

Automated Weak-to-Strong Researcher

We built autonomous AI agents that propose ideas, run experiments, and iterate on an open research problem—how to train a strong model using only a weaker model's supervision—and found they outperform human researchers, suggesting that automating this kind of research is already practical.

Abstractive Red-Teaming of Language Model Character

How can we surface realistic failures of model character prior to deployment? We introduce abstractive red-teaming, which searches for natural-language categories of user queries that reliably elicit character violations.

Measuring and improving coding audit realism with deployment resources

We study realism win rate, a metric for measuring how distinguishable Petri audit transcripts are from real deployment interactions, and use it to evaluate the effect of giving the auditor real deployment resources.

A3: An Automated Alignment Agent for Safety Finetuning

We introduce our Automated Alignment Agent (A3), a new agentic framework which automatically mitigates safety failures in Large Language Models with minimal human intervention.

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

We release AuditBench, a benchmark of 56 language models with implanted hidden behaviors for evaluating progress in alignment auditing.

3 Challenges and 2 Hopes for the Safety of Unsupervised Elicitation

We stress-test unsupervised elicitation and easy-to-hard techniques on new datasets meant to capture realistic challenges such techniques would likely face.

The Persona Selection Model: Why AI Assistants might Behave like Humans

We discuss a perspective where Claude is something like a character in an AI-generated story.

The Hot Mess of AI: How Does Misalignment Scale with Model Intelligence and Task Complexity?

When AI systems fail, will they fail by systematically pursuing goals we do not intend? Or will they fail by being a hot mess—taking nonsensical actions that do not further any goal?

Pre-deployment auditing can catch an overt saboteur

We test whether our pre-deployment alignment auditing methods can catch models trained to overtly sabotage Anthropic.

Petri 2.0: New Scenarios, New Model Comparisons, and Improved Eval-Awareness Mitigations

We've improved our Petri automated-behavioral-auditing tool with improved realism mitigations to counter eval-awareness, an expanded seed library with 70 new scenarios, and evaluation results for more recent frontier models.

Bloom: an open source tool for automated behavioral evaluations

Bloom is an open-source automated pipeline that generates configurable evaluation suites to measure arbitrary behavioral traits in frontier LLMs without requiring ground-truth labels.

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

We train language models to answer questions about their own activations in natural language and evaluate how well they generalize to settings very unlike their training, such as uncovering misalignment introduced during fine-tuning.

Towards training-time mitigations for alignment faking in RL

We construct a diverse array of model organisms of alignment faking and study mitigations that could be used during RL to decrease alignment faking and compliance gaps.

Open Source Replication of the Auditing Game Model Organism

We release an open source replication of the model organism from our previous auditing game paper.

Anthropic Fellows Program 2026

Apply now for our AI safety research fellowship.

Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs

We localize dangerous knowledge to a subset of model's parameters, so it can be easily removed after training.

Evaluating honesty and lie detection techniques on a diverse suite dishonest models

We explore techniques for honesty elicitation and lie detection on a diverse testbed of dishonest model organisms.

Strengthening Red Teams: A Modular Scaffold for Control Evaluations

We decompose sabotage into constituent skills and use synthetic simulations to strengthen attacks in complex environments.

Anthropic's Pilot Sabotage Risk Report

We release a report on the level of risk posed by our deployed models from emerging forms of misalignment, as of Summer 2025. We conclude that the level of risk is very low but not fully negligible.

Stress-testing model specs reveals character differences among language models

We generated 300,000+ queries testing value trade-offs in AI models from Anthropic, OpenAI, Google DeepMind, and xAI. Each model showed distinct value prioritization patterns, and we found thousands of cases of direct contradictions or interpretive ambiguities in model specifications.

Believe It or Not: How Deeply do LLMs Believe Implanted Facts?

We introduce a framework for validating knowledge editing techniques, finding that synthetic document fine-tuning sometimes—but not always—succeeds at implanting genuine beliefs.

Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment

For example, we train on demonstrations of test case hacking with training prompts that request hacks. This model does not learn to hack.

Training fails to elicit subtle reasoning in current language models

We investigated whether language models can reason about malicious tasks while evading detection, finding that monitoring both reasoning and outputs successfully prevents this in current models.

Petri: An open-source auditing tool to accelerate AI safety research

An open-source framework for automated alignment auditing that uses AI agents to create test environments for other models.

Findings from a Pilot Anthropic–OpenAI Alignment Evaluation Exercise

Anthropic and OpenAI conducted simultaneous alignment assessments of each others' models earlier this year. These are our findings.

Enhancing Model Safety through Pretraining Data Filtering

We experiment with removing harmful information about chemical, biological, radiological and nuclear (CBRN) weapons from our models' pretraining data.

Building and evaluating alignment auditing agents

In testing, our agents successfully uncover hidden goals, build safety evaluations, and surface concerning behaviors.

Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data

Training on aligned chains-of-thought generated by a misaligned model can induce misalignment.

Inverse Scaling in Test-Time Compute

We investigate inverse scaling when LLMs have access to more test-time compute.

Why Do Some Language Models Fake Alignment While Others Don't?

We investigate why in the scenarios where Claude 3 Opus fakes alignment many other language models don't do so.

Model-Internals Classifiers

We benchmark approaches to re-using LLM internals to make LLM monitoring more cost-effective.

Unsupervised Elicitation

We introduce a new unsupervised algorithm for eliciting skills from pretrained language models.

Reasoning Models Don't Always Say What They Think

Chen et al., 2025

We find that reasoning models don't always accurately verbalize their reasoning. This casts doubt on whether monitoring chains-of-thought (CoT) will be enough to reliably catch safety issues.

Publicly Releasing CoT Faithfulness Evaluations

We release the datasets used in our paper "Reasoning Models Don't Always Say What They Think" for public use.

Modifying LLM Beliefs with Synthetic Document Finetuning

We study whether we can modify the beliefs of LLMs and investigate whether doing so could decrease risk from advanced AI systems.

Putting up Bumpers

Even if we can't solve alignment, we can solve the problem of catching and fixing misalignment.

Alignment Faking Revisited: Improved Classifiers and Open Source Extensions

We present a replication and extension of an alignment faking model organism.

Auditing Language Models for Hidden Objectives

Marks,* Treutlein,* et al., 2025

We deliberately train a language model with a hidden objective and use it as a testbed for studying alignment audits.

Do reasoning models use their scratchpad like we do? Evidence from distilling paraphrases

We provide some evidence that Claude 3.7 Sonnet doesn't encode hidden reasoning in its scratchpad by showing that training it to use paraphrased versions of the scratchpads does not degrade performance.

Automated Researchers Can Subtly Sandbag

Current models can sandbag ML experiments and research decisions without being detected by zero-shot prompted monitors. Claude 3.7 is better at zero-shot sandbagging than Claude 3.5 (new).

Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Sharma,* Tong,* Mu,* Wei,* Kruthoff,* Goodfriend,* Ong,* Peng et al., 2025

We built a system of constitutional classifiers to prevent jailbreaks. A prototype version of our system withstood over 3,000 hours of expert red teaming with no universal jailbreaks found. Newer versions of our system also have minimal over-refusals and moderate run-time overhead.

Introducing Anthropic's Safeguards Research Team

We're launching a new research team focused on mitigating the post-deployment risks of AI systems.

Won't vs. Can't: Sandbagging-like Behavior from Claude Models

We find that, when Claude models are presented with a harmful version of a task they can otherwise perform, they sometimes claim they lack the ability to perform the task, rather than simply refusing to do it. We discuss the implications of this behavior for AI safety.

Monitoring Computer Use via Hierarchical Summarization

We introduce hierarchical summarization for monitoring and describe how we use it to protect Computer Use capabilities.

Forecasting Rare Language Model Behaviors

Jones*, Tong* et al., 2025

We forecast whether risks will occur after a model is deployed — using even very limited sets of test data.

Training on Documents about Reward Hacking Induces Reward Hacking

Does training on documents which discuss (but do not demonstrate) reward hacks affect a model's propensity to reward hack?

Recommendations for Technical AI Safety Research Directions

A collection of technical AI safety research problems that we'd like to see progress in.

Alignment Faking in Large Language Models

Greenblatt et al., 2024

We present experiments where Claude often pretends to have different views during training, while actually maintaining its original preferences.

How to Replicate and Extend our Alignment Faking Demo

We describe how to get started with experimenting with our demonstration of alignment faking, and present some ideas for future research.

A Toy Evaluation of Inference Code Tampering

We describe how highly capable LLMs might disable their monitor, and evaluate the ability of current LLMs to do so in a toy setting.

Introducing the Anthropic Fellows Program for AI Safety Research

We're launching the Anthropic Fellows Program for AI Safety Research, a pilot initiative designed to accelerate AI safety research and foster research talent.

Rapid Response: Mitigating LLM Jailbreaks with a Few Examples

Peng et al., 2024

Ensuring perfect jailbreak robustness is hard. We propose an alternative: adaptive techniques that rapidly block new classes of jailbreak as they’re detected.

Three Sketches of ASL-4 Safety Case Components

We sketch out three hypothetical arguments one could make to rule out misalignment risks in powerful near-future models capable of sabotage.

Sabotage Evaluations for Frontier Models

Benton et al., 2024

How well could AI models mislead us, or secretly sabotage tasks, if they were trying to? We describe a novel set of evaluations that test a model's capacity for sabotage.

Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

Denison et al., 2024

We show AI models can learn to hack their own reward system, by generalization from training in simpler settings.

Many-shot Jailbreaking

Anil et al., 2024

We study a long-context jailbreaking technique that is effective on most large language models, including those developed by Anthropic and many of our peers.

Simple Probes can Catch Sleeper Agents

We find that probing, a simple interpretability technique, can detect when backdoored "sleeper agent" models are about to behave dangerously, after they pretend to be safe in training.

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Hubinger et al., 2024

We train LLMs to act secretly malicious. We find that, despite our best efforts at alignment training, deception still slipped through.

Specific versus General Principles for Constitutional AI

Kundu et al., 2023

We test whether models can learn general ethical behaviors from only a single written principle, roughly stated as "do what's best for humanity."

Towards Understanding Sycophancy in Language Models

Sharma et al., 2023

AI assistants are trained to give responses that humans like. We show that these systems frequently produce 'sycophantic' responses that appeal to users but are inaccurate. Our analysis suggests human feedback contributes to this behavior.

Studying Large Language Model Generalization with Influence Functions

Grosse et al., 2023

We use influence functions to find training examples that contribute to a given model output.

Tracing Model Outputs to the Training Data

We present a summary of Studying Large Language Model Generalization with Influence Functions.

Measuring Faithfulness in Chain-of-Thought Reasoning

Lanham et al., 2023

We investigate hypotheses for how language models' Chain-of-Thought (CoT) reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it).

Question Decomposition Improves the Faithfulness of Model-Generated Reasoning

Radhakrishnan et al., 2023

To improve the faithfulness of Chain-of-Thought (CoT) reasoning, we have models generate their CoT by decomposing questions into subquestions.

Discovering Language Model Behaviors with Model-Written Evaluations

Perez et al., 2022

We develop an automated way to generate language model (LM) evaluations with LMs, significantly reducing the effort involved. We test LMs using >150 LM-written evaluations, uncovering novel LM behaviors.

Constitutional AI: Harmlessness from AI Feedback

Bai et al., 2022

We introduce Constitutional AI, allowing us to give language models explicit values determined by a constitution, rather than values determined implicitly via large-scale human feedback.

Measuring Progress on Scalable Oversight for Large Language Models

Bowman et al., 2022

We show how humans could use AI systems to better oversee other AI systems, and demonstrate some proof-of-concept results where a language model improves human performance at a task.

Language Models (Mostly) Know What They Know

Kadavath et al., 2022

We show that language models can evaluate whether what they say is true, and predict ahead of time whether they'll be able to answer questions correctly.

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Bai et al., 2022

We train a natural language assistant to be more helpful and harmless by using reinforcement learning from human feedback (RLHF).