Tech paper

Constitutional AI: Harmlessness from AI Feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. · 2022

Read this as

New to this. Just the gist and why it matters.

How deep

The gist, about 2 minutes.

In one line

The short of it

Teach an AI to be harmless by giving it a short written set of principles (a constitution) and having it critique and revise its own answers, instead of relying on humans to label every harmful response.

Why it exists

The problem it tackles

The usual way to make a model safer, reinforcement learning from human feedback, needs people to read and rate huge numbers of harmful outputs. That is slow, costly, and hard on the human labelers. The authors wanted safety with far less human labeling and more transparency about the rules.

In plain words

The key idea

Write down a set of plain-language principles. Then have the model look at its own answer, criticize it against those principles, and rewrite it to be better. The model's own feedback, guided by the constitution, replaces most of the human safety labeling.

Think of it like: Like giving a new employee a written code of conduct and asking them to proofread their own emails against it before sending, instead of a manager rewriting every message by hand.

What they did

The method

Two stages: first the model revises its own responses using written principles (supervised), then it trains a preference model from AI-generated comparisons (reinforcement learning from AI feedback).

What they found

The results

Models trained this way were both harmless and less evasive, with far less human harm-labeling than standard RLHF.

Does it hold up?

The honest part. What is solid, and where to be careful.

Harmlessness is evaluated on the authors' own preference judgments, which are subjective.
A model critiquing itself can share the same blind spots the constitution does not name.
Results depend heavily on how the principles are written and which values they encode.

⚠ The constitution encodes specific value choices; 'harmless' is not neutral and depends on who writes the rules.

So what, for you

If you are a curious newcomer

This is one way modern AI assistants learn to say no gracefully. Instead of humans labeling every bad answer, the AI checks its own answers against a short list of written rules.