How Anthropic's watermarking (maybe) works (probably): It changes patterns
LLMs are all about patterns. They create content by picking from a list of likely words. They determine what is “likely” based on all the language they’ve got in the model. Watermarking plays with that list, making some words more or less likely than the LLM would otherwise. Then they can check content for matching patterns. If the text they’re checking has the same modified word patterns, it’s probably generated by Claude. Probably.
For this demo: Drag the slider and watch how some words get more or less probable. Click “new secret key” to change the words the watermark favors.
Not watermarked
offsubtlemoderatestrong
Watermarkedbias δ = 0.0
key k-7f3a91
Generated passage
Green rate
—
50% without a watermark. Higher number = more likely Claude-generated.
z-score
—
across this passage's 0 words. Higher = more likely intentional drift, and Claude generation
Text needed
—
words to reach z = 4 (probable watermarking) at this rate
Flipped
—
word choices moved off the top pick
Preference
—
avg. unbiased probability of chosen words. Lower = the model was less likely to choose these words without watermarking
Every choice the model makes
unbiased probabilityafter watermark biasG green list R red listfaded word = near-certain, resists the bias
Does Ian have any clue?! Nope. This is based on Anthropic’s announcement and the paper they cited. I’m not that smart. Claude helped me generate this demo, by the way.