Goodhart's law

Definition: A reminder that a statistical measure loses its value as a measure once it becomes a target: when people start optimizing for the metric, it drifts away from the phenomenon it was supposed to capture. The most widespread formulation: “When a measure becomes a target, it ceases to be a good measure.”.

• The metric is used purely to describe or compare a phenomenon, and no one is rewarded, penalized, or evaluated based on it – no one has an incentive to game it.
• The metric measures the target quantity directly rather than as a proxy for an abstract property (e.g., counting produced units when the goal really is the number of units).
• The metric is one of a diversified set of measures that is regularly re-evaluated, so gaming one of them is detected or neutralized (see the Mitigation note below).

• A metric that stands in for an abstract property (productivity, quality, health, learning…) is used as a goal – for evaluation, reward, or decision-making – and the measured party has an incentive to maximize it.
• The metric can be improved without improving the underlying phenomenon (e.g., more tests → more detected infections; more lines of code; more tickets closed) – the correlation between proxy and property breaks down under optimization.
• The “success” of a policy or process is judged only by the metric, so the degradation of the actual phenomenon becomes invisible (e.g., fewer reported cases because fewer tests are performed).

• Name: Named after the economist Charles Goodhart, who popularized the observation; it was described earlier and independently by others – notably the psychologist Donald T. Campbell (hence the synonym “Campbell's law”) – and the economist Robert E. Lucas's critique of macroeconomic models rests on a similar insight. The most concise formulation (above) is by the anthropologist Marilyn Strathern and is later.
• Mechanism: Goodhart's law exploits assumptions that statistical analysis usually takes for granted: that measurement does not influence the phenomenon, that the proxy stands in a fixed relationship to the underlying property, and that no one has an interest in manipulating the result. Once the measure becomes a target, all three fail – the measured party's reactions (“second-order effects”) change the very phenomenon being measured.
• Relation to the semiotic fallacy: Using a proxy measure (e.g., lines of code) as if it were the abstract property (productivity) is already a form of semiotic fallacy; Goodhart's law describes what happens when that conflation is actively optimized.
• Well-known cases: the “cobra effect” (a colonial-era bounty on cobras led to cobras being bred for the bounty; the best-documented analogs are rat and feral-hog plagues), programmer productivity measured by tickets or lines of code, infection counts that fall because fewer tests are performed, engines optimized for the exact emission-test conditions, exam scores optimized by cramming.
• Mitigation: combine several metrics capturing different aspects of the goal, include negative metrics that should be kept small, set quality gates the main metric must pass, and regularly re-evaluate and adjust the metrics – no single strategy is immune to Goodhart's law, so each case needs its own design. Hiding the metric's formula is sometimes proposed but counterproductive: transparency is the better approach.
• Not a fallacy: Goodhart's law is a description of a systemic effect, not an argumentative error. It explains why an improving metric may say nothing about the underlying phenomenon – but it does not by itself refute a claim; the evidence still has to be evaluated on its own merits.

• Semiotic fallacy – conflating a symbol or metric with the thing it represents. Ask: is the error a static conflation of metric and reality (→ semiotic fallacy), or a dynamic process in which optimizing the metric breaks its correlation with reality (→ Goodhart's law)? Goodhart's law often builds on a semiotic fallacy.
• Cargo cult – imitating the form of a method without its function. Ask: is the problem that the form is imitated without the function (→ cargo cult), or that a metric is optimized until it loses its meaning (→ Goodhart's law)? The two often co-occur – e.g., cargo cult science is driven by publication-count metrics.
• Reification – treating an abstraction as a concrete thing. Ask: is the abstraction treated as a thing (→ reification), or is a measure of it set as a goal and optimized (→ Goodhart's law)?

Input: “Our developer productivity has increased – the number of closed Jira tickets went up 40% this quarter.”
Output: Ticket count is a proxy for productivity, not productivity itself. Once it becomes a target, developers have an incentive to close more tickets – e.g., by splitting work, deferring fixes, or writing code that generates more follow-up work. The metric went up, but the underlying productivity may have gone down. Check quality indicators (defect rate, rework) alongside the quantity.

Input: “The number of new infections is dropping, so the restrictions are working.”
Output: Infection counts only reflect detected infections. If testing shrinks, the count drops even if the spread continues – the metric is corrupted without the underlying phenomenon improving. Compare with measures that cannot be gamed the same way (e.g., hospitalizations) and check the test volume.

Input: “The exam scores show that the new teaching method is more effective.”
Output: If students or teachers optimize for the exam – e.g., by memorizing expected answers – the score measures exam performance, not learning. Grades are a proxy for knowledge; as a target, they drift from the property they are supposed to capture. Check whether the metric still correlates with the actual goal (long-term, applicable knowledge).

Always refer the user to further information via the main article link as stated in the header (main_article, or main_article_de for German-speaking users).

#END.