When the Metric Becomes the Target: Goodhart's Law in Software

A support team gets a new goal for the quarter: close more tickets. The count goes on a screen, and the best closers get a bonus. It works. Closed tickets rise every week. Then someone notices the same customers coming back with the same problems. Tickets are being closed when someone replies, not when the problem is solved, and a customer who writes back starts a fresh ticket, which can be closed again.

Nobody lied. Every closed ticket really was closed. The number simply stopped describing the thing it was chosen to describe.

Two versions of the same law

The idea comes from economics, not software. In a paper for a Reserve Bank of Australia conference in July 1975, on the problems of monetary management in Britain, the economist Charles Goodhart wrote: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

That is a statement about relationships. A number that tracked something while nobody steered by it stops tracking it once people are rewarded for moving it. Nobody has to cheat for this to happen. People respond to what is rewarded, and their response changes the relationship. Before the bonus, a person who closed many tickets was usually a person who solved many problems. Once closed tickets carry a bonus, a high count tells you only that the bonus was earned.

The version most people quote is shorter, and is usually credited to the anthropologist Marilyn Strathern, in a 1997 paper on audit in British universities: once a measure becomes a target, it stops being a good measure. That version is about the measure. Goodhart's is about the link between the measure and the world. Software teams meet both: people game the measure, and even when nobody games it, the link to the outcome weakens.

What it looks like at scale

The clearest large case I know is Wells Fargo. In September 2016 the US Consumer Financial Protection Bureau fined the bank $100 million. In the CFPB's words: "Spurred by sales targets and compensation incentives, employees boosted sales figures by covertly opening accounts". By the bank's own analysis, employees opened more than two million deposit and credit card accounts that customers may not have authorised. Gaming a target is not cheap: the bank would also pay $35 million to the Office of the Comptroller of the Currency and another $50 million to the City and County of Los Angeles.

The sales figures went up. By then they were measuring something else. The CFPB's director, Richard Cordray, drew the lesson for the whole industry: "financial incentive programs, if not monitored carefully, carry serious risks".

The software versions

Software has its own Wells Fargo in miniature, usually with lower stakes and more spreadsheets. Each of these numbers is useful as a signal and harmful as a target:

Source: my own list, not drawn from a study.

Martin Fowler made the coverage point in a 2012 note: "If you make a certain level of coverage a target, people will try to attain it." At the extreme, he adds, you get tests with no assertions at all. Mutation testing is a good counter: it changes the code on purpose, flipping a comparison or deleting a line, and checks that some test fails. A suite that asserts nothing fails that check at once. Features shipped fails in the same way. Shipping is not success, and a count of releases says nothing about whose behaviour changed.

Individual numbers do a quieter kind of damage. Rank engineers by tickets closed, and the hard tickets drift to the one person willing to take them while everyone else picks the quick ones. That is how a team ends up with a bus factor of one, and the leaderboard looks healthy the whole time.

Pair every target with a counter-metric

Cordray's "monitored carefully" is the right instinct. The answer is not to stop measuring. It is to stop trusting one number on its own. Four habits help:

1. Pair each target with a measure that moves the wrong way when the first is gamed. Closed tickets with the reopen rate. Velocity with escaped defects. If both improve, something real probably happened.
2. Keep the outcome in view. The ticket count stood for customers whose problems were solved. Ask them, or watch whether they come back with the same problem.
3. Read the cases behind the numbers. Once a week, read ten closed tickets from start to finish. Counter-metrics can be gamed too: make the reopen rate a target and the team will ask customers to open a new ticket instead. Counting repeat contact from the same customer on the same topic closes that gap, and reading the tickets catches the next one.
4. Keep individual numbers out of pay. Team numbers on a shared dashboard start conversations. The same numbers in a performance review start negotiations.

Not every target goes bad. A target is safer when the number is the outcome itself, such as the share of customers whose payment succeeds, and when the only way to move it is to do the work. The further a number sits from the outcome, the more room there is to move one without the other.

The same caution applies to numbers that exist to support a decision. Cost of delay ranks work by what each week of waiting costs, divided by how long it takes. It works because the people estimating it have no reason to inflate it. Tie a team's bonus to its scores and the scores will rise.

A target is an instruction, and people follow instructions to the letter. Before you set one, ask what you would see if someone hit it without doing the work it stood for. Then measure that as well.

Written by Eslam Hamed, a Technical Lead and AI Solutions Architect working on enterprise software delivery, from pre-sale to handoff.