When Metrics Become Targets, Trouble Surely Follows

Charles Goodhart, chief economic adviser to the Bank of England for many years, is credited with formulating the law that “As soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.” This has been shortened and generalized to “When a measure becomes a target, it ceases to be a good measure.”

Beverly L Hall

Dr. Beverly L. Hall Superintendent of the Atalanta Schools District 1999-2010

There was a striking illustration of this in last Saturday’s New York Times, which reported on the indictment for corruption of 35 Atlanta educators by a grand jury.  The indictment reads like a mobster’s rap sheet: racketeering under the RICO statute, false statements, and theft. The charges stem from a period when the Atlanta schools made apparently extraordinary progress in their performance on the Criterion-Referenced Competency Tests (CRTC), Georgia’s primary measure of academic ability through the eighth grade. Although statistically the improvements had seemed “too good to be true” and there had been rumours of cheating, no evidence could be found. School administrators attributed the results to “talented” teachers, hard work, strong teaching programs and the “absolutely no-excuses” attitude of Dr. Beverly L. Hall, the District Superintendent.

“If It Can’t Be Measured, It Can’t Be Managed”

Bush NCLB 2001

George Bush signing the No Child Left Behind Act in 2001

Beverly Hall came to the Atlanta School District in 1999, after what had apparently been a stellar rise to prominence as an educator. Standardized testing was introduced in 2001 by George W. Bush in the No Child Left Behind Act (NCLB): “An act to close the achievement gap with accountability, flexibility, and choice, so that no child is left behind.” Furthermore the schools had to rely on “research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs.”

A bit of context helps one understand why both Republicans and Democrats passed this legislation so readily. In business in 2001 the Shareholder Value (SHV) movement was at its peak. It seemed back then that the rigorous measurement of corporate performance by focusing on returns to shareholder was producing excellent results, especially when coupled with pay for performance. The business-minded Bush administration was eager to introduce similar methods into the education system and standardized testing coupled with demanding goals was assumed to be the way to go.

The autocratic Dr. Hall seems to have taken to the new system like a duck to water. She gave school principals three years to meet demanding “stretch goals” over and above those of CRTC. If they did not meet them they were fired – “excuses for not meeting targets would not be tolerated”. She replaced 90% of the teachers in the district in the ten years she was superintendent. Those who complained about the process were characterized as weak teachers, who were afraid to have their teaching performances assessed. Those who met their goals were rewarded with tenure and performance payments.

It did not take long for the improvements in academic performance to appear. Students began to show year-on-year improvements in math and reading ability that were quite outstanding. So outstanding, in fact, that they defied the statistical probabilities, with some scores many standard deviations above what was likely. However, all efforts to penetrate the administrative veil over the schools failed. Although she held herself aloof from parents, teachers and principals, Beverly Hall had developed close ties with Atlanta’s elite. Here was an African-American school administrator producing outstanding results in a largely African-American school district. The schools were attracting grants and the city itself was attracting corporations, drawn by the lure of an apparently improving school system with a progressive administration. What was there not to like about that?

If It Can Be Measured, It Can Be Gamed

Despite the failure of various blue-ribbon commissions to get a true picture of the situation, the one revealed by the Georgia Bureau of Investigation in July 2011 was a devastating. Spearheaded by two special prosecutors with subpoena powers and fifty investigators at their disposal, it revealed systematic cheating in 44 out of 56 schools. 178 teachers and principals fixed incorrect answers from the students. Sometime this involved putting a select group of teachers in a locked room during testing week to erase wrong answers and replace them with the correct ones. One principal wore gloves, so as not to leave her fingerprints on the answer sheet! The apparent improvement in academic results was bogus. Dr. Hall has said she had no knowledge of the schemes.

In the business world we now know the longer-term results of the Shareholder Value movement – it too has led to the widespread gaming of the measurement system in order to generate substantial benefits to the gamers. Indeed there is now considerable doubt whether the SHV criterion, like those of standardized testing in America’s schools, are really measuring what we want, rather than just what’s easy to measure.  Are the results measured by SHV sustainable? Does an intensive focus on financial outcomes not drive out employee engagement and innovation? Similarly, does the ability to pass a battery of standardized academic tests mean that one is educated and able to contribute, not just to the economy, but also to the community?

A Faith-based Initiative

Consider the Finnish school system which, based on OECD surveys, has consistently produced graduates that perform at excellent levels. How do they do it? Not by focusing on test performance. There are no standardized tests except for a National Matric exam. They assign less homework than their East Asian rivals and do not depend on rote learning. Every child receives a report card every semester with individual grades from each teacher. The teachers, who have to have a master’s degree, are trained to the highest standards. If a teacher is bad, then it is the responsibility of the principal to weed them out.

Instead of academic excellence, the Finnish schools focus on leveling out social inequality.  They are not interested in training an elite but in building an entire community, giving everyone the same learning opportunity, no matter what their background. The focus is not on teaching but on learning how to learn and building contexts that encourage learning. In that process they achieve academic excellence. Curiously enough this is exactly what one finds in the corporate world. Those who produce the best returns to their shareholders don’t focus on SHV; they focus on building cultures. Rosabeth Moss Kanter describes the six facts of this “institutional logic” as; a common purpose, longer-term view, emotional engagement, community-building, innovation and self-organization. Like the Finns, they treat their people as true professionals who they can trust to use their judgement in ethical ways.

Results like these highlight the dangers of mindlessly transferring scientific concepts from the natural sciences to complex human systems. People, unlike objects, react and respond to the ways in which they are treated; often in perverse ways. One wonders if the American confidence in macho-management styles, based only on short-term, easily-measured outcomes, is not just as faith-based as any religion.

This entry was posted in Change, General, Leadership and tagged , , , , , , , , , , , , , , , , . Bookmark the permalink.

4 Responses to When Metrics Become Targets, Trouble Surely Follows