When Metrics Become Targets, Trouble Surely Follows

Charles Goodhart, chief economic adviser to the Bank of England for many years, is credited with formulating the law that “As soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.” This has been shortened and generalized to “When a measure becomes a target, it ceases to be a good measure.”

Beverly L Hall

Dr. Beverly L. Hall Superintendent of the Atalanta Schools District 1999-2010

There was a striking illustration of this in last Saturday’s New York Times, which reported on the indictment for corruption of 35 Atlanta educators by a grand jury.  The indictment reads like a mobster’s rap sheet: racketeering under the RICO statute, false statements, and theft. The charges stem from a period when the Atlanta schools made apparently extraordinary progress in their performance on the Criterion-Referenced Competency Tests (CRTC), Georgia’s primary measure of academic ability through the eighth grade. Although statistically the improvements had seemed “too good to be true” and there had been rumours of cheating, no evidence could be found. School administrators attributed the results to “talented” teachers, hard work, strong teaching programs and the “absolutely no-excuses” attitude of Dr. Beverly L. Hall, the District Superintendent.

“If It Can’t Be Measured, It Can’t Be Managed”

Bush NCLB 2001

George Bush signing the No Child Left Behind Act in 2001

Beverly Hall came to the Atlanta School District in 1999, after what had apparently been a stellar rise to prominence as an educator. Standardized testing was introduced in 2001 by George W. Bush in the No Child Left Behind Act (NCLB): “An act to close the achievement gap with accountability, flexibility, and choice, so that no child is left behind.” Furthermore the schools had to rely on “research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs.”

A bit of context helps one understand why both Republicans and Democrats passed this legislation so readily. In business in 2001 the Shareholder Value (SHV) movement was at its peak. It seemed back then that the rigorous measurement of corporate performance by focusing on returns to shareholder was producing excellent results, especially when coupled with pay for performance. The business-minded Bush administration was eager to introduce similar methods into the education system and standardized testing coupled with demanding goals was assumed to be the way to go.

The autocratic Dr. Hall seems to have taken to the new system like a duck to water. She gave school principals three years to meet demanding “stretch goals” over and above those of CRTC. If they did not meet them they were fired – “excuses for not meeting targets would not be tolerated”. She replaced 90% of the teachers in the district in the ten years she was superintendent. Those who complained about the process were characterized as weak teachers, who were afraid to have their teaching performances assessed. Those who met their goals were rewarded with tenure and performance payments.

It did not take long for the improvements in academic performance to appear. Students began to show year-on-year improvements in math and reading ability that were quite outstanding. So outstanding, in fact, that they defied the statistical probabilities, with some scores many standard deviations above what was likely. However, all efforts to penetrate the administrative veil over the schools failed. Although she held herself aloof from parents, teachers and principals, Beverly Hall had developed close ties with Atlanta’s elite. Here was an African-American school administrator producing outstanding results in a largely African-American school district. The schools were attracting grants and the city itself was attracting corporations, drawn by the lure of an apparently improving school system with a progressive administration. What was there not to like about that?

If It Can Be Measured, It Can Be Gamed

Despite the failure of various blue-ribbon commissions to get a true picture of the situation, the one revealed by the Georgia Bureau of Investigation in July 2011 was a devastating. Spearheaded by two special prosecutors with subpoena powers and fifty investigators at their disposal, it revealed systematic cheating in 44 out of 56 schools. 178 teachers and principals fixed incorrect answers from the students. Sometime this involved putting a select group of teachers in a locked room during testing week to erase wrong answers and replace them with the correct ones. One principal wore gloves, so as not to leave her fingerprints on the answer sheet! The apparent improvement in academic results was bogus. Dr. Hall has said she had no knowledge of the schemes.

In the business world we now know the longer-term results of the Shareholder Value movement – it too has led to the widespread gaming of the measurement system in order to generate substantial benefits to the gamers. Indeed there is now considerable doubt whether the SHV criterion, like those of standardized testing in America’s schools, are really measuring what we want, rather than just what’s easy to measure.  Are the results measured by SHV sustainable? Does an intensive focus on financial outcomes not drive out employee engagement and innovation? Similarly, does the ability to pass a battery of standardized academic tests mean that one is educated and able to contribute, not just to the economy, but also to the community?

A Faith-based Initiative

Consider the Finnish school system which, based on OECD surveys, has consistently produced graduates that perform at excellent levels. How do they do it? Not by focusing on test performance. There are no standardized tests except for a National Matric exam. They assign less homework than their East Asian rivals and do not depend on rote learning. Every child receives a report card every semester with individual grades from each teacher. The teachers, who have to have a master’s degree, are trained to the highest standards. If a teacher is bad, then it is the responsibility of the principal to weed them out.

Instead of academic excellence, the Finnish schools focus on leveling out social inequality.  They are not interested in training an elite but in building an entire community, giving everyone the same learning opportunity, no matter what their background. The focus is not on teaching but on learning how to learn and building contexts that encourage learning. In that process they achieve academic excellence. Curiously enough this is exactly what one finds in the corporate world. Those who produce the best returns to their shareholders don’t focus on SHV; they focus on building cultures. Rosabeth Moss Kanter describes the six facts of this “institutional logic” as; a common purpose, longer-term view, emotional engagement, community-building, innovation and self-organization. Like the Finns, they treat their people as true professionals who they can trust to use their judgement in ethical ways.

Results like these highlight the dangers of mindlessly transferring scientific concepts from the natural sciences to complex human systems. People, unlike objects, react and respond to the ways in which they are treated; often in perverse ways. One wonders if the American confidence in macho-management styles, based only on short-term, easily-measured outcomes, is not just as faith-based as any religion.

This entry was posted in Change, General, Leadership and tagged , , , , , , , , , , , , , , , , . Bookmark the permalink.

4 Responses to When Metrics Become Targets, Trouble Surely Follows

  1. warren says:

    The documentary I watched, focused on South Korea &
    Finland, showing the difference in teaching styles,
    yet both styles resulting in incredible learning,
    apparently. Cannot find documentary, but here is
    a link to what I am talking about if interested:
    http://www.bbc.co.uk/news/world-asia-china-20521422

    Here is what I think (not based on any research study)
    I have asked hundreds of people, high school educated
    or above………”What is the value of PI”. Most say,
    3.14 or 3.1459, or 3.14 never ending. Then I ask them
    to explain it verbally or draw it on paper for the
    explanation. Most of the time, they cannot. They
    mostly remember learning about it in high school, but
    cannot explain it. Why?

    My theory. Because most schooling, and University as
    well, is based on MEMORIZATION. It is NOT based on
    UNDERSTANDING. The concept is, “Here, memorize the
    formula, and plug in the numbers”. This, in my opinion
    is a JOKE.

    For me, when I understand something, which takes me a
    long time and effort to do so, I usually remember if
    ‘for life’. When I memorize something, I usually forget
    most of it, or part of it. And UNDERSTANDING, is what I
    believe education, and every education system should be
    about. NOT memorizing.

    Warren 🙂

    • David says:

      Thanks Warren,
      You make an excellent point. Knowledge that is taught in that way is inert and cannot be mobilized when needed. Its much better to help learners to fit their knowledge and experience into patterns, so that they can call upon it easily when they encounter a familiar pattern.

    • Geoff says:

      Warren, it has become highly fashionable to criticize memorization – but the simple reality is that memorization is a critical link in the chain that has allowed our species to flourish.

      A better statement is ‘memorization without understanding is a joke’.

      To analyze ever more complex situations, problems and environments, the human brain must draw upon the knowledge that is captured within. You can call it memorization, or remembering, or thinking. And there is nothing wrong with consulting reference material to refresh a memory – but real, independent analysis does not occur without understanding AND memory.

      Remembering things is a neurocognitive skill that we should all hope to possess in great heaps. If you can’t memorize, you certainly can’t learn to read or write.

      I for one have absolutely no problem with requiring students to memorize multiplication tables, important dates and events, poems, and, yes, formulae. Of course, these things should be taught within a larger context to build understanding – and understanding occurs when you create links to information that already exists within the memory.

      But if we continue to advance this idea that memorization is unnecessary, we are doing our children a tremendous disservice as they face an increasingly complex world.

      • warren says:

        Geoff,

        My understanding of the human mind is limited to my
        own experience, my undergrad classes, and now EMBA
        classes. I am not a Medical Doctor, nor do I have
        teaching experience.

        Only stating my observations. I may be “out to lunch” 🙂

        However, respectfully, I do not follow “fashion”. I was not
        aware the critique of memorization was the latest flavour of
        the day or “fashion” or “fad”.

        Warren 🙂