I love me some data. As a result, I have a penchant for metrics and measurements of everything regardless of the need or if its easily quantifiable. However, I have been using metrics for long enough, and watching them be used by organizations and programs, to know that there are obvious faults in measuring as a default answer to a complex problem.
I just finished the book, The Tyranny of Metrics by Jerry Z. Muller, a historian at Catholic University of America in Washington, D.C. His interest in the subject, which is not his traditional forte, came from his experience as a faculty member at that university and the growing use of performance metrics in complex decision making and performance evaluation. His book discusses the perverse incentives that can crop up in cases where performance metrics are used poorly. Most corrupting in Muller’s mind is the use of metrics in pay-for-performance schemes. These set the stage for a high-stakes game of gaming the metrics.
The Problem with Metrics
Muller identifies the following problems with bad metrics.
1. Goal Displacement. They can quickly displace the overarching goals of the organization and create an atmosphere of using the organization to drive the metrics and not the other way around. When a person’s success (and even their job) is hanging on a few measures, then they will find a way to game the measures at the expense of the big picture.
2. Short Term Thinking. Short term thinking abounds with bad measures. Meeting a target for this month is important, but the can create blinders to the long-term concerns of the organization.
3. Cost of Compliance and Diminishing Utility. Measures can be a huge boon to an organization in the beginning, but the more measures added, the greater the cost in employees’ time to collect, input, and discuss the metrics. Over time, additional metrics don’t add as much value, and can produce diminishing utility, but require the same cost commitments as more useful early metrics. It can eventually hurt the organization’s bottom line if it goes to far.
4. Perverse Incentives. Bad metrics can reward employees for improvements in measures that were not their doing. Or the opposite – employees can be punished for a declining measure. A lot of factors could play into metric performance, and as such, most of the reason for random fluctuation may be out of the control of the person responsible for the metric. Because of this problem, measures can discourage risk-taking and innovation; something a failing organization may need the most. And if the organization evaluates individuals based on personalized performance, don’t expect a great deal of voluntary cooperation and camaraderie.
What Can be Done?
Muller Identifies a number of questions that should be asked by anyone interested in implementing metrics in their organization.
1. What kind of information am I measuring? Be very specific about the thing you really want to measure. And remember that the measure become less reliable the more abstract the subject.
2. How useful is this information? Is all this data collection really worth it, or am I measuring for the sake of having more data?
3. How useful are more measures? What is the worst-case scenario (financially and otherwise) of not having these metrics?
4. Who gets to see the information? Certain measures will make more impact on different groups, and those groups will react differently. The public, for example, reacts differently than the Board of Directors.
5. What is the cost of gathering this data? This is one I have pickup over the years. Organizations I have worked with have wanted to look for data to aid in decision-making. Many times, I tell them that the data is out there, but getting it will be a challenge; an expensive and time-intensive challenge. Ask yourself if it’s worth it.
6. Who wants the metrics? Who will manage them? Who will define success? Figuring this part out is critical to ensuring this metrics doesn’t become just another rule from the top that is either a burden, or completely ignored.
7. How can this measure be misused, misunderstood, and gamed? If people don’t want rules, they will first find a way to work around them. This is true of metrics as well. Before implementing new measures, take some time to try and “break” the measure.
Ultimately, metrics can’t replace solid decision-making. They are a supporting tool for managers, supervisors, executives, elected officials; that can be the critical factor in deciding the direction or an organization. But metrics should never decide for the people in the organization. Its quite easy for rules to dominate an organization and stifle creativity and success. Metrics should not be made into data-driven rules that stifle and organization.