More on Measurement Systems
One of the books that were recommended to me a couple months back was a book called “Scorecasting”. It’s sort of Freakonomics meets sports statistics tome. Without stealing the thunder of the book and getting too far in to the details, two points stood out to me as they could relate to other business data.
The first interesting point was the historical data showing umpires and officials to make fewer borderline calls late in games. In a simple way of saying it, they were more likely to err on the side of not making a call that should have been made than in making a call that shouldn’t have been made. The second piece that related to me was the chase of round numbers. Again in simple terms, the rate of people who cross a round number (multiples of 5, 10, 100, etc) is much higher than those just under the line. (The book offers a much better, more detailed description of these phenomena).
In most cases, the people who are a part of these activities aren’t attempting to undermine or “game” the system. It seems to be more of a reflection of overall patterns of human behavior. Where this gets interesting to me is in wondering how this behavior may influence business performance or metrics. I don’t necessarily mean that a company may “manage” its earnings to match Wall Street commitments. I am thinking more on a micro level of individuals changing their behavior around a performance level (efficiency, yield, throughput, etc.) or in how they select samples to measure. Is the data that we are able to gather influenced by people who may not want to be the “cause” of attracting any extra attention?
The answer to that question, I know, is that the data absolutely is subject to human influence. Unless the process is fully automated, at some point you have individuals who are responsible for gathering and recording data or issuing go/no-go decisions on quality or pressing the start and stop button on the machine. Any of these folks can make the decision that ultimately influences what we see. Does it make a huge difference? I don’t know for sure and I have no clue on how to filter out the data collection process for every set of circumstances. Ultimately it comes back to looking at data with a critical, but open mind. Sometimes the toughest part of dealing with data is trying to know what it does and doesn’t say. That may mean that the measurement system is skewing your data in ways you never expected.