Lies, damn lies, and statistics is a phrase usually attributed to Lord Disraeli. With the profusion of rating and ranking schemes, the time has come to bring it up to date, as the headline proclaims. Wal-Mart has recently announced its plans to rate all of the hundreds of thousands of products it sells. We have ratings of plain colored colleges, green colleges, corporations in many shapes (Fortune 500, Dow Jones Sustainability Index. . .) and now the [Newsweek list of the 500 greenest big U. S. corporations](http://greenrankings.newsweek.com/).
The idea of ranking things of all sorts is an old and frequently useful one. The truthfulness and utility of each scheme is, however, as the devil is reputed to have said, in the details. The outcome of any rating system that combines more than a single factor depends entirely, on the arbitrary choice of weights used in combining multiple factors. To be perfectly honest, Newsweek should have labeled their listing as the “Greenest Big Companies in America, based on combining three independent organizations scoring of environment impact (45%), green policies (45%), and reputation (10%), normalized to 100 as the top score. . . They did provide enough information in the body of the article that a careful reader could try to interpret the ratings armed with more detailed data. But few people are such careful readers and so the ratings take on a life of their own.
I was going to argue that these ratings have little to do with what green means, but I have a much more interesting subject to talk about. I was curious to determine how the ratings were normalized so that the highest one is given a score of 100. So I did exactly what they said to determine the combined score. I multiplied the three normalized scores by the respective weighting factors and added them up. The results were inconsistent with the printed table. Dell comes out first, and HP drops to number 3 or 4. Intel shouldn’t be anywhere in the top group. I did go to their website to check the methodology, but, unless I have erred in following what I think they say, I cannot come up with the same set of figures. Either Newsweek hasn’t explained their ranking scheme clearly or they have tons of egg on their face.
> The ENVIRONMENTAL IMPACT SCORE, based on data compiled by Trucost, is a comprehensive and standardized quantitative performance measurement that captures the total cost of all environmental impacts of a corporation’s global operations. Over 700 variables are summarized in the EIS. This figure is normalized against a company’s annual revenues, so that companies of all sizes and industries can be compared.
>
I looked at the details to find how they collected data on the more than 700 metrics, but could not find any information. The method used to compare companies by normalizing them by revenue is completely arbitrary as it assumes that the aggregate impact of any company is proportional to their size.
There is a further serious error in reporting the result to two decimal places. The components are merely estimates and have little connection to reality. A. N. Whitehead called the practice of reporting nature more precisely than nature really is the “Fallacy of Misplaced Concreteness.” Even if the numbers were algebraically correct, which they do not seem to be, the real difference between a score or 98.87 and 98.56 is of no meaningful consequence, except that it will allow some firm to boast about being Number One or Two as if that meant something special.
Special to whom. Maybe to the stockholders or some regulatory agency or a prospective employee or. . . The party with the most interest in the matter is Mother Nature, and her question might be, “Do these numbers have anything to do with how you are treating me?” Yes, in some small way, but not in any case directly correlated with how badly our environmental world is faring. The environment is affected by the actions we take that interact with it, not by our good intentions or what people say about us. Green policies are just writing on a piece of paper; they mean little until enacted. My MIT students found more than 15 years ago that companies that sign on to voluntary, industry-based regulatory programs with mandated performance targets show great disparities in how they perform under the mandates. Similarly, reputation can certainly be earned, but is also created by careful public relations. Only measures directly related to environmental interactions are meaningful in a static index. Reputation and policies might be helpful in guessing who is likely to rank higher in the future.
The commodification of a complex relationship between what firms do and what happens to the world as a consequence leads to misunderstanding and mistakes, and to the appearance of unintended consequences. Further, I could find nowhere any indication that the environmental impact score was based on a lifecycle analysis. This omission is very serious as it may miss the biggest impact that a corporation produces. The major impact of making and selling cars comes after the car is driven off the dealer’s lot. The biggest impact from Target or Wal-Mart is derived from what goes out the front door.
Those who argue that arbitrary rating schemes like this, and say Wal-mart’s Sustainability Index, are useful because they incentivise firms to strive to be Number One. This sounds good, but if the numbers don’t relate to the state of the world and how it is changing, the scores become a target more for the public relations staff and not for the people in the firm that understand that what is important may not raise the score.