Don’t Build Weapons of Math Destruction

Face Validity

Upon reviewing a model with my first manager at Arthur D. Little, I learned an important lesson. He taught me that while it is important to demonstrate that a model has scientific validity, it is equally important to have “face validity.”

Translation: To be useful, a model needs to be statistically sound in representing the real world. But, it must also make sense to managers by incorporating factors that make sense to them.

I am astounded to find that many of the Internet-based Big Data models today only have face validity; apparently, developers don’t care about scientific validity. They believe that if a model’s factors seem to relate to dependent factors, those are sufficient.

For business models, face validity is necessary, but not sufficient. Indeed, face validity alone might be potentially dangerous. I gained this insight by reading “Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy,” a book by Dr. Cathy O’Neil.

What is a weapon of math destruction?

Dr. O’Neil, a former quantitative analyst at a financial hedge fund, experienced first-hand the damage financial models did leading up to and following the mortgage market meltdown of the past decade.

They helped the collapse of financial institutions. She became disillusioned with mathematical models that affect society. Or, as one description of the book puts it: “A former Wall Street quant sounds an alarm on the mathematical models that pervade modern life - and threaten to rip apart our social fabric.”

Her premise is that the vast amount of Big Data is used in ways that are: “1) opaque; 2) unquestioned; and 3) unaccountable.” In simpler terms: 1) the detailed data is not transparent to a person affected by its decision-making; 2) the use of the data is beyond reproach in the mind of the modeler; and 3) modelers refuse to defend the model other than to say “it is what it says.” It’s akin to using the court of public opinion rather than the law to assess whether someone has committed a crime.

In addition, she states that models often result in behavior that has “vicious, self-reinforcing feedback loops,” whereby things get worse for those affected—especially minorities and the poor. While there are many examples in the book, I’d like to focus on two.

Illustrative WMDs

U.S. News & World Report (USNWR) was founded in 1933. Fifty years later, in 1983, it was a second-rate publication, lagging Time and Newsweek, which were then the industry leaders. To set itself apart, the magazine decided to start a service ranking colleges and universities with the intent to help young people make their first big decision.

This was a game-changer for USNWR because today it markets itself on its website as “a multi-platform publisher of news and information, which includes usnews.com and annual print and e-book versions of its authoritative rankings of Best Colleges, Best Graduate Schools and Best Hospitals.”

The initial rankings involved weighted factors that journalists (not educators) felt were reasonable and could be quantified. Essentially, all were face-valid factors, and not necessarily ones proven to be related to educational quality. Dr. O’Neill contends the rankings were too successful, and its model over time was a WMD to education.

The rankings started a “race-to-the-top” among universities, whereby they did everything to raise rankings. Because costs were not included among the ranking factors, this incentivized colleges to hire expensive faculty, beef up athletic programs, construct new luxurious dorms and enhance dining menus.

Some colleges even resorted to cheating their way to the top, fabricating the numbers they reported to USNWR. Parents and students spent a lot of money on college admission planners to get into the top-ranked colleges. In addition, some international students cheated on standardized exams. The author contends that this competition contributed to today’s exorbitant tuition costs and student loans, leaving too many minority and poor students saddled with debt they will never pay back.

Another illustration has to do with credit scoring activities of marketers and others using Big Data.

The author contends that the well-known FICO scores, used by credit card and other loan providers, are not WMDs. FICO is regulated and transparent to borrowers. A FICO score is based on the financial history of a borrower - not those similar to them. However, these scores, while valid for creditworthiness, are often used for hiring; they wrongly assume that a high score means a better worker. This can lead to wealthy applicants getting jobs over poor applicants (who arguably need them more).

Meanwhile, “e-scores” developed for marketing purposes include factors in addition to the FICO ones. The biggest offender is a borrower’s zip code because average loan-default rates vary significantly by zip code. An e-score is a WMD because it assumes that if my neighbors default on loans, I have a high chance of doing so. Thus, poorer loan applicants may not get loans, or if they do, they are subject to paying higher interest rates. A self-fulfilling prophecy, this increases the chance of these applicants defaulting because payments were set too high.

WMD models in business

The examples I just discussed can wreak havoc on societies, and in particular on minorities and the poor. But, do we have WMD models in the business world? I would say there aren’t many, because in business the focus is on attracting and retaining loyal customers and working with the best suppliers—certainly no harm intended. Some business models that appear to be WMDs, might at worst be models of moderate distortion. Below are three examples.

Certainly, Gartner’s Top 25 Supply Chains ranking looks like USNWR’s Best College rankings. Gartner uses six face-valid weighted factors in rating supply chains: peer opinions, Gartner opinions, return on assets (ROAs), inventory turns, revenue growth and a newly added corporate social responsibility score.

While Gartner’s intent is to recognize best supply chain practices with regard to its demand-driven value network model, the supply chain community initially believed it to be the 25 best/excellent supply chains.

Like the Best Colleges report, it started a race to the top. Managers sought access to Gartner analysts and focused on attaining high scores. As I’ve written in past columns*, it cannot identify excellent supply chains. It just includes big companies, is reliant on opinions based on limited knowledge of supply chain operations and gives too much credit to a supply chain organization for revenue growth to which supply chain managers are not held responsible.

However, this does not make the Top 25 a WMD. After all, it was created by supply chain experts with an intent to stimulate healthy dialogue about what is possible. It would only hurt a company if an organization gamed its way to the top. This, however, would be fully transparent to its company because the only valid judge of whether a supply chain is excellent is the company in which it resides - not a third party.

Starting in late 2004, I wrote a three-part series on sales and operations planning (S&OP) because practitioners were asking for advice. It provided managers with a four-stage S&OP process maturity model that I believed was needed in support of the resurgence in S&OP.** Similar to the maturity models developed by consultants, my S&OP model gauged how developed a company’s process was relative to an ideal.

Consultants often use these to recommend a journey to get to an ideal over time. The S&OP model assessed a company’s stage in terms of factors including the meetings conducted; how integrated and extended were the processes; and the extent to which enabling software technologies were integrated. Because the S&OP process involves collaboration and consensus building among the supply, demand and financial components of a company, I assumed getting to Stage 4 would offer the greatest benefit without any data to support this.

Was the model a WMD? I don’t think so because I believed, as did the industry, that internal and external collaboration would help companies achieve financial InSIGHTSobjectives. So, a strong S&OP process would cause no harm, as long as a company installed a truly collaborative process instead of an extremely contentious one that turned out to be detrimental to corporate culture.

Lastly, I wrote about unconstrained vs. constrained demand forecasts, arguing that supply chain planning should be based on true supply-neutral demand forecasts.*** These are forecasts that reflect demand devoid of distortions related to supply surpluses, shortages and other supply factors. Over time, forecasting demand that is not supply-neutral might condition customers to demand products based on availability, rather than on true needs.

I’ve seen a number of examples of demand distortions. For example, many companies forecast customer demand from shipment data. In some cases, shipments are not the same as true demand, such as when supply factors cause orders to be filled imperfectly (not delivered on-time or as split shipments). When this is the case, a shipment forecast is not the best representation of true demand.

Thus, while this type of forecasting model is not a WMD, it is a model of moderate distortion; especially when customers become conditioned to accept imperfect order fulfillment. However, a shipment forecast can become a WMD when customers get tired of poor delivery performance and buy from a competitor.

In summary, while there is not much evidence to support the fact that the business community develops WMDs, it might develop models of moderate distortion if they rely too heavily on face-valid factors, without checking for validity. However, if a model leads to decision-making that results in the loss of a customer, I’d say it is destructive.