Analysis Paralysis

In May, Eleanor Laise of the Wall Street Journal (WSJ) published the article, Odds-On Imperfection: Monte Carlo Simulation – Financial Planning Tool Fails to Gauge Extreme Events, which was further reviewed in the September 2009 issue of Financial Advisor Magazine (FA) in the article, What Are the Odds? Certainly the point of Ms. Laise’ article is correct, that Monte Carlo simulations failed to predict the economic downturn being experienced today.  However, the report calls these simulations into question and highlights the rash conclusions that some analysts are too eagerly jumping to: that Monte Carlo is a risky predictive methodology.  This certainly could not be farther from the truth.

Although some are extremely complex, with multiple built-in contingencies and data relationships, a Monte Carlo simulation is, quite simply, a fancy random number generator.  Based on the analyst’s assumption, a probability distribution function (PDF) is developed along which results are generated.  For example, if data is modeled along a normal bell distribution, the simulation will randomly select values within that distribution, with greater emphasis on values trending toward the center.  The goal is to run these random samples enough times to obtain confidence in the predictions, but not so many that the original distribution is simply recreated.  The key risk to the Monte Carlo simulation is, therefore, the analyst’s original assumptions in developing the PDF or PDFs utilized within a given scenario.

When developing a PDF, analysts should utilize as much data as is available, with some exceptions.  Any systematic changes resulting in data fluctuations must first be removed.  For example, if an analysis of widget failure rates is being predicted and, five years ago, an inherent flaw in the factory was corrected, it would generally be inappropriate to use data prior to the correction without first standardizing that data.  So, the key to these analyses is, as always in data analysis, knowing your data.  Perhaps this is where the WSJ article should have focused its discussions – not on potentially debunking the usefulness of Monte Carlo simulations, but on stressing the importance of selecting the most accurate and representative underlying dataset to utilize within that simulation.

So, did Monte Carlo simulations fail to predict the current economic downturn?  Most assuredly.  Should this method of predicting the future be stopped to prevent similar, future errors?  Certainly not.  According to the WSJ article, “[s]ome firms are considering revising Monte Carlo models to reflect a world where big market swings happen more often.”  Increasing the risk factors within Monte Carlo simulations will certainly increase the confidence level of those simulations, but will simply result in useless data – if you always predict that something will fail, you’ll never be wrong when it does fail.  However, this will not assist individuals in planning efforts and will certainly not help prevent losses in future recessions, since the analyses will become so fundamentally useless as to be ignored.  This isn’t to say that models should not be reviewed and revised.  The ever shrinking global financial markets are becoming more interdependent every day, which changes the game – basic assumptions do need to be reset to take these variables into account.  Without further data, however, models should not be revised to predict increases in market downturns.  There was the Great Depression in the 30s, market inclines and declines over the years, a downturn in the 80s, and a dot-com bubble burst in the 90s.  It happens.  Certainly, models could be improved to better predict when these potentially cyclical events may occur, but planning for the worst will just diminish the returns of today.

Monte Carlo simulations are extremely useful tools, as long as the analyst designing them and the end user making decisions both understand all the assumptions and potential pitfalls.  The best simulation is only a reasonable estimate of the future over many time periods – it should not be the only thing considered, but it also shouldn’t be ignored, and the methodology shouldn’t become a tool of the past.