Smash that Crystal Ball

It's that time of year again.

Yep, the time of year when the major Bay Street and Wall Street firms, along with the major mutual fund companies, parade their gaggle of economists and strategists in front of every camera, microphone and scribbling print journalist in order to emboss their firm's logo on the impressionable minds of investors.

Eventually, the thinking goes, your current manager will have a poor year, or a poor twenty years, and you will inevitably start thinking about moving your hard-earned capital to another, more prospective wealth management group. If their firm has caught your attention over the years more often than other firms, the thinking goes, you are more likely to seek them out than their competition. 

This is one of the more common ways in which firms compete for your business.

They know from experience that you won't remember what their so-called 'expert' proclaimed on the news last year, or last month, about the shape of things to come. They know if doesn't matter what they say so much as the fact that they are out there, in the media, saying something.

But there is another, more insidious reason why each major firm has a variety of experts on staff loudly proclaiming their views year after year. The reason is simple: banks, mutual fund companies and investment firms make no money while clients are sitting in cash. Each time the investing public observes an expert loudly proclaiming that gold is going up, or interest rates are going up, or banks are going up, a few of them take heed, call their Advisor or consultant, and make changes to their portfolio. And each time they make a change, the Advisor, investment firm, or mutual fund company makes money.

Unfortunately, on average the investing public doesn't. In fact, by acting on this silliness, the average investor earned 4% per year less than the stock market from 1991 -2011.

Actual results to mutual fund investors vs. stock and bond benchmarks - 1991 to 2011

Source: Dalbar

That's why, at this time of year it is especially important to remind yourself about the abysmal track record these 'experts' have had over the years.

But before presenting the ugly details, I want to emphasize that investors should not feel disheartened by the evidence that financial marketing and media is dominated by loud, overconfident shills and mountebanks. On the contrary, investors should feel liberated to pursue other interests rather than reading or watching business news. For those that enjoy the cognitive 'sport' of investing from the standpoint of strategy and game theory, feel free to explore the latest economic, financial, ideological or philosophical fads with your colleagues and friends as provocative dinner conversation. This type of thinking keeps the mind young, after all.

Just don't orient your portfolio on the basis of your conclusions, or the conclusions of other prognosticators. We are all bound to be wrong far more often than we are right. For that is the nature of complex, dynamic systems like the markets.

The Truly Dismal Science

Now, here is the evidence. The following charts show aggregate forecasts from Wall Street's most famous oracles through time, next to the actual trajectory of the forecast variable. Note that these charts are sourced from James Montier's book Behavioural Investing (2007):

Chart 1. Consensus bond yields forecasts 1 year out vs. actual

Chart 2. Consensus S&P500 level 1 year forecasts vs. actual

Chart 3. Consensus S&P500 aggregate earnings 1 year forecasts vs. actual

In all cases the analysts appear to do a noteworthy job of describing what just happened, but appear to have no vision whatsoever about what is about to happen next. This applies to interest rates, the level of stock indices, and aggregate earnings.

Source: Despair.com

Do any experts get it right? What about the experts at the Federal Reserve who are in charge of setting interest rates? Can they predict the magnitude or direction of interest rates over the next six months?

A working paper entitled "History of the Forecasters: An Assessment of the Semi-Annual U.S. Treasury Bond Yield Forecast Survey" (Brooks & Gray, 2003) studied the ability of Federal Reserve economists, including Alan Greenspan, from 1982 - 2002 to discover whether the group of experts that sets interest rates is able to effectively forecast their trajectory through time. 

Chart 3.

Source: (Brooks & Gray, 2003)

Again we see a strong talent for describing what has just happened, but no talent whatsoever for predicting what will happen next. Just how poor was the forecasting ability of Fed economists, including sitting Fed Chairmen like Alan Greenspan, over the 20 year survey?

Chart 4. 

Source: (Brooks & Gray, 2003)

The scatter plot above shows how Fed forecasts of interest rates six months out are negatively correlated with actual outcomes. The r-squared of the regression is 0.07, which is not statistically significant, but if you were a betting man, your best bet would be in the opposite direction of what is forecast to happen by the people who actually set interest rates for the economy.

Overconfidence - of course

The error rate would not be so worrisome if it weren't for the high level of confidence that investment professionals imbue on their predictions. This effect is perhaps best illustrated using the results of a study by Torngern and Montgomery (2004). The study set laypeople (psychology undergraduates, the perennial guinea pigs) against investment professionals in a competition to select the stock that they thought would outperform over the next month from pairs of stocks. All the stocks were well known companies, but participants were given information such as the industry and prior 12-month performance for each stock as well. Participants were asked to choose the best performer from the pair, and to provide their level of confidence in their choice.

Over many picks, one might hope that when participants were 50% confident that their choice was right, they were accurate about half the time, and when they were 90% confident, they were right almost all the time. In fact, as you can see from the chart below, a person's confidence level was largely irrelevant to their accuracy over time. In other words, having greater confidence in a choice did not lead to higher accuracy levels. In fact, at extreme levels of confidence (>80%), professionals were actually less likely to get it right. At a 90% level of confidence, professional investors actually got it right only 15% of the time, while at a 55% - 75% level of confidence they achieved about 40% accuracy.

Chart 6: Accuracy and confidence on a stock selection task

Source: Torngren and Montgomery (2004)

It is important to remember that over a 1-month time horizon the results of these stock choices are almost random, so we are not out to skewer professionals on the basis of their accuracy in this test. Instead, we are left to wonder why anyone should have expressed such high levels of confidence in their choices. When asked this question, the layperson group admitted that they were mostly guessing, but also placed some emphasis on the previous month's returns (ahh, momentum at work). In contrast, almost no professionals admitted to guessing; instead, they attributed their choices to 'Other knowledge' about the stocks, and 'Intuition'. Incidentally, the only factor with any predictive power in this example, however small, is the previous month's results (momentum).

Chart 7. Average rating of decision input importance

Source: Torngren and Montgomery (2004)

So Now What?

The quantum leap in thinking that we strive to compel with this post is toward an understanding that the world is too complex to enable accurate forecasting. Axiomatically, people should consider expert forecasts as no more than entertaining narratives - brain candy to stimulate the imagination. 

The best we can hope for is an assessment that an existing dynamic or trend is likely to stay on a certain course, or alternatively that the dynamic is reverting to the mean. Forecasting the direction or the magnitude of the change in trend is empirically impossible.

Fortunately, statistics offers a useful toolkit for evaluating the probability of trend continuation or mean reversion, and offers an estimate about the magnitude of the change. Further, statistical models tell us how confident we should be in our estimates given the amount of data we have at our disposal, and the type of problem we are trying to solve.

Quantitative systematic approaches to investing explicitly leverage the power of statistics to make a large number of bets with better than even odds of success. Quantitative risk management techniques and optimizations can further stabilize results by minimizing the impact of being wrong, which in markets will happen a lot. 

Quantitative approaches are no silver bullet - they suffer periods of poor performance too. But they are explicitly built to take advantage of the law of large numbers, and to stay out of trouble when statistical forecast accuracy is poor.

For more information about quantitative approaches, please read the following articles.

http://gestaltu.blogspot.com/2012/08/if-you-could-manage-portfolio.html

http://gestaltu.blogspot.com/2012/05/intuition-is-for-suckers.html

http://gestaltu.blogspot.com/2012/05/despite-thousands-of-mutual-funds-and.html

http://gestaltu.blogspot.com/2012/05/volatility-analysis-for-lower-risk-and.html

http://gestaltu.blogspot.com/2012/03/another-expert-bites-dust.html