The difference between a good expert and a bad expert

Who should you listen to for financial advice?

I’ve read a few stories and blog posts lately about the seeming ineptitude of “experts” to predict financial crises, pick stocks, or generally, live up to their expert status. Picking on experts is painfully easy. Having been pegged as an “expert” from time to time myself, I know how hard it is to live up to. First off, rarely does the expert decide he is the expert. Most of the time, it’s a desperate media-type looking for a source to add legitimacy to a story. Much of the time, the expert is actually knowledgeable in something tangentially related but not nearly the same. So the credit card expert gets interviewed about something mortgage-related, and everyone ends up looking foolish.

But I’m especially sympathetic to economists, investors, and other experts related to finance. Sure, they’re inaccurate much of the time. But modern finance is still a young science. When you look at stock market data, the most reliable data sets only go back 90 years or so. So deriving accurate predictions on market behavior from that is nearly impossible. Slamming stock market experts is like criticizing a doctor from the 15th-century, before germ theory was invented. Kind of a straw man if you ask me.

Still, I’d argue that there are good experts and bad ones. And a professor at the University of California at Berkeley has done studies to prove it. Philip Tetlock conducted a nearly 25-year study of political experts, quantifying their predictions and finding similarities in which ones were accurate and which weren’t. Here are his findings:

1. Good experts aren’t sure of themselves.

I’d add that good experts for the media are sure of themselves. It’s no good to hear someone on TV say, “Well, Erin, I’m not really sure where the market’s going to be tomorrow.” Or “I’m about 55% confident that the market will be up in 5 years.” Much better to have a really confident bull or bear throwing jabs at each other.

But Tetlock found that experts who equivocate end up being much more accurate than those who don’t. Said Tetlock in an interview:

You know the famous line that [philosopher] Isaiah Berlin borrowed from a Greek poet, “The fox knows many things, but the hedgehog knows one big thing”? The better forecasters were like Berlin’s foxes: self-critical, eclectic thinkers who were willing to update their beliefs when faced with contrary evidence, were doubtful of grand schemes and were rather modest about their predictive ability. The less successful forecasters were like hedgehogs: They tended to have one big, beautiful idea that they loved to stretch, sometimes to the breaking point.

I love talking to an investor who says things like, “Well, I think the market will be higher in 10 years, but I’m only 60% sure of that prediction.” That means he’s open to being wrong. He’s also probably always looking at evidence that would contradict his theory. Jeremy Grantham of GMO is one investor who constantly writes stuff like that in his investing notes. He also happened to be dead-on seven years ago when he predicted how asset classes would perform.

2. Good experts accept randomness as a fact of life.

I’ve never read a Dan Brown book cover-to-cover. But I’ve read a chapter or two over my girlfriend’s shoulder. Brown’s books are chock full of puzzles, all of which can be solved. If Brown writes about a string of seemingly random symbols, you just know Robert Langdon is going to figure out what’s up in a page or two.

It’s tempting to view investing that way. We look for patterns in everything. If the Dow hits a short-term peak at 10,560 five times in a row, investors start to view it as some sort of breaking point that can help them predict future returns. Of course in reality, the fact that the Dow did this or that a few times was probably just random. Good experts realize that and will point out that some things, like short-term market moves, are just naturally unpredictable.

3. Good experts admit when they were wrong and aren’t particularly bothered by it.

Bad experts, on the other hand, will say they were wrong because of someone else: “The market would have been fine if not for the politically expedient but economically devastating decision to let Lehman Brothers collapse.”

Or, they’ll have been so ambiguous about their prediction, that they’ll claim they’ll be vindicated sometime in the distant future: “Real estate will be the greatest investment in our lifetimes. We still have a lot of life left!”

Good experts will understand they were wrong and change their position to fit the circumstances. Moreover, the mistake won’t bother them, because they were never really that confident in the prediction in the first place.

So, next time you hear a talking head on TV, try to discern whether you’re listening to a fox or a hedgehog. They don’t let many foxes on television, but I think you’ll find that when you find one, whether it be on the topic of politics, economics, or any other sort of expertise, you’ll find their insights invaluable. And make sure you’re holding them to the right standard. A “good” expert is right more often than he’d be by picking winners at random. So really, we should be happy if our experts are wrong 49% of the time!

Tagged as: forecasting, media