Nehaveigur

Yesterday, the S&P 500 closed at 5,469 points. Do you think it’s going to be higher or lower by the end of the year?

You have an opinion, but really, you don’t know. But maybe I’m wrong. Maybe you’re a hundred percent certain you know which way the stock market is headed. If you are, why not re-mortgage your house and short some stock? Or go long?

What do people really mean when they say that they’re one hundred percent certain? Actually, this has been tested. You ask people to predict something. You also ask them how confident they are. It turns out that when people say that they’re hundred percent certain, they are actually right only 70 to 80 percent of the time. People are over-confident.

Clearly, it’s not a good idea to say you’re hundred percent certain when you’re actually trying to accurately communicate your confidence in a prediction. People who do are probably trying to sell you something: I’m absolutely certain this Subaru is not going to fail. And no, that’s not rust, that’s… protective coating.

A lot of people are over-confident and therefore bad at forecasting. But how do we know who’s good at forecasting? How do we distinguish the charlatans from the geniuses? People have been thinking about this for some time. One of them is Philip Tetlock, a professor at the University of Pennsylvania.

For years now, he’s been organized forecasting tournaments. Hundreds of participants try to predict things like if Vladimir Putin will still be president of Russia by the end of the year or if North Korea will test another nuclear device.

They use the Brier score to measure how well a forecaster is doing across all of their forecasts. The Brier score also takes into account the level of confidence in each forecast. In short, the Brier score tells you how good a forecaster someone is.

Insight one: Some people are better at forecasting than others. They are the superforecasters. Tetlock’s term, not mine. Superforecasters are better at predicting the future even than prediction markets. How are superforecasters different from other people? Generally, they are comfortable with holding contradictory ideas in their mind. They also use meta-cognition: They think about the way they think. Also, they’re not experts in the fields in which they excel at forecasting.

Insight two: There are things you can do to improve your own forecasting ability. Be open-minded. Draw up a list of reasons why you may be wrong. That’s easy to say but hard to do. It’s also known as self-subversion. Sounds bad but is good. Another thing to do is to use reference classes. For example, if you’re trying to predict how long a specific congressman is going to serve before they’re voted out, the first thing to consider is the distribution of the average tenures of congressmen and congresswomen. Your forecast should mainly be based on that, and only then on the current political environment and so on. That way you avoid recency bias.

Insight three: Even the best forecasters can’t see more than a year into the future. Of course that’s a generalization and depends on the type of question you ask. For example, demographic forecasts are reasonable over longer time periods. But for anything that’s not a straightforward extrapolation and that’s more than a year in the future, even superforecasters don’t do better than a dart-throwing chimp.

Here’s a prediction of my own: We’ll never be able to predict the future with complete confidence. But the most important takeaway from the forecasting tournaments is that it’d be wrong to be fatalistic about our abilities. We can do better, and that’s good enough.

Related: My baseline expectations that will allow me to judge if things are going well.