Only The Good (don't) Die Young: Survivorship Bias and Statistical Inference (from Catallarchy)

Patri Friedman at Catallarchy has some great thoughts on survivorship bias. In the first post, Selection Bias and Risky Strategies, he writes about payoffs to players in poker tournaments. He writes:

...The same thing happens in the poker tournament world. Certain styles of play trade EV for variance, allowing people to build up huge stacks occasionally, but usually go bust. Such players often win tournaments - but that doesn’t mean they are playing right. How many times do they fail for each victory? Do they fail more often compared to the money they win than a more conservative player? Some of these “maniacs” are smart players, carefully choosing their gambles and maximizing their returns. But some of them, frankly, are just maniacs, gambling and getting lucky, and giving the false impression that high-variance play is the way to go, because we don’t notice the hundreds of people playing that way and losing.

The application for financial economists is that when analyzing only the firms which "survived", you end up with a skewed picture. As I was putting together a piece on mutual funds, Friedman followed up with a second post titled Wall Street's Shell Game, which pretty much stole my thunder. He starts off with this analogy:

Get a list of email addresses of people interested in sports betting. Say you have 32,000. Email 16,000 of them to say that the home team will win this week’s big team, and 16,000 to say the home team will lose. Now, half of the people will have gotten the correct prediction, and the next week, you do the same thing with them. After 5 weeks, you’ll have 1,000 email addresses of people who have seen you pick the winner five times in a row!. Now you pitch your 1-900 number or paid email list subscription to this amazed group.

If you consider that there's thousands of mutual funds, at least some are likely to beat the market 5 years running. It's entirely possible that there are some managers that can continue to sonsistently beat the market. However, you can't tell which are good, and which are lucky.

For a discussion of this idea, see a recent post titled "Is He Good, or Is He Lucky?" Which explains that decause there's such high variance, even beating the market by 2-3% annually over a five year period could result from chance the vast majority of the time.

For those with an academic bent, there have been quite a few academic pieces on the effect of survivorship bias in capital market research. In mutual funds research, one of the major data sources is the Carhart database, which includes data on both surviving and failed mutual funds. For a link to some of Carhart's research (on the SSRN), click here.