Selection Biased History

More about selection bias in business from Jerker Denrell; a great place to start with his work is "Selection Bias and the Perils of Benchmarking"

For more (oh so much more) about selection bias in general, see my book Thinking Statistically.


The other day, I was skimming through old photos of myself on Facebook, reminiscing about the year gone by. I noticed that almost all of my photos were either taken by or included one specific friend. Now, I'm not saying I'm a popular guy, but I do have more than one friend. At least, I think I have more than one friend.

It's pretty simple to see what happened here. How many photos you have with someone is not a representation of how much time you spent together this year. It's a representation of how much they like to take photos. This is an example of something that statisticians call selection bias where the information that reaches us isn't representative of the underlying trends, but rather reflect a particular filter.

In this case, which of my friends are most "snap happy"?

[camera snapping sound]

You might not think that getting an accurate record of who I spend time with this year is a very important problem to civilization as a whole, and you would be right. But the same mechanism that makes it look like I spent all of my year with just one friend is also responsible for some serious real world effect.

For example, it changes the way that we view history. It's easy to see how a civilization that was meticulous about making records, whether through written language or through other documentation, would be much more visible to us and well-known to us many centuries later than another civilization that may have been equally advanced at the time, but didn't leave records.

The selection bias problem when thinking about history, though, is a little more complex than that. See, a second mechanism by which we might hear or not hear about different civilizations, what they thought, felt, and did is because some civilizations were victorious in battle while other civilizations were not. We're much more likely to hear about the civilizations that won.

To understand why, you'd have to ask a real historian, which I'm not. But the general idea is that, for example, the losing civilizations may have had their records destroyed by the victors or because the victors set up huge museums and institutions which still run in our countries today and which improves the way that we think about the past, those victorious civilizations may or may not have been correct. They may or may not have kept good records. The reason that we hear from them is not because they were right. It's because they won. If you've ever heard someone say the phrase, "History is written by the victors," this is exactly what they're talking about. Similarly, in a business context, the scholar Jerker Denrell,

"Link's here. Also, here." [laughs] "You have no idea how hard it is to do that."

has documented extensively how the case studies and examples that we see when reading and thinking about businesses, whether it's in business school or business journals or in newspapers, magazines, and books, tend to be biased towards documenting successful examples.

If a company's a huge success, it'll have all kinds of things written up about it. It'll have its methods studied. It'll have its culture studied. It'll have wonderful things written about all its executives, but it could be that failed businesses had exactly the same traits. It's that we don't even go looking at them.

As was the case with my Facebook photos, some parts of business life are heavily documented.

[camera snapping sound]

Other parts aren't documented at all. This becomes especially important when we try to draw inferences from the case studies of businesses that we've heard about. Perhaps, there are some strategies, let's say, very risky strategies, that make it more likely a business will either succeed spectacularly or completely fail.

If most case studies and business contexts are about successful businesses, and we see that successful businesses shared certain traits, it could be that those traits do lead to success, but it could also be that those traits make it more likely a business will have an extreme outcome one way or the other. But because we fail to document the failures, we don't make the right inference.

In general, whenever we're in a situation where the information that reaches us is filtered in some way, and the cases that reach us and the ones that don't are affected by some of the key traits that those cases have, if we don't learn to spot and correct the selection bias, then we're not going to get the real picture.

[snapping sound]

The Best from Uri Bram

Limited time only: Get a free copy of Write Harder + (very) occasional & thoroughly excellent emails.

Thinking Statistically in Chinese

Thinking Statistically 的中文版将于2016年底推出。如欲及时获取本书出版资讯,请在此输入您的电邮地址:

Start Thinking Statistically

Thinking Statistically Book Cover

"Thinking Statistically explains essential concepts in statistics with wit and flair. Instead of page after page of mathematical mumbo-jumbo, Uri Bram tells stories that clearly illustrate the core ideas."

Get Thinking Statistically at now!