“Good To Great” …to pretty bad

I went down a deep rabbit hole with examples of survivorship bias yesterday.

Survivorship bias or survival bias is the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. This can lead to some false conclusions in several different ways. It is a form of selection bias.

Wikipedia

I came across this post which talks about the book Good To Great by Jim Collins. That book is respected as a solid, research driven analysis of what has made successful companies successful. But since the book was written, the companies it talked about haven’t done so “great” at all.

Overall, a portfolio of the “good to great” companies looks like it would have underperformed the S&P 500.

Steven D. Levitt, Freakonomics

Two thoughts come to mind here:

Thought 1: Anything that points out common themes among the successful (or unsuccessful) is probably confusing correlation with causation. This sucks for anyone who reads a lot of business or leadership books, like me. Most of them are some variation of “lost of successful people do X, therefore if you do X then you’ll be successful too.” Hello survivorship bias. Could there be an invisible horde of folks who did X and still failed?

Thought 2: A claim is only as good as its ability to be replicated. The replication crisis continues to rear its ugly head. Lots of well known studies are almost impossible to replicate, making them more or less meaningless. For example, the studies in the much worshipped book Thinking Fast and Slow have an average replication index of 14 (out of 100). That’s garbage.

This result confirms Kahneman’s prediction that priming research is a train wreck and readers of his book “Thinking Fast and Slow” should not consider the presented studies as scientific evidence that subtle cues in their environment can have strong effects on their behavior outside their awareness.

Replication Index

Books are written by people, and people have biases. People are often wrong. The existence of a respected book (Kahneman won a dang Nobel Prize) doesn’t necessarily mean what it says is true.

With a subject as complex as social sciences, we should mine for ideas and suggestions, not facts and commands.