Comparative Luck
A performance comparison contains uncertainty in both the person’s result and the benchmark. Taleb argues that a large gap can reflect differences in luck, so the gap alone does not establish a difference in skill.
THE IDEA
Comparative Luck
A far more acute problem relates to the outperformance, or the comparison, between two or more persons or entities. While we are certainly fooled by randomness when it comes to a single time series, the foolishness is compounded when it comes to the comparison between, say, two people, or a person and a benchmark. Why? Because both are random. Let us do the following simple thought experiment. Take two individuals, say, a person and his brother-in-law, launched through life. Assume equal odds for each of good and bad luck. Outcomes: lucky-lucky (no difference between them), unlucky-unlucky (again, no difference), lucky-unlucky (a large difference between them), unlucky-lucky (again, a large difference).
I recently attended for the first time a conference of investment managers where I sat listening to a very dull presenter comparing traders. His profession is to select fund managers and package them together for investors, something called “funds of funds” and I was listening to him as he was pouring out numbers on the screen. The first revelation was that I suddenly recognized the speaker, a former colleague biologically transformed by the passage of time. He used to be crisp, energetic, and nice; he became dull, portly, and inordinately comfortable with success. (He was not rich when I knew him—can people react to money in different ways? Do some take themselves seriously while others do not?) The second revelation was that while I suspected that he was fooled by randomness, the extent had to be far greater than one could imagine, particularly with the survivorship bias. A back of the envelope calculation showed that at least 97% of what he was discussing was just noise. The fact that he was comparing performances made the matter far worse.
When to use it
Comparing investment fund performance to a benchmark.
Comparing employee performance in performance reviews.
Comparing sports teams' records over a short season.
Comparing startup success rates among founders.
FROM THEORY TO EVERYDAY LIFE
In practice
Fund of funds manager comparing traders
Taleb recalls hearing a fund-of-funds manager compare traders and estimates, in a back-of-the-envelope calculation, that at least 97% of the discussion was noise. The excerpt does not supply that calculation.
Thought experiment with two individuals
Assume two equally skilled people each have equal chances of good and bad luck, independently. There is then a 50% chance that one is lucky and the other unlucky, creating a performance gap without a skill difference.
A MOMENT TO REFLECT
Put it to work.
When you see a comparison of two people's performances, what factors of luck might be at play?
How can you account for randomness when evaluating fund managers or employees?
What assumptions and data would you need to estimate how much of a performance gap is attributable to luck?
CONTINUE THE CONVERSATION
Source texts
- Fooled by Randomness The Hidden Role of Chance in the Markets and in Life (Nassim Nicholas Taleb)