The Expert Problem, or the Tragedy of the Empty Suit
Experts often have inflated confidence in their error rates, leading to two cases: mild arrogance with some competence, and severe arrogance mixed with incompetence (the empty suit). Questioning an expert's confidence and error rate is more valuable than questioning their procedure.
THE IDEA
So far we have not questioned the authority of the professionals involved but rather their ability to gauge the boundaries of their own knowledge. Epistemic arrogance does not preclude skills. A plumber will almost always know more about plumbing than a stubborn essayist and mathematical trader. A hernia surgeon will rarely know less about hernias than a belly dancer. But their probabilities, on the other hand, will be off—and, this is the disturbing point, you may know much more on that score than the expert. No matter what anyone tells you, it is a good idea to question the error rate of an expert’s procedure. Do not question his procedure, only his confidence. (As someone who was burned by the medical establishment, I learned to be cautious, and I urge everyone to be: if you walk into a doctor’s office with a symptom, do not listen to his odds of its not being cancer.)
I will separate the two cases as follows. The mild case: arrogance in the presence of (some) competence, and the severe case: arrogance mixed with incompetence (the empty suit). There are some professions in which you know more than the experts, who are, alas, people for whose opinions you are paying—instead of them paying you to listen to them. Which ones?
When to use it
Evaluating medical diagnoses by asking about the false positive or negative rates of tests.
Assessing financial advisors by reviewing their past prediction accuracy rather than their credentials.
Making decisions in legal settings by questioning expert witnesses' error rates in their field.
Choosing a consultant by focusing on their track record of outcomes rather than their persuasive confidence.
FROM THEORY TO EVERYDAY LIFE
In practice
Medical Diagnosis
A doctor confidently tells a patient that their symptom is very unlikely to be cancer, but the doctor's historical error rate for such predictions may be high. The patient, by questioning that error rate rather than the doctor's procedure, can make a more informed decision about seeking a second opinion or further tests.
Financial Advice
An investment advisor claims a high probability of market returns, but their track record shows frequent overestimation. A savvy client might recognize that the advisor's confidence exceeds their actual predictive accuracy and choose to diversify accordingly.
A MOMENT TO REFLECT
Put it to work.
When have you trusted an expert's confidence without checking their historical error rate?
In what areas of your life might you be an 'empty suit'—confident yet incompetent?
How can you distinguish between an expert's genuine competence and their overconfidence in probabilities?
CONTINUE THE CONVERSATION
Source texts
- The Black Swan The Impact of the Highly Improbable With a new section On Robustness and Fragility (Nassim Nicholas Taleb)