The Idea of Positive and Negative Model Error

The Idea of Positive and Negative Model Error

Now what I believe is my true specialty: error in models.

When I was in the transaction business, I used to make plenty of errors of execution. You buy one thousand units and in fact you discover the next day that you bought two thousand. If the price went up in the meantime you had a handsome profit. Otherwise you had a large loss. So these errors are in the long run neutral in effect, since they can affect you both ways. They increase the variance, but they don’t affect your business too much. There is no one-sidedness to them. And these errors can be kept under control thanks to size limits—you make a lot of small transactions, so errors remain small. And at year end, typically, the errors “wash out,” as they say.

But that is not the case with most things we build, and with errors related to things that are fragile, in the presence of negative convexity effects. This class of errors has a one-way outcome, that is, negative, and tends to make planes land later, not earlier. Wars tend to get worse, not better. As we saw with traffic, variations (now called disturbances) tend to increase travel time from South Kensington to Piccadilly Circus, never shorten it. Some things, like traffic, do rarely experience the equivalent of positive disturbances.

This one-sidedness brings both underestimation of randomness and underestimation of harm, since one is more exposed to harm than benefit from error. If in the long run we get as much variation in the source of randomness one way as the other, the harm would severely outweigh the benefits.

So—and this is the key to the Triad—we can classify things by three simple distinctions: things that, in the long run, like disturbances (or errors), things that are neutral to them, and those that dislike them. By now we have seen that evolution likes disturbances. We saw that discovery likes disturbances. Some forecasts are hurt by uncertainty—and, like travel time, one needs a buffer. Airlines figured out how to do it, but not governments, when they estimate deficits.

This method is very general. I even used it with Fukushima-style computations and realized how fragile their computation of small probabilities was—in fact all small probabilities tend to be very fragile to errors, as a small change in the assumptions can make the probability rise dramatically, from one per million to one per hundred. Indeed, a ten-thousand-fold underestimation.

Finally, this method can show us where the math in economic models is bogus—which models are fragile and which ones are not. Simply do a small change in the assumptions, and look at how large the effect, and if there is acceleration of such effect. Acceleration implies—as with Fannie Mae—that someone relying on the model blows up from Black Swan effects. Molto facile. What I can say for now is that much of what is taught in economics that has an equation, as well as econometrics, should be immediately ditched—which explains why economics is largely a charlatanic profession. Fragilistas, semper fragilisti!