Scarce Information
The concept of scarce information, rooted in information theory, states that to uniquely identify N distinct objects, at least log N bits of information (binary cues) are required. This fundamental limit means that with fewer cues than log N, perfect predictability is impossible.
THE IDEA
Scarce Information
In order to illustrate the concept of scarce information, let us recall an important fact from information theory: a class of N objects contains log N bits of information. This means that if we were to encode each object in the class by means of binary cue profiles of the same length, this length should be at least log N if each object is to have a unique profile. The example in Table 31.6 illustrates this relation for N = 8 objects. The eight objects are perfectly predictable by the three (log8 = 3) binary cues. If there were only two cues, perfect predictability simply could not be achieved.
When to use it
Determining the minimum number of survey questions needed to reliably segment consumers into distinct groups.
Designing a diagnostic test with sufficient accuracy to differentiate between multiple diseases using a limited set of binary symptoms.
Evaluating whether a machine learning dataset has enough informative features to achieve perfect classification of target categories.
Planning an experiment where the number of binary factors is restricted, ensuring that the number of distinct treatment conditions does not exceed the information capacity.
FROM THEORY TO EVERYDAY LIFE
In practice
Eight Objects and Three Binary Cues
Consider a set of 8 distinct objects. According to the principle, we need at least 3 binary cues (since log2 8 = 3) to assign each object a unique binary profile (e.g., 000, 001, ..., 111). With only 2 cues, some objects would share the same profile, making perfect identification impossible. Unambiguous encoding requires this minimum amount of information.
Fruit Classification
Suppose you have 8 different types of fruit and want to classify them using only attributes like 'red/green', 'round/oval', and 'seeded/seedless'. With these three binary cues, each fruit can be uniquely identified. If you had only two cues (e.g., color and shape), at least two fruits would become indistinguishable, demonstrating the scarcity of information.
A MOMENT TO REFLECT
Put it to work.
How does the concept of scarce information apply to designing a classification model when you have limited features?
Can you think of a real-world scenario where too few predictors lead to ambiguity or misclassification?
In what ways does this principle influence the interpretation of statistical models with small sample sizes?
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)