Notes
Short notes capturing the intuitions that clicked while building things. Usually one question I had and what I figured out.
For a fair coin, the expected value of the flip equals the probability of heads. Defining probability through expected value seems circular — but it's actually a bridge between math and reality.
The mean you learned in school is just expected value when every outcome is equally likely. Drop the uniform-probability assumption and you get a weighted sum — which leads directly to entropy.
Entropy measures how much you'll be surprised on average. Surprise measures how much information you gain. These are not two related concepts — they're literally identical.
If every character gets 8 bits, you're implicitly assuming uniform probability — and uniform probability means zero compression.