Entropy

Definition.

Let $X:\Omega \to \mathbf{R}$ be a random variable. Then the entropy of $X$ is:

$$H(X):=\mathbb{E}[I(X)]=\mathbb{E}[-\log P(X)]=-\sum _{x}P(X=x)\log P(X=x).$$

---

Essentially, the entropy of a random variable captures the expected amount of information in it.