2.1

2.1

Author: rammmaa

_posts/2025-08-04-02.Information-Theory.md

@@ -8,12 +8,176 @@ layout: post
 
 ## 2.1 Entropy
 
-long contents .....
+> **놀람도 (Surprisal)**  
+> 어떤 사건이 발생했을 때 느끼는 놀라움의 정도.
+
+$$
+S(x) = -\log_2 p_X(x)
+$$
+
+- 사건이 드물수록 $p_X(x)$ 값이 작아져 놀람도가 커진다.  
+- 단위는 **bits** (밑이 2인 로그 사용).
+
+---
+
+**Example 2**  
+$X$: 지진 발생 여부를 나타내는 indicator  
+
+- $X = 1$: 큰 지진 발생  
+- $X = 0$: 지진 아님  
+
+만약 $p_X(1)$이 매우 작다면, $X=1$은 매우 놀라운 사건이다.
+
+---
+
+> **엔트로피 (Entropy)**  
+> 확률변수 $X$의 평균 놀람도 (기댓값)  
+
+$$
+H(X) = E[-\log_2 p_X(X)]
+     = \sum_{x \in X} p_X(x) \log_2 \frac{1}{p_X(x)}
+$$
+
+---
+
+**Example 4**  
+Binary random variable, $p(1) = 0.01$, $p(0) = 0.99$
+
+$$
+H(X) = 0.01\log_2\frac{1}{0.01} + 0.99\log_2\frac{1}{0.99}
+\approx 0.080793\ \text{bits}
+$$
+
+---
+
+**Example 5**  
+Uniform distribution on a fair six-sided die $(p = 1/6)$
+
+$$
+H(X) = \sum_{i=1}^{6} \frac{1}{6}\log_2\frac{1}{1/6}
+= \log_2 6
+\approx 2.584963\ \text{bits}
+$$
+
+---
+
+**Example 6**  
+$X \sim \text{Bern}(1/2)$, $Y$ with $(0.99, 0.01)$
+
+$$
+H(X) = -\frac12\log_2\frac12 - \frac12\log_2\frac12 = 1\ \text{bit}
+$$
+
+$$
+H(Y) = -0.99\log_2(0.99) - 0.01\log_2(0.01) \approx 0.080793\ \text{bits}
+$$
+
+$\Rightarrow\ H(X) > H(Y)$
+
+---
+
+**Example 7** (Guessing game)  
+
+Q1: $P(\text{Yes}) = 1/2$, Q2: $P(\text{Yes}) = 1/4$
+
+$$
+H(\text{Q1}) = -\frac12\log_2\frac12 - \frac12\log_2\frac12 = 1\ \text{bit}
+$$
+
+$$
+H(\text{Q2}) = -\frac14\log_2\frac14 - \frac34\log_2\frac34
+= \frac14 \cdot 2 + \frac34\log_2\frac{4}{3}
+\approx 0.811278\ \text{bits}
+$$
+
+$\Rightarrow\ \text{Q1 is more informative}$
+
+---
+
+**Example 8**  
+
+$$
+D_1: \left(\frac19,\frac19,\frac19,\frac29,\frac29,\frac29\right), \quad
+D_2: \left(\frac12,\frac14,\frac18,\frac{1}{16},\frac{1}{32},\frac{1}{32}\right)
+$$
+
+$$
+H(D_1) = 3\cdot\frac{1}{9}\log_2 9 + 3\cdot\frac{2}{9}\log_2\frac{9}{2}
+\approx 2.503258\ \text{bits}
+$$
+
+$$
+H(D_2) = \frac12\cdot 1 + \frac14\cdot 2 + \frac18\cdot 3 + \frac{1}{16}\cdot 4 + 2\cdot\frac{1}{32}\cdot 5
+= 1.9375\ \text{bits}
+$$
+
+$\Rightarrow\ H(D_1) > H(D_2)$
+
+---
+
+**Example 9**  
+
+$$
+x_1: (0.6,0.1,0.1,0.1,0.1), \quad x_2: (0.4,0.4,0.2,0,0)
+$$
+
+$$
+H(x_1) = 0.6\log_2\frac{1}{0.6} + 4\cdot 0.1\log_2\frac{1}{0.1}
+\approx 1.770951\ \text{bits}
+$$
+
+$$
+H(x_2) = 0.4\log_2\frac{1}{0.4} + 0.4\log_2\frac{1}{0.4} + 0.2\log_2\frac{1}{0.2}
+\approx 1.521928\ \text{bits}
+$$
+
+$\Rightarrow\ H(x_1) > H(x_2)$ ( $x_1$ has more uncertainty )
+
+---
+
+**Example 10**  
+
+$$
+S_1: (0.6,0.1,0.1,0.1,0.1), \quad
+S_2: (0.4,0.4,0.2,0,0)
+$$
+
+$$
+H(S_1) = 0.6\log_2\frac{1}{0.6} + 4\cdot 0.1\log_2\frac{1}{0.1}
+\approx 1.770951\ \text{bits}
+$$
+
+$$
+H(S_2) = 0.4\log_2\frac{1}{0.4} + 0.4\log_2\frac{1}{0.4} + 0.2\log_2\frac{1}{0.2}
+\approx 1.521928\ \text{bits}
+$$
+
+$\Rightarrow\ H(S_1) > H(S_2)$ ( $S_1$ is more explorative )
+
+---
+
+**Example 11**  
+
+Geometric distribution, $p = \frac12$, $x \ge 1$  
+$p_X(x) = 2^{-x}$
+
+$$
+H(X) = \sum_{x=1}^{\infty} 2^{-x} \log_2\frac{1}{2^{-x}}
+= \sum_{x=1}^{\infty} 2^{-x} \cdot x
+= \sum_{x=1}^{\infty} x\left(\frac12\right)^{x}
+$$
+
+$$
+G(r) = \sum_{x=1}^{\infty} x r^{x} = \frac{r}{(1-r)^2}, \quad |r|<1
+$$
+
+$$
+H(X) = G\!\left(\frac12\right)
+= \frac{\frac12}{\left(1-\frac12\right)^2}
+= 2\ \text{bits}
+$$
+
 
-1. a
-2. b
-3. c
-4. d
 
 ## 2.2 Properties of Entropy