NLPA Math Review

1. information theory

1.1. entropy

1.2. joint entropy

1.3. conditional entropy

1.4. mutual information

1.5. noisy channel model

1.6. channel capacity

1.7. relative entropy, Kullback-Leibler divergence

1.8. cross entropy

1.9. perplexity

1.10. entroy of English

2. statistics

2.1. basics

2.1.1. probabilities

2.1.2. conditional probabilities

2.1.3. independence, conditional independence

2.1.4. random variables

2.1.5. expectation and variance

2.1.6. joint distributions

2.1.7. conditional distributions

2.2. standard distributions

2.2.1. binomial distribution

2.2.2. multinomial distribution

2.2.3. normal distribution

2.2.4. central limit theorem

2.3. Bayesian methods

2.3.1. Bayes theorem

2.3.2. Bayesian parameter estimation

2.3.3. Bayesian updating

2.3.4. Bayesian decision theory