Probability Distributions Review & Cheatsheet (2): Discrete Distributions, Continuous Distributions, Exponential Family Distributions, Conjugate Priors.
1. Discrete Distributions
1.1 Bernoulli Distribution
Suppose has Bernoulli distribution, , it is about a trial performed with probability of to be “success”, and takes 1 if it is success and 0 if it is failure.
The expectation and variance of is and .
1.2 Binomial Distribution
Suppose has Binomial distribution, , it models independent trials with probability of to be “success”. Therefore it could be regarded as the sum of i.i.d. random variables.
The sum of independent binomial distributions with parameter and is an another binomial distribution with parameter and .
The expectation and variance of is and .
1.3 Poisson Distribution
Suppose has Poisson distribution, . It models the number of times that rare event occurs with an average rate (per unit time).
Poisson distribution is an approximation to Binomial distribution with large and very small . The sum of independent Poisson distribution with parameter is another Poisson distribution with parameter .
The expectation and variance of are both .
1.4 Geometric Distribution
Suppose has Geometric distribution, , it models the number of trials related to first success. There are two scenarios: (1) is the number of trials before first success, i.e. total number of failures before first success; (2) is the total number of trials until first success. The distribution of second scenario is actually a “shifted” version of first scenario. The PMF of first scenario is
The expectation and variance of Geometric distribution with parameter is
If the Geometric distribution is for the total number of trials (second scenario), since it is a “shifted” of first scenario, therefore the expectation will increase by one unit, i.e.
and variance remains same.
1.5 Negative Binomial Distribution
Suppose has Negative Binomial distribution, . It models number of failures before -th success.
As i.i.d. Bernoulli distribution sum up to Binomial distribution, here i.i.d. Geometric distribution sum up to Geometric distribution.
The expectation and variance of is
2. Continuous Distributions
2.1 Uniform Distribution
Suppose has Uniform distribution, , with PDF
The expectation and variance of is
2.2 Exponential Distribution
Suppose has Exponential distribution, , with PDF
The shape of PDF is strict decreasing with decay rate . The CDF is given by
Exponential distribuion could be used to model lifetimes and time between events. The expectation and variance is
Exponential distribution has memoryless property, i.e. .
2.3 Gamma Distribution
Suppose has Gamma distribution, , with PDF
It is easy to see that when it is an Exponential distribution . Here Gamma function is defined as
where , .
Also, is actually distribution.
2.4 Beta Distribution
Suppose has Beta distribution, , with PDF
where
2.5 Normal Distribution
Suppose has Normal distribution, , with PDF
2.6 Exponential Family Distributions
This is a family of distributions, distributions like bernoulli distribution, poisson distribution, exponential distribution, gamma distribution, beta distribution, normal distribution all belong to exponential family.
Where is the sufficient statistic. The data and parameter interact through the linear term in the exponent. The MLE of satisfies
3. Conjugate Prior (Bayesian Statistics)
List of commonly used conjugate prior.
3.1 Model Binomial Data
If and , then .
Beta prior is conjugate for Binomial likelihood, means posterior has same parameteric form as prior. Beta prior has interpretation as “prior data” of success and tries.
The mean of the posterior is a weighted average of prior mean and likelihood mean.
3.2 Model Event Count Data
If and , then
The mean of the posterior is a weighted average of prior mean and likelihood mean.