5.1. Basic Probability#
Here we will look at the definitions and meaning of some important concepts in Probability Theory
5.1.1. Random Variable#
A random variable, usually written X, is a variable whose possible values are numerical outcomes of a random phenomenon. There are two types of random variables, discrete and continuous.
The range space of a discrete RV is discrete, while for a continuous RV it is continuous. Discrete RV’s have a probability distribution whereas continous RV’s have a probability density function
Probability theory works on random variables, and all definations examples are best understood in this framework
Symbols:
5.1.1.1. Cdf#
Cdf (cumulative density function) of a Random variable X is denoted by
The cdf at x is the probability that if we sample/measure X, its value will be less than x.
Cdf functions can be calculated by:-
Discrete:
Continuous:
Quantile:
5.1.1.2. Multiple Random variables#
For the case of two random variables, let the RVs be
a. Joint pdf:
Joint probability of both variables having a specific value
b. Marginal pdf:
the probabilty of the random variable X having value x, if we have not observed y. Ie, pdf of X if we have the joint pdf
c. Conditional pdf:
Probability of RV X having a value x, conditioned on some value y for RV Y. The denominator is for normalisation.
Similary this can be extended to more than two multiple random variables
5.1.2. Expection, Variance, Moments#
Expectations are weighted averages of functions of random variables, where the weights are the pdf of the random variable and the values are values of the random variable.
The expected value of a function
It means that if X is a RV then
Any transformation of a RV is also a RV, hense here g(X) is also a RV.
The pdf if the weight as more the pdf more probabilty of that value occuring
The fact that we sample many times leads to a useful estimate of
It does not mean the expected value of the RV so never say that, rather it is always reffered to as the expectation value
Means
The mean or the first moment of a Random Variable is given by:
Also, the
5.1.2.1. Variance#
Variance of any RV X is its secound central moment defined by:
After some calculations, we can simplify it to:
5.1.2.2. Multiple Random Variables#
Expectation Value of the product of two random variables is given by:
Covariance of X,Y is given by:
Coorelation of X,Y is given by:
the correlation actually has a meaning between the physical relations of the two radom varaibles, as to how they vary with each other.
5.1.2.3. Independence:#
Two random variables X and Y can be called independent if
Which means that if
A set of multiple Random Variables
5.1.3. Random Sample#
Let X be a random variable with pdf
A set of random samples is set to be independent and identically distributed (iid) if:
5.1.4. Properties of Expectation and Variance#
1.
2.
3.
4. Var(a) = 0 : Variance of a scalar is zero
5.
Variance of Linear Transformation of a RV
6.
Proof:
7. For random samples: if
For the sample mean is
Proof:
as
here, we have independent samples
Note: in the important case that we will study ie. MCMC, the sample will not be independent.
5.1.5. 3 Fundamental Laws#
5.1.5.1. Weak Law of Large Numbers (WLLN)#
The Weak Law of Large Numbers states that the sample mean converges in probability to the population mean.
Suppose (
Then, for every
5.1.5.2. Strong Law of Large Numbers (SLLN)#
The Strong Law of Large Numbers states that the sample mean converges almost surely to the population mean.
That is, for every
5.1.5.3. Central Limit Theorem (CLT)#
Probabilty distribution (pdf) of the sample mean
Also stated as:
If (
is the standard distribution.
Here we will try a small example and see the central limiting tendency of sample mean for a Beta(2,5) pdf.
# Take a double normal and make histograms of samples of means of random samples of size 1,2,5, 10
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')
sample_mean_sizes = [1, 2, 5, 10]
# Set up the figure
fig, ax = plt.subplots(1, 4, figsize=(14, 3))
# Using a beta distribution with alpha = 2, beta = 5
# Plot the samples
for i, sample_mean_size in enumerate(sample_mean_sizes):
sample_means = []
for j in range(2000):
sample = np.random.beta(2, 5, sample_mean_size)
sample_means.append(np.mean(sample))
ax[i].hist(sample_means, bins = 20, density = True, label='sample size: ' + str(sample_mean_size) )
ax[i].set_xlim(0, 1)
ax[i].legend(loc='best', frameon=False)
plt.show()
