Joint Probability Distribution and Conditional Probability

Last Updated: 6th October, 2023

Joint distributions and conditional probability are important concepts in probability theory that describe the probability of two or more events occurring simultaneously and the probability of an event occurring given that another event has occurred. Marginal probability distributions and conditional probability distributions are used to model the behavior of random variables in joint distributions.

Introduction to Joint Distributions and Conditional Probability

Joint distributions are a fundamental concept in probability theory that describe the probability of two or more random variables taking on specific values simultaneously. In other words, they allow us to model the relationships between variables and calculate the probability of specific events occurring.

Conditional probability, on the other hand, describes the probability of an event occurring given that another event has already occurred. It is a way to update our probabilities based on new information.

Marginal Distributions and Joint Densities

Marginal distributions describe the probability distribution of a single variable in a joint distribution, ignoring the other variables. They are obtained by integrating the joint density function over the other variables. The formula for calculating marginal density is:

f_X(x) = ∫ f_XY(x,y) dy

Where f_XY(x,y) is the joint density function of the variables X and Y.

Example:

Suppose we have the joint density function f(x,y) = 2x for 0 ≤ y ≤ x ≤ 1. To find the marginal density of X, we integrate over y as follows:

f_X(x) = ∫ 2x dy 

from

y=0 to y=x = 2x^2

Hence, the marginal density of X is

f_X(x) = 2x^2 for ≤ x ≤ 1.

Properties of Joint Distributions and Conditional Probability

Joint distributions have several important properties, including independence, covariance, and correlation. Two random variables X and Y are independent if and only if the joint density function f_XY(x,y) can be expressed as the product of the marginal densities f_X(x) and f_Y(y). That is:

f_XY(x,y) = f_X(x) * f_Y(y)

Covariance and correlation measure the strength of the linear relationship between two random variables. The covariance of X and Y is defined as:

Cov(X,Y) = E[(X - E[X])(Y - E[Y])] 

And the correlation coefficient is defined as:

ρ(X,Y) = Cov(X,Y) / (σ_X * σ_Y)

Where σ_X and σ_Y are the standard deviations of X and Y, respectively.

Independent and Dependent Events

Two events A and B are independent if and only if the probability of A occurring is not affected by the occurrence of B. Mathematically, this is expressed as:

P(A | B) = P(A)

Conversely, two events A and B are dependent if the probability of A occurring is affected by the occurrence of B. The conditional probability of A given B is defined as:

P(A | B) = P(A and B) / P(B) 

Example: Suppose we have two coins, one fair and one biased with a probability of heads of 3/4. We choose one of the coins at random and flip it twice. Let A be the event that the first flip is heads and B be the event that the second flip is heads. Then:

P(A and B) = P(A | fair) * P(fair) + P(A | one-sided) * P(biased)
= 1/2 * 1/2 + 3/4 * 1/2
= 5/8

P(B) = P(B | fair) * P(fair) + P(B | biased) * P(biased)
= 1/2 * 1/2 + 3/4 * 1/2
= 5/8

In this manner, the conditional likelihood of A given B is:

P(A | B) = P(A and B) / P(B)
= (1/2 * 1/2) / (5/8)
= 2/5

Conditional Expectation

Conditional expectation is a concept in probability theory that extends the idea of expected value to conditional probabilities. It is defined as the expected value of a random variable given the value of another random variable. Let X and Y be two random variables, and let P(X, Y) be their joint distribution. The conditional expectation of X given Y = y is defined as:

E[X | Y = y] = ∑x x P(X = x | Y = y)

The conditional expectation of X given Y is a function of Y and is denoted by E[X | Y]. It can be thought of as the expected value of X if we know the value of Y.

Example:

Suppose we have two dice, one red and one green. We roll the red die first, and then we roll the green die. Let X be the result of rolling the red die, and let Y be the sum of the two dice. Find E[X | Y = 7].

We can use the joint distribution of X and Y to calculate E[X | Y = 7]:

P(X = x, Y = y) = 1/36 for x = 1, 2, ..., 6 and y = 2, 3, ..., 12

P(Y = 7) = P(X = 1, Y = 7) + P(X = 2, Y = 7) + ... + P(X = 6, Y = 7)
                 = 6/36 
                 = 1/6

P(X = x | Y = 7) = P(X = x, Y = 7) / P(Y = 7) for x = 1, 2, ..., 6

E[X | Y = 7] = ∑x x P(X = x | Y = 7)
                         = (1/6) * (1/6 + 2/36 + 3/36 + 4/36 + 5/36 + 6/36)
                         = 4

Therefore, if the sum of the two dice is 7, the expected value of the result of rolling the red die is 4.

Covariance

Covariance is a measure of the linear relationship between two random variables. It is defined as:

Cov(X, Y) = E[(X - E[X]) * (Y - E[Y])] where E[X] and E[Y] 

are the expected values of X and Y, respectively. If Cov(X, Y) > 0, then X and Y have a positive linear relationship, meaning that when X is above its expected value, Y tends to be above its expected value as well, and when X is below its expected value, Y tends to be below its expected value as well. If Cov(X, Y) < 0, then X and Y have a negative linear relationship, meaning that when X is above its expected value, Y tends to be below its expected value, and vice versa. If Cov(X, Y) = 0, then X and Y are uncorrelated, meaning that there is no linear relationship between them.

Example: Suppose we have two random variables X and Y with the following joint distribution:

We can calculate the covariance of X and Y as follows:

E[X] = (1/12)*1 + (1/6)*2 + (1/12)*1 + (1/3)*2 + (1/6)*1 + (1/6)*2 = 5/3

E[Y] = (1/12)*1 + (1/6)*1 + (1/12)*2 + (1/3)*1 + (1/6)*2 + (1/6)*2 = 7/3

E[XY] = (1/12)*0 + (1/6)*1 + (1/12)*2 + (1/3)*2 + (1/6)*2 + (1/6)*4 = 5/3

Cov(X, Y) = E[(X - E[X]) * (Y - E[Y])]
                    = (1/12)*(1 - 5/3)*(0 - 7/3) + (1/6)*(2 - 5/3)*(1 - 7/3)
                        + (1/12)*(1 - 5/3)*(2 - 7/3) + (1/3)*(2 - 5/3)*(2 - 7/3)
                        + (1/6)*(1 - 5/3)*(2 - 7/3) + (1/6)*(2 - 5/3)*(2 - 7/3)
                    = -1/9

Therefore, X and Y have a negative linear relationship, as their covariance is negative.

Applications of Joint Distributions and Conditional Probability

Joint distributions and conditional probability are important concepts in probability theory with many applications in different fields, including:

• Finance:

Joint distributions are used to model the behavior of financial assets, such as stocks, bonds, and commodities. For example, the joint distribution of the returns of two stocks can be used to calculate their covariance, which is a measure of their co-movement.

• Engineering:

Joint distributions are used in reliability analysis to model the behavior of complex systems, such as aircraft engines, nuclear power plants, and communication networks. For example, the joint distribution of the lifetimes of the components of a system can be used to calculate the system's reliability.

• Epidemiology:

Joint distributions are used to model the spread of infectious diseases in populations. For example, the joint distribution of the number of susceptible, infected, and recovered individuals in a population can be used to calculate the basic reproduction number of the disease, which is a measure of its transmission potential.

• Machine learning:

Conditional probability is used in machine learning to model the relationship between input and output variables in a dataset. For example, the conditional probability of a certain output given a certain input can be used to predict the output for new inputs.

• Statistics:

Joint distributions and conditional probability are used in statistical inference to make inferences about populations based on samples. For example, the joint distribution of the sample mean and sample variance can be used to construct confidence intervals for the population mean and variance.

These are just a few examples of the many applications of joint distributions and conditional probability in different fields. They are powerful tools that allow us to model complex systems and make predictions based on data.

Conclusion

Joint distributions and conditional probability are essential concepts in probability theory that allow us to model and analyze complex systems with multiple variables. By understanding these concepts and their applications, we can make more informed decisions and predictions in a wide range of fields, from finance to engineering to healthcare.

Key Takeaways

• Joint probability distributions are used to describe the probability of two or more events occurring simultaneously.
• The marginal probability distribution is the probability distribution of one variable in a joint distribution, ignoring the other variables.
• Conditional probability is the probability of an event occurring given that another event has occurred.
• The conditional probability distribution is a probability distribution of one variable in a joint distribution, given a fixed value of another variable.
• The conditional expectation is the expected value of a random variable given the occurrence of another event or the knowledge of another random variable.
• The covariance of two random variables measures the degree to which they vary together.
• Joint distributions and conditional probability have many applications in various fields, including finance, engineering, epidemiology, machine learning, and statistics.
• Understanding joint distributions and conditional probability is essential in modeling complex systems and making predictions based on data.

Quiz

1. What is the purpose of a marginal probability distribution in a joint distribution? 

A) To determine the probability of two or more events occurring simultaneously 

B) To determine the probability of one variable in a joint distribution, ignoring the other variables 

C) To determine the probability of an event occurring given that another event has occurred 

D) To determine the expected value of a random variable given the occurrence of another event

Answer: B) To determine the probability of one variable in a joint distribution, ignoring the other variables

2. What is the conditional probability distribution in a joint distribution? 

A) A probability distribution of one variable in a joint distribution, given a fixed value of another variable 

B) The probability of an event occurring given that another event has occurred 

C) The expected value of a random variable given the occurrence of another event 

D) The probability distribution of two or more events occurring simultaneously

Answer: A) A probability distribution of one variable in a joint distribution, given a fixed value of another variable

3. What is the conditional expectation? 

A) The probability of an event occurring given that another event has occurred 

B) The expected value of a random variable given the occurrence of another event 

C) The degree to which two random variables vary together 

D) The probability distribution of one variable in a joint distribution, ignoring the other variables

Answer: B) The expected value of a random variable given the occurrence of another event

4. What is the covariance of two random variables? 

A) The probability of two or more events occurring simultaneously 

B) The expected value of a random variable given the occurrence of another event 

C) The degree to which two random variables vary together 

D) The probability distribution of one variable in a joint distribution, given a fixed value of another variable

Answer: C) The degree to which two random variables vary together