Discrete Random Vectors

• Discrete Random Vectors
  • Joint Probability Mass Function
  • Marginal Densities
  • Mean of Random Vector
  • Covariance
    • Computing Covariance
  • Correlation Coefficient
  • Independent Random Variables
  • Conditional PMF
    • Independence
  • Conditional Mean and Variance
    • Independence
  • Conditional Mean As A Function
  • Two Step Average
  • Total Variance

We are interested in joint behavior of multiple random variables. An example of random vector would be rolling two dies and we are looking at the joint behavior of two random variables.

$$\mathbf{x}=\left(\begin{array}{c}{X}_1\\ X_2\end{array}\right)=\left(\begin{array}{c}\text{sum of points}\\ \text{difference of points}\end{array}\right)$$

There are continuous and discrete random variables.

Discrete Random Vectors

Joint Probability Mass Function

The probability for a realization of all considered random variables.

$$f(x_1{x}_2,\dots ,x_m)=\mathbb{P}(X_1=x_1,X_2=x_2,\dots ,X_m=x_m)$$

Example: consider the random vector from above (rolling two dies)

Out of all possibilities, sum of points being 6 has 6 possible outcomes: (1, 5), (2, 4), (3, 3), (4, 2), (1, 6). Thus the probability of $X_1=6$ is $\frac{5}{36}$. Then out of those possibilities, only (3, 3) has the absolute difference 0, so $\mathbb{P}(X_2=0|X_1=6)=\frac{1}{5}$.

Therefore the joint probabilities is $\mathbb{P}(X_1=6,X_2=0)=\frac{1}{36}$

Marginal Densities

PMF that only takes interest in a single random variable in a random variable.

For instance if the random vector has 2 elements, then $f(x_1,x_2)$ is the discrete joint PMF function.

Then the marginal densities is the PMF with the other variables (not of interest) summed up.

$$f_1(x_1)=\sum _{x_2}f(x_1,x_2)$$ $$f_2(x_2)=\sum _{x_1}f(x_1,x_2)$$

Mean of Random Vector

The mean/expected value of a random vector just applies to each individual random variables.

$$\mathbb{E}(\mathbf{X})=\mathbb{E}\left(\begin{array}{c}{X}_1\\ ⋮\\ X_m\end{array}\right)=\mathbb{E}\left(\begin{array}{c}\mathbb{E}(X_1)\\ ⋮\\ \mathbb{E}(X_m)\end{array}\right)=\overrightarrow{\mu }$$

Covariance

The variance of the random vector can be described as covariance, in a covariance matrix.

$$\text{Cov}(\mathbf{X})=\mathbb{E}[(\mathbf{X}-\mu )(\mathbf{X}-\mu {)}^T]$$

Where $\mathbf{X}-\mu$ the random vector subtracting the mean of the random vector, and $(\mathbf{X}-\mu {)}^T$ is the transpose. This results in a matrix multiplication that returns a matrix:

$$\text{Cov}(\mathbf{X})=\left[\begin{array}{ccccc}{\mathit{\sigma }}_{11}& {\sigma }_{12}& {\sigma }_{13}& \cdots & {\sigma }_{1m}\\ {\sigma }_{21}& {\mathit{\sigma }}_{22}& {\sigma }_{23}& \cdots & {\sigma }_{2m}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{m1}& {\sigma }_{m2}& {\sigma }_{m3}& \cdots & {\mathit{\sigma }}_{mm}\end{array}\right]$$

Where for any $i$,

$${\sigma }_{ii}=\mathbb{E}[(X_i-{\mu }_i{)}^2]=\text{Var}(X_i)$$

And for any $i$ and $j$,

$${\sigma }_{ij}=\mathbb{E}[(X_i-{\mu }_i)(X_j-{\mu }_j)]=\text{Cov}(X_i,X_j)$$

The covariance of the two individual random variable is

$$\text{Cov}(X_i,X_j)=\mathbb{E}[(X_i-{\mu }_i)(X_j-{\mu }_j)]=\mathbb{E}[(X_i-{\mu }_i)]\mathbb{E}[(X_j-{\mu }_j)]$$

Computing Covariance

$$\begin{array}{rl}{\sigma }_{ij}& =\mathbb{E}[(X_i-{\mu }_i)(X_j-{\mu }_j)]\\ & =\mathbb{E}[X_i{X}_j]-\mathbb{E}[X_i]{\mu }_j-\mathbb{E}[X_j]{\mu }_i+{\mu }_i{\mu }_j\\ & =\mathbb{E}[X_i{X}_j]-{\mu }_i{\mu }_j-{\mu }_i{\mu }_j+{\mu }_i{\mu }_j\\ & =\boxed{{\displaystyle \mathbb{E}[X_i{X}_j]-{\mu }_i{\mu }_j}}\end{array}$$

Correlation Coefficient

It describes how correlated two random variables are. It is defined as follows.

$${\rho }_{ij}=\frac{{\sigma }_{ij}}{\sqrt{{\sigma }_{ii}{\sigma }_{jj}}}=\frac{\text{Cov}(X_i,X_y)}{\text{SD}(X_i)\text{SD}(X_j)}$$

Where $\rho \in [-1,1]$.

Independent Random Variables

The random variables $X_1,X_2,\dots ,X_m$ are independent from each other if and only if

$$f(x_1,x_2,\dots ,x_m)=f_1(x_1)f_2(x_2)\dots f_m(x_m)$$

Consider random variables $X$ and $Y$, if they are independent then,

$$\mathbb{E}(X_i{X}_j)=\mathbb{E}(X_i)\mathbb{E}(X_j)$$

Covariance is 0 if they are independent.

$${\sigma }_{XY}=\mathbb{E}(XY)-\mathbb{E}(X)\mathbb{E}(Y)=0$$

Note that independence of $X$ and $Y$ $\Rightarrow {\sigma }_{XY}=0$, but ${\sigma }_{XY}\nRightarrow$ that $X\$and$Y$$ are independent.

Conditional PMF

Consider two random variables $X_1$ and $X_2$. Then the conditional PMF is:

$$f(x_2|x_1)=\frac{f(x_1,x_2)}{f_1(x_1)}$$

Thus, it’s clear to see that

$$\begin{array}{rl}f(x_1,x_2)& =f_1(x_1)f(x_2|x_1)\\ f(x_1,x_2)& =f_2(x_2)f(x_1|x_2)\end{array}$$

Independence

If $X_1$ and $X_2$ are independent, then

$$f(x_1,x_2)=f_1(x_1)f_2(x_2)$$

It follows that

$$f(x_2|x_1)=f_2(x_2)$$

and similarly

$$f(x_1|x_2)=f_1(x_1)$$

Conditional Mean and Variance

Conditional Mean is the expected value of one random variable given the realization of another random variable.

$${\mu }_{y|x}=\mathbb{E}(Y|X=x)=\boxed{{\displaystyle \sum _{y}yf(y|x)}}$$

Conditional Variance:

$${\sigma }_{y|x}^2=\text{Var}(Y|X=x)=\sum _y(y-{\mu }_{y|x}{)}^{2}f(y|x)=\boxed{{\displaystyle \mathbb{E}(Y^2|X=x)-{\mu }_{y|x}^2}}$$

Independence

If events $X$ and $Y$ are independent, then:

• Conditional PMF $\equiv$ marginal PMF
  • Since $f(y|x)=f_Y(y)$ and $f(x|y)=f_X(x)$
• Conditional means and variances $\equiv$ marginal means and variances
  • $\mathbb{E}(Y|X=x)=\mathbb{E}(Y)$ and $\text{Var}(Y|X=x)=\text{Var}(Y)$
  • $\mathbb{E}(X|Y=y)=\mathbb{E}(X)$ and $\text{Var}(X|Y=y)=\text{Var}(X)$
• $$\mathbb{E}(g(X,Y)|X=x)=\mathbb{E}(g(x,Y))$$
  • An example would be $\mathbb{E}(e^{X+Y}|X=x)=e^x\mathbb{E}(e^Y)$

Conditional Mean As A Function

Generally the conditional mean is expressed as $\mathbb{E}(Y|X=x)$, which is a function of $x$. Thus, we can express that as:

$$h(x)=\mathbb{E}(Y|X=x)$$

Since $x$ is only a realization of a random variable, we can consider $h(x)$ as a random function as:

$$h(X)=\mathbb{E}(Y|X)$$

Two Step Average

We may find the conditional mean of a random variable in two steps.

$$\mathbb{E}(Y)=\mathbb{E}(\mathbb{E}(Y|X))$$

Where $\mathbb{E}[Y]=\mathbb{E}[g(X,Y)]$ and $\mathbb{E}[\mathbb{E}[Y|X]]=\mathbb{E}[\mathbb{E}[g(X,Y)|X]]$. It follows that

$$=\sum _x(\sum _{y}g(x,y)f(y|x))f_X(x)$$

Total Variance

Consider the following plot of two random variables $X$ and $Y$. Where the red circles represent $\mathbb{E}(Y|X=x)$ and the red whiskers represents the variance $\text{Var}(Y|X=x)$. Then the blue circle is the overall mean $\mathbb{E}(Y)$ and the blue whisker is the overall variance $\text{Var}(Y)$.

In this case, the Total Variance is given by

$$\begin{array}{rl}\text{Var}(Y)& =\text{Unexplained variance}+\text{Explained variance}\\ & =\mathbb{E}[\text{Var}(Y|X)]+\text{Var}[\mathbb{E}(Y|X)]\end{array}$$

In the example above, the unexplained variance is the inner variance (red whiskers). The explained variance is the variance due to the red dots increasing.

The Percentage of Explained Variance is given by:

$$\frac{\text{Explained variance}}{\text{Total variance}}\times 100\mathrm{\%}$$

This gives an idea of how good a prediction is. If the percentage of explained variance is closer to 1, then one could use it for a good prediction. On the other hand, a percentage of near 0 is useless.

For the best prediction, use $g(x)=\mathbb{E}(Y|X=x)$.