Method used to calculate the multivariate joint distribution.
Today I got a friend’s question about probability:
“ Random variables \(X\) and \(Y\) are independent where \(X\) and \(Y\) are Gamma distributions with different scale parameters but same rate parameter. Proof that \(Y\) and \(X/Y\) are independent. “
This reminds me of multivariate transformation which is used in calculating the joint density of functions of multi-random variables. So let’s get started.
Since \(X\) and \(Y\) are independent, therefore the joint density is
\[
p(x, y) = \frac{\beta ^{\alpha_{1}}}{\Gamma(\alpha_{1})}e^{-\beta x}x^{\alpha_{1}-1} \frac{\beta ^{\alpha_{2}}}{\Gamma(\alpha_{2})}e^{-\beta y}y^{\alpha_{2}-1} = \frac{\beta ^{\alpha_{1}+\alpha_{2}}}{\Gamma(\alpha_{1})\Gamma(\alpha_{2})}e^{-\beta(x+y)}x^{\alpha_{1}-1}y^{\alpha_{2}-1}.
\]
Applying the same rule, we know if \(X+Y\) and \(X/Y\) are independent, their joint density could be expressed like this
\[
p(X+Y, X/Y) = g(X+Y)h(X/Y).
\]
Therefore if we can get the joint density of \(X+Y\) and \(X/Y\) and factorize it into two functions, we are able to solve it. Here comes the Multivariate transformation theorem.
We denote \(U=f_{1}(x, y) = X+Y\) and \(V=f_{2}(x, y) = X/Y\), then we inverse the transformation:
\[
X = g_{1}(u, v) = \frac{UV}{1+V}, Y = g_{2}(u, v) = \frac{U}{1+V}.
\]
Then the joint density of \(U\) and \(V\) is
\[
p_{UV}(u, v) = p_{XY}(g_{1}(u, v), g_{2}(u, v))|J|, \text{ for } u, v \text{ on their support},
\]
where \(|J|\) is the absolute value of determinant of Jacobian,
\[
J=
\begin{vmatrix}
\frac{\partial g_{1}}{\partial u} & \frac{\partial g_{1}}{\partial v}\\
\frac{\partial g_{2}}{\partial u} & \frac{\partial g_{2}}{\partial v}
\end{vmatrix}.
\]
Actually the above transformation is just an extension from univariate to multivariate, remember in unvariate case we have:
“ Let \(X\) be a continuous (while it’s easier for discrete case) random variable having density \(p_{X}(x)\), then the density of function of random variable \(X\), \(Y=g(x)\), is
\[
p_{Y}(y) = p_{X}(g^{-1}(y))\Bigg|\frac{dg^{-1}(y)}{dy}\Bigg|, \text{ for } y = g(x).
\]
“
Back to our problem, we solved that the Jacobian is
\[
|J| =
\begin{vmatrix}
\begin{vmatrix}
\frac{v}{1+v} & \frac{u}{(1+v)^2} \\
\frac{1}{1+v} & \frac{-u}{(1+v)^2}
\end{vmatrix}
\end{vmatrix}
= \frac{u}{(1+v)^2}.
\]
Then
\[
\begin{aligned}
p_{UV}(u, v) =& \frac{\beta ^{\alpha_{1}+\alpha_{2}}}{\Gamma(\alpha_{1})\Gamma(\alpha_{2})}e^{-\beta(u)}\left(\frac{uv}{1+v}\right)^{\alpha_{1}-1}\left(\frac{u}{1+v}\right)^{\alpha_{2}-1}\frac{u}{(1+v)^2} \\
\propto& u^{\alpha_{1}+\alpha_{2}}e^{-\beta u}\frac{v^{\alpha_{1}}}{(1+v)^{\alpha_{1}+\alpha_{2}}} \\
=& g(u)h(v).
\end{aligned}
\]
QED.
And since here \(X\) and \(Y\) are independent, we can also apply convolution, for example,
\[
p_{U}(u) = \int^{\infty}_{0}p_{X}(u-y)p_{Y}(y)dy.
\]
Update on 10/4/2017
There’s a theorem relate to this practice (thanks to the notes from Advanced Machine Learning).
- Luckacs’ theorem:
If \(X\) and \(Y\) are independent random variables (non-negative and not constant), then
\[
\frac{X}{X+Y}\text{ is independent with } X+Y
\]
if and only if \(X\) and \(Y\) are Gamma distribution with same shape parameter \(\beta\) (or sometimes \(\lambda\)).