To continuous distritbutions, confidence interals, and hypothesis testing
Tue, Feb 13, 2024
Plotting a histogram is easy with Pandas/Python.
With a little more work, we can illustrate how the histogram is tied to data and the normal distribution
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
# Extract 2000 rows
df = df.iloc[0:2000]
# and plot
df.hist(column = ['height'], grid=False,
edgecolor='black', figsize = (12,6), density=True);
# Plot points below the histogram
z = df.height.apply(lambda z: -0.001)
plt.plot(df.height.apply(lambda x: x+np.random.random()), z, 'ok', alpha = 0.02)
# Compute and plot the normal pdf
m = df.height.mean()
s = df.height.std()
x = np.linspace(50,85,500)
y = norm.pdf(x,m,s)
plt.plot(x,y, '-');Pushing the previous slide a bit further, suppose we choose a number uniformly at random from an interval \(I\). The probability of picking a number from a sub-interval should be proportional to the length of that sub-interval.
If the big interval \(I\) has length \(L\) and the sub-interval \(J\) has length \(\ell\), then I guess we should have
\[P(X \text{ is in } J) = \ell/L.\]
A common way to visualize continuous distributions is to draw the graph of a curve in the top half of the \(xy\)-plane. The probability that a random value \(X\) with that distribution lies in an interval is then the area under the curve and over that interval. This curve is often called the density function of the distribution.
In the case of the uniform distribution over an interval \(I\), the “curve” is just a horizontal line segment over the interval \(I\) at the height 1 over the length of \(I\). In the picture below, for example, \(I=[0,2]\). On the left, we see just the density function for the uniform distribution over \(I\). On the right, we see the area under that density function and over the interval \([0.5,1]\). The area is \(1/4\) since \[P(0.5<X<1) = 1/4.\]
There should be an obvious relationship between the continuous uniform distribution and the discrete uniform distribution. The picture below illustrates this relationship and also helps us see how a continuous distribution might arise as a limit of discrete distributions.
As another simple example, let’s take \[f(x) = \begin{cases} \frac{3}{10}(x^2+1)(2-x) & 0<x<2 \\ 0 & \text{else}.\end{cases}\] It’s not too hard to show that \[\int_{-\infty}^{\infty} f(x) \ dx = \int_0^2 \frac{3}{10}(x^2+1)(2-x) \ dx = 1.\] If \(X\) has distribution \(f\) and we want to know, for example, \(P(\frac{1}{2}<X<1)\), we can compute directly: \[P(\frac{1}{2} < X < 1) = \int_{1/2}^1 \frac{3}{10}(x^2+1)(2-x) \ dx = \frac{187}{640} = 0.2921875.\]
Many important continuous distributions are positive over an unbounded interval. The exponential distribution, for example, has the form
\[f_{\lambda}(x) = \begin{cases} \lambda e^{-\lambda x} & x > 0 \\ 0 & \text{else}.\end{cases}\]
Note that the exponential distribution depends on a parameter, \(\lambda\). No matter what the value of \(\lambda\), though, we have
\[\int_{-\infty}^{\infty} f_{\lambda}(x) \ dx = \int_0^{\infty} \lambda e^{-\lambda x} \ dx = 1.\]
The larger \(\lambda\), the more concentrated the distribution is near zero:
Probability and statistics are rife with acronyms. Here are a couple of important ones:
You might see Density replaced with Distribution; the preferred reading can be context dependent. The essential difference between PDF and CDF, though, is that PDF is local, while CDF is cumulative. Put another way, the PDF \(f\) is the function that you integrate to get the CDF \(F\):
\[F(x) = \int_{-\infty}^x f(\chi) \ d\chi.\]
The the CDF should always be continuous and non-decreasing with \[\lim_{x\to -\infty} F(x) = 0 \: \text{ and } \: \lim_{x\to\infty} F(x) = 1.\]
Here’s the CDF for the exponential distribution with \(\lambda = 1\):
