Correlation vs Causation
A spreadsheet column of hours slept and a column of exam scores drift up together, and `corr()` hands you one number for the whole pattern, but that number stays silent about which one moved the other.
- ▸Standardize two columns, multiply point by point, average: one unit-free score that compares across any units.
- ▸Four datasets can share one correlation yet look nothing alike, and a single far-off point can prop it up alone.
- ▸The same number fits cause, reverse-cause, a shared driver, or luck, and only random assignment rules three out.
Line up two columns for the same ten students: hours slept before an exam, and the exam score each one earned. Plot one against the other and the dots drift up to the right, more sleep, higher score. But "drifts up to the right" is a feeling, not a number. You want one value that says how tightly the two columns track together, and says it in a form you could compare against a completely different pair of columns measured in completely different units.
The trick reuses standardization from the last topic. Rewrite each column as z-scores: subtract that column's mean, divide by that column's standard deviation, so both columns lose their units and center on zero. Now walk the students one at a time and multiply each student's two z-scores together. A student above average on both, or below average on both, contributes a positive product. A student above on one and below on the other contributes a negative product.
Average those products across all ten students and you have the correlation coefficient: one unit-free number that runs large and positive when the columns rise together, large and negative when one rises as the other falls, and sits near zero when they wander independently. It is nothing more exotic than the average of standardized products, but it is the single most reported summary of how two columns relate.