Propensity scores: matching, weighting and what they cannot fix
A propensity score balances the variables you measured. It does nothing about the ones you did not, which in observational research are usually the ones that mattered.
A propensity score is the estimated probability of receiving the treatment given the measured covariates. Comparing treated and untreated patients with similar scores approximates comparing patients who were equally likely to be treated.
It is a way of handling many measured confounders at once. It is not a substitute for randomisation, and the distinction is where most of the trouble comes from.
The methods, and what each estimates
- Matching pairs treated patients with untreated ones of similar score. Unmatched patients are discarded, so the estimate applies to the matched population — usually the average effect among the treated.
- Inverse probability of treatment weighting reweights everyone so that measured covariates are balanced across the whole sample. Nobody is discarded, but extreme weights can let a few patients dominate.
- Stratification divides the sample into score bands and pools within-band estimates. Simple, and residual imbalance within bands is a known limitation.
- Covariate adjustment on the score is the weakest of the four and depends most heavily on the model being right.
These estimate different quantities. Which one you want is a question about the decision the study informs, and it should be stated.
Checking balance, correctly
Standardised mean differences, before and after, for every covariate. Below 0.1 is the usual working threshold.
Not hypothesis tests. A p-value comparing groups after matching is a function of the sample size you have left, so discarding patients makes imbalance look better while making the study smaller. The standardised difference does not have that property.
Report the balance table. A propensity score analysis without one asks the reader to trust that it worked.
What it cannot fix
Unmeasured confounding. If sicker patients were preferentially treated on the basis of clinical judgement that never entered the record, no function of the recorded variables can balance it.
This is not a marginal caveat. In observational treatment comparisons, the reason for treatment is usually related to prognosis, and much of that reasoning is not in the data.
Strengthening the analysis
- A negative control outcome — something the treatment could not plausibly affect. If it shows an effect, residual confounding is present.
- A quantitative bias analysis: how strong would an unmeasured confounder have to be to explain the result? The E-value makes this concrete.
- An active comparator rather than untreated patients, which removes much of the confounding by indication.
- A new-user design, which avoids the survivor bias built into prevalent-user comparisons.
Frequently asked questions
What does a propensity score do?
It estimates the probability of receiving the treatment given measured covariates, so that treated and untreated patients with similar scores can be compared. It handles many measured confounders at once but does nothing about unmeasured ones.
How do you check balance after propensity score matching?
With standardised mean differences for every covariate, before and after, using 0.1 as the usual threshold. Not with hypothesis tests: a p-value after matching depends on how many patients remain, so discarding patients makes imbalance look better.
Can propensity scores replace randomisation?
No. They balance what was measured. In observational treatment comparisons the reason for treatment is usually related to prognosis, and much of that reasoning never enters the record.