How to choose a statistical test

The usual request — "which test should I use for this data?" — cannot be answered from the data. The same dataset supports different analyses depending on what is being asked of it.

Work in this order instead.

1. State the question as a comparison

Not "is there a relationship between X and Y" but "how much does Y differ between people with and without X, among this population, over this period, holding what constant?"

If the question cannot be written that way, no test will rescue it.

2. Identify the outcome and its type

• Continuous, and roughly symmetric — means and differences in means.
• Continuous and skewed — consider whether the median is the clinically meaningful summary, or whether a transformation or a model on the natural scale is better than a rank test that answers a subtly different question.
• Binary — risks, risk differences, risk ratios, odds ratios.
• Time-to-event — survival methods, because censoring is information and ignoring it discards it.
• Counts — rates, with an offset for time at risk.
• Ordinal — ordered methods, rather than collapsing categories until the outcome becomes binary.

3. Identify the structure of the data

This is the step most often skipped, and the one that invalidates most analyses.

• Are observations independent? Repeated measures, multiple lesions per patient, patients within centres and matched pairs are all clustered, and a test assuming independence will report intervals that are too narrow.
• Are groups paired or unpaired?
• How many groups, and were the comparisons planned or chosen after seeing the data?

4. Decide what has to be adjusted for

In observational data, the comparison you want is rarely the comparison you observe. Which variables confound the relationship is a question about the causal structure, answered from clinical knowledge — not by which variables happen to reach significance in a univariable screen.

Adjusting for a variable on the causal pathway removes part of the effect you are trying to measure. Adjusting for a collider introduces bias that was not there before.

5. Only now, choose the method

By this point the choice is usually obvious, and often it is a regression model rather than a test, because a model can carry adjustment and produce an effect estimate rather than only a p-value.

Report the estimate, not the verdict

A p-value tells you how compatible the data are with a null hypothesis. It does not tell you how large the effect is, whether it matters clinically, or how uncertain it is.

Report the effect estimate with its confidence interval and interpret both. "No significant difference" in a small study frequently means "this study could not have detected a difference that would matter" — a statement about the study, not about the intervention.