Risk of bias: choosing a tool and using it properly
Risk of bias is an assessment of whether a study's result can be believed, not a score of how well it was written up. The distinction changes how the assessment is done.
A risk-of-bias assessment asks whether features of a study's design and conduct could have produced its result independently of any real effect. It is not a measure of reporting quality, and it is not a mark out of ten.
Choose by design, not by convenience
- Randomised trials: RoB 2, which assesses bias by domain for a specified result rather than for the study as a whole.
- Non-randomised studies of interventions: ROBINS-I, which starts by asking what target trial the study is trying to emulate.
- Diagnostic accuracy: QUADAS-2.
- Cohort and case-control studies in a review not using ROBINS-I: the Newcastle-Ottawa Scale is common, though its star system is widely criticised for producing a total that has no clear interpretation.
The mistake that undermines the whole assessment
Judging a study as high risk because the report is thin.
Absent information is not the same as bias. It means the risk is unclear, and "unclear" is a legitimate and informative judgement. Recording it as high risk conflates two different problems and makes the assessment less useful, because a reader can no longer tell which studies were badly conducted and which were badly described.
Assess the result, not the paper
A trial may be at low risk of bias for its primary outcome, which was objective and completely reported, and at high risk for a secondary outcome assessed unblinded with substantial missing data.
RoB 2 is built around this: the assessment attaches to a specific result. Reporting one overall judgement per study discards most of the information.
Two assessors, and a record of the disagreements
Independent duplicate assessment is standard because these judgements involve interpretation. What matters as much is documenting where the assessors disagreed and how it was resolved — that record tells a reader which domains were genuinely ambiguous.
Use it in the synthesis, or do not bother
A risk-of-bias table that appears in the appendix and affects nothing is decoration.
Use it: restrict the primary analysis to low-risk studies, or run a sensitivity analysis excluding high-risk ones, or stratify. If the pooled estimate is driven by studies at high risk of bias, that is the most important thing a reader could know about the review, and it should be in the abstract rather than the supplement.
Frequently asked questions
Which risk-of-bias tool should I use?
RoB 2 for randomised trials, ROBINS-I for non-randomised studies of interventions, QUADAS-2 for diagnostic accuracy. The Newcastle-Ottawa Scale is common for cohort and case-control studies, though its star total has no clear interpretation.
Is a poorly reported study at high risk of bias?
No. Absent information means the risk is unclear, which is a legitimate and informative judgement. Recording it as high risk conflates bad conduct with bad description, and a reader can no longer tell which is which.
Should risk of bias be assessed per study or per outcome?
Per result. A trial can be at low risk for an objective, fully reported primary outcome and at high risk for a secondary outcome assessed unblinded with missing data. One overall judgement per study discards most of the information.