How To Analyze Systematic Review Data | Plan Pool Judge

To analyze systematic review data, set outcomes, extract effect sizes, check bias, pick a model, test heterogeneity, grade certainty.

Why This Guide Works

Systematic reviews pull many trials into an answer. The data step decides how sturdy that answer is. This guide lays out a path to turn extracted numbers into findings a reader can trust.

Pick the right effect size, check study quality, choose a model, gauge inconsistency, probe bias, state certainty.

Analyzing Data In Systematic Reviews: The Core Steps

Step 1 — Lock The Outcomes And Effect Measures. Nail down each primary and secondary outcome before crunching numbers. Match each outcome to a suitable effect size and a consistent time point. Keep units aligned across studies, or convert them with a rule you apply the same way every time.

Common Effect Sizes And When To Use Them

Outcome Type	Effect Size	Use When
Dichotomous (yes/no)	Risk Ratio (RR) or Odds Ratio (OR)	Events per group are reported; RR reads clean for most readers; OR suits rare events or case–control data.
Time-to-event	Hazard Ratio (HR)	Follow-up varies; survival or failure time is the focus.
Continuous, same scale	Mean Difference (MD)	All studies use the same units (e.g., mmHg or minutes).
Continuous, different scales	Standardized Mean Difference (SMD)	Scales differ; you need a unit-free effect.
Rates	Rate Ratio	Events can repeat per person-time.

Step 2 — Extract Clean, Consistent Data. Set up a sheet that captures group sizes, events, means, SDs, and follow-up. Record the analysis set used in each paper. Double data entry or a second check reduces slips. Note imputed values, conversions, and any contact with authors.

Step 3 — Judge Risk Of Bias Study By Study. Review randomization, deviations from the plan, missing data, outcome measurement, and selective reporting. Tag each domain and keep notes on why you chose that tag. You’ll need those notes for sensitivity runs and the final certainty grade.

Step 4 — Choose A Model Before You Peek At I². A common-effect (often called fixed-effect) model treats the true effect as the same across studies. A random-effects model treats the true effect as a distribution. Pick the approach that fits your question and the mix of populations and methods, and write it in your protocol. Don’t flip models later just because I² looks high.

For random-effects, record which between-study variance estimator you use (e.g., REML, Paule-Mandel, or DerSimonian–Laird) and whether you apply Hartung–Knapp. List these choices in the methods so anyone can rerun your code and get the same numbers.

Step 5 — Check Statistical Heterogeneity. Report the Q statistic, tau², and I² with confidence intervals if your software provides them. I² near zero hints at little inconsistency; higher values suggest wider spread beyond chance. Numbers alone don’t tell the whole story, so always add a short note on clinical or method mix that could explain the spread.

Step 6 — Look For Small-Study And Publication Bias. If you have a decent set of studies, a funnel plot might help. When the plot looks skewed, run tests for asymmetry, yet treat p-values with care, especially with few studies. Use trim-and-fill or selection models as probes, not as the last word.

Step 7 — Plan Subgroups And Meta-Regression Sparingly. Limit subgroup checks to a short list set in the protocol. Use within-study contrasts when you can. For meta-regression, keep the number of covariates low, use continuous covariates when they make sense, and avoid kitchen-sink models that fit noise.

Step 8 — Run Sensitivity Analyses. Re-fit the model after removing high-risk studies, switching effect sizes, changing tau² estimators, or applying Hartung–Knapp. A leave-one-out plot or table shows whether a single outlier drives the pool. Stable findings across these runs build confidence.

For deeper method notes, the Cochrane Handbook chapter on meta-analysis lays out effect sizes, models, and heterogeneity tools. For write-up and transparency, use the PRISMA 2020 checklist during drafting so every core item ends up in the paper.

Prepare Your Data And Plan The Synthesis

Define the unit of analysis. Decide whether you run the analysis per participant, limb, eye, or cluster. If cluster trials didn’t adjust, apply the design effect. If a study reports multiple measures of one construct, set a rule to pick one or combine them.

Align time windows. When time points differ, set windows around a target and extract the closest match. If studies report both adjusted and unadjusted results, choose one approach across the board.

Handle missing or odd formats. Convert medians and IQRs only when a valid method fits and the distribution looks near-symmetric. If scale directions differ, flip signs so higher always means the same thing. Log every change.

Run The Meta-Analysis And Report It Cleanly

Build the forest plot. Show study IDs, weights, effect sizes with CIs, and the pooled line. Add a minimal clinically meaningful difference when one exists. Pair relative effects with absolute numbers so readers can see real-world impact.

Explain your model choice. If you used a common-effect model, say you assumed one true effect and why. If you used random-effects, say true effects vary and the pool is an average. State the estimator and any small-sample tweaks clearly.

Describe the heterogeneity story. Report I² and tau² and add a one-paragraph take on sources of spread. If spread stays wide, say the average should be read with care and show prediction intervals when you have them.

Grade The Certainty And Write The Takeaway

Use GRADE to rate how sure the body of evidence is. Judge risk of bias, inconsistency, indirectness, imprecision, and publication bias for each outcome. Say why you lowered or raised the rating, and link that rating to the size and direction of the effect. A short “Summary of findings” table that shows absolute risk, relative effect, and certainty helps readers see the full picture.

Balance effect size with certainty. A big relative effect with low certainty doesn’t carry the same weight as a modest effect with high certainty. Spell this out with numbers and plain words so decision makers don’t need to guess.

When Pooling Doesn’t Fit The Question

Not every review needs a pooled number. Skip meta-analysis when outcomes don’t match, when designs clash, or when bias and reporting gaps make a pool shaky. In those cases, follow a structured narrative plan: group studies by design or population, line up effect directions, and state where findings agree or split. You can still grade certainty for each outcome and point out research gaps.

Common Pitfalls And Simple Fixes

Switching models mid-stream. Set the model in the protocol and stick with it. If you change it, say why and show both results.

Merging apples and oranges. Pool only when the construct and measure align. If you must mix, use SMD and explain the direction.

Over-testing subgroups. Keep the list tight. Mark them as exploratory unless backed by strong theory and enough studies.

Ignoring absolute effects. Always pair relative measures with baseline risk to show real counts or minutes gained.

Thin methods. Readers should be able to rebuild your analysis. State software, packages, and version numbers.

A Simple Workflow You Can Reuse

Here’s a clean loop many review teams use:

1. Name outcomes, time points, effect sizes, and the model.
2. Build a tidy extraction sheet and a log for conversions.
3. Tag risk-of-bias domains with notes and page numbers.
4. Pool effect sizes; record tau², I², Q; save code.
5. Probe heterogeneity with the pre-set subgroups.
6. Run sensitivity checks (leave-one-out, bias-restricted set).
7. Inspect funnel shape when the dataset is large enough.
8. Build the summary table and grade certainty.
9. Write a short takeaway for each outcome.

Quick Guide: Signals And Useful Actions

Signal	What It Suggests	Action
High I² with overlapping CIs	Spread beyond chance; average may hide differences	Report prediction interval; check subgroups set a priori; check data entry
Asymmetric funnel plot	Small-study or reporting bias	Run sensitivity with trim-and-fill or selection models; state limits
One study dominates weight	Pool driven by a single large trial	Run leave-one-out; present both pooled and narrative reads
Critical risk-of-bias tags	Domains differ across studies	Sensitivity by bias level; rate certainty with those tags in mind
Different scales across trials	Mix of measures for same construct	Use SMD or convert to a common unit; keep direction consistent

Tools And Settings That Save Time

Stats software. R with the metafor or meta packages, Stata with metan, or RevMan for quick runs. Pick one stack, lock versions, and share code in a supplement.

Reproducible files. Keep raw extraction, a cleaned dataset, the script, and outputs in one folder. Use a seed for any resampling.

Visual checks. Plot study effects over time, by size, or by risk-of-bias tag. Outliers often jump out faster on a chart than in a table.

Plain-Talk Advice For Better Systematic Review Analysis

Write methods that a new teammate could follow next week. Say what you planned, what you did, and why any change made sense. Keep tables tidy, plots legible, and claims lined up with the numbers. Report both benefits and harms with the same care. If your pool is shaky, say so and show what would need to change for confidence to rise.

Do that, and your systematic review data won’t just sit in a spreadsheet. It will guide choices in clinics, labs, classrooms, and boards.