To assess publication bias in systematic reviews, combine planned searches, small-study diagnostics, and ROB-ME judgments to gauge missing results.
Publication bias hides negative or null results and can tilt any pooled estimate. A careful plan, transparent searches, and the right diagnostics make that tilt visible. This guide lays out practical steps that fit real review workflows without bloated math or jargon.
Why Publication Bias Warps Your Review
When studies with eye-catching results surface while quieter studies sit in drawers, a meta-analysis can drift away from the truth. Selective outcome reporting, delayed publication, and language filters can add more blind spots. Detecting the pattern needs both design choices and analysis checks, not one quick plot.
Core guidance sits in the Cochrane Handbook chapter on missing evidence, which frames the problem and introduces ROB-ME for a structured judgment. PRISMA 2020 also asks authors to describe how risk of bias due to missing results was assessed and how it shaped the synthesis; see the PRISMA 2020 statement in BMJ.
Quick Screening And Deep Checks
| Step | What You Do | What You Record |
|---|---|---|
| Pre-specify | Write a protocol with outcomes, time points, and analysis plans. | Protocol link, deviations, and reasons. |
| Search widely | Include registries, preprints, and theses; harvest trials by outcome. | Sources searched and dates. |
| Map availability | List eligible studies and flag which outcomes are missing. | Study-outcome matrix. |
| Funnel basics | Draw funnel plots when you have at least ten studies. | Plot file and notes. |
| Small-study tests | Run a regression test matched to your effect type. | Test chosen and p-value. |
| Contours | Add p-value contours to the funnel. | Whether gaps sit in not-statistically-clear zones. |
| Model checks | Apply trim-and-fill or a selection model as sensitivity. | Adjusted effect and change size. |
| Contact authors | Ask about unreported outcomes or subgroup results. | Replies and any new data. |
| Rate ROB-ME | Judge risk of bias due to missing evidence. | Low, some concerns, or high; with rationale. |
Assessing Publication Bias In A Review: Step-By-Step
Start With A Protocol
Protocols reduce hindsight tweaks. Lock core decisions before screening begins: outcomes, comparators, follow-up windows, subgroups, and meta-analysis models. Register the plan, cite it, and keep a change log. Later, readers can see whether the analysis path drifted toward exciting results.
Search Beyond Journals
Journals are not the only gatekeepers. Search trial registries, regulatory documents, preprints, and dissertations. Scan conference books and funder reports. Build forward and backward citation chains. Where the topic is small or moving fast, email investigators; a short, specific request often surfaces a missing outcome table.
Check For Small-Study Effects
Plot each study’s effect against its standard error. With enough studies, symmetry points to a balanced evidence base. Skew can point to selective publication or to genuine differences across study sizes. Treat the plot as a signpost, not a verdict.
Pick The Right Test
Regression tests look for a link between study size and effect. For continuous outcomes, a common pick is Egger’s test. For odds ratios or risk ratios, use a method suited to binary data, such as Harbord or Peters tests. These tests have low power with few studies, so a liberal threshold like 0.10 is often used in practice, and results should be read with care when heterogeneity is wide.
Use Contour-Enhanced Funnel Plots
Overlay lines for common p-value cut-offs (0.10, 0.05, 0.01). If missing points cluster in the not-statistically-clear band, selective publication becomes a live suspect. If gaps fall in areas where results would still be statistically clear, asymmetry may reflect other forces such as design quirks or chance.
Probe With Selection Methods
Next, run sensitivity models that mimic how missing studies might alter the pooled effect. Trim-and-fill “fills” the funnel to restore symmetry and gives an adjusted estimate. Selection models specify rules where studies with smaller p-values are more likely to appear; they often fit well when publication bias drives the pattern. PET-PEESE and p-curve or p-uniform* offer added views on small-study patterns. None of these is a magic wand, so present them as sensitivity work, side by side with the main random-effects model.
Rate Risk Of Bias Due To Missing Results
ROB-ME gives a transparent way to judge the whole picture. Weigh what you saw in searches, outcome availability, plots, tests, and models. Then rate the synthesis as low risk, some concerns, or high risk. Explain the call in plain terms and state what it means for decision-making.
Interpreting And Reporting Without Spin
Readers want clarity on three things: what might be missing, how you looked for it, and what changed after sensitivity work. Say how many eligible studies lacked results for the main outcome. Name the diagnostics you used and why they fit the data type. If adjusted models shrink the effect, state the size of the shift and whether the clinical take-home would change.
Write up the methods and findings so a peer can reproduce the steps. Include plot images in the supplement, code snippets if you can, and a short paragraph that ties the judgment to any rating scale you use downstream.
Statistical Tests And When To Use Them
| Method | Best For | Watch Outs |
|---|---|---|
| Egger regression | Continuous outcomes; standardised mean differences. | Low power with few studies; type I error may rise with wide heterogeneity. |
| Harbord / Peters | Binary outcomes with odds ratios or risk ratios. | Sensitive to sparse data; match the test to the effect scale. |
| Trim-and-fill | Quick symmetry repair to gauge possible shift. | Assumes missingness drives asymmetry; can over-correct. |
| Selection models | Model publication processes and re-estimate effects. | Results hinge on model choice; fit can be unstable. |
| PET-PEESE | Meta-regression approach to small-study bias. | Needs enough studies; sensitive to outliers. |
| p-curve / p-uniform* | Shape of reported p-values for evidential value. | Requires exact p-values; narrow scope. |
Worked Example Outline
Say your review of exercise programs for knee pain includes 18 trials and the main outcome is pain at 12 weeks. You registered a protocol, searched registries and preprints, and built a matrix of eligible outcomes. Five trials lacked the 12-week pain outcome; three had trial records with the right time point but no matching table in the paper.
You drew a funnel plot of the standardized mean difference. The plot showed sparse points on the left at high standard errors. Egger’s test gave p=0.08. A contour overlay suggested gaps sit in the not-statistically-clear zone. Trim-and-fill added three studies and moved the pooled effect from −0.30 to −0.22. A selection model placed more weight on studies with small p-values and produced −0.20.
You then rated ROB-ME as “some concerns,” explained the likely under-reporting of null results at the main time point, and kept the adjusted estimate beside the primary model in the abstract. The clinical read did not flip, but the range of plausible benefit narrowed.
Sensitivity Moves That Raise Confidence
Run leave-one-out checks to spot undue influence by single small studies. Refit with a different τ² estimator. Limit to pre-registered trials or to studies at low risk of within-study bias. Add unpublished outcome tables where authors supplied them and rerun the synthesis. Where subgroups might differ by study size, add a meta-regression on standard error.
When You Have Fewer Than Ten Studies
Formal tests sink in power when the dataset is small. In that case, keep the emphasis on design. Show the study-outcome matrix, describe search reach, and report which outcomes were unavailable and why. Draw a funnel only as a descriptive figure, and state plainly that you are not running tests. Use leave-one-out runs and simple meta-regressions for context, but avoid firm claims about bias patterns.
Language You Can Reuse In A Methods Section
“We pre-specified an approach to gauge publication bias and other small-study effects. Where at least ten studies were available, we drew funnel plots and ran a size-effect regression test matched to the effect type (Egger for continuous outcomes; Harbord or Peters for binary outcomes). We added contour lines at p=0.10, 0.05, and 0.01 to aid interpretation. We ran trim-and-fill and a selection model as sensitivity analyses. We judged risk of bias due to missing evidence with ROB-ME and described how any shifts influenced the clinical read.”
Common Pitfalls And How To Avoid Them
Too few studies: Skip formal tests and report that power is low; keep the descriptive plot for context. One-sided searching: Restricting to English or published journals only can starve the evidence base. Over-confident plots: Asymmetry has many causes, so pair visuals with design checks and sensitivity models. Heterogeneity: Large between-study variance can mask or mimic asymmetry; flag this when you report test results. Bias stacking: Small trials at high risk of within-study bias can inflate effects; try an analysis that excludes them.
Reporting Checklist You Can Copy
In your manuscript, include: a link to the protocol; dates and sources for each search; a study-outcome matrix with blanks marked; rules for handling multiple time points and overlapping samples; a statement about language limits; the minimum number of studies you required before you ran small-study tests; which test you used and why it matches the effect scale; the contour settings on your funnel; the software and packages you used; results from trim-and-fill or a selection model shown next to the main model; the ROB-ME judgment with justification; and a note on what missing results could mean for readers or decision makers.
From Assessment To Action
Your readers want a straight answer to one question: if some results are missing, how much could that matter for decisions? Place the main estimate beside one or two adjusted estimates and state the shift. Link that shift to any rating system you use for certainty. Close with the ROB-ME call and a one-line plain-language note about missing results.