Skip to main content
  • AACR Publications
    • Blood Cancer Discovery
    • Cancer Discovery
    • Cancer Epidemiology, Biomarkers & Prevention
    • Cancer Immunology Research
    • Cancer Prevention Research
    • Cancer Research
    • Clinical Cancer Research
    • Molecular Cancer Research
    • Molecular Cancer Therapeutics

AACR logo

  • Register
  • Log in
  • Log out
  • My Cart
Advertisement

Main menu

  • Home
  • About
    • The Journal
    • AACR Journals
    • Subscriptions
    • Permissions and Reprints
    • Reviewing
  • Articles
    • OnlineFirst
    • Current Issue
    • Past Issues
    • CEBP Focus Archive
    • Meeting Abstracts
    • Progress and Priorities
    • Collections
      • COVID-19 & Cancer Resource Center
      • Disparities Collection
      • Editors' Picks
      • "Best of" Collection
  • For Authors
    • Information for Authors
    • Author Services
    • Best of: Author Profiles
    • Informing Public Health Policy
    • Submit
  • Alerts
    • Table of Contents
    • Editors' Picks
    • OnlineFirst
    • Citation
    • Author/Keyword
    • RSS Feeds
    • My Alert Summary & Preferences
  • News
    • Cancer Discovery News
  • COVID-19
  • Webinars
  • Search More

    Advanced Search

  • AACR Publications
    • Blood Cancer Discovery
    • Cancer Discovery
    • Cancer Epidemiology, Biomarkers & Prevention
    • Cancer Immunology Research
    • Cancer Prevention Research
    • Cancer Research
    • Clinical Cancer Research
    • Molecular Cancer Research
    • Molecular Cancer Therapeutics

User menu

  • Register
  • Log in
  • Log out
  • My Cart

Search

  • Advanced search
Cancer Epidemiology, Biomarkers & Prevention
Cancer Epidemiology, Biomarkers & Prevention
  • Home
  • About
    • The Journal
    • AACR Journals
    • Subscriptions
    • Permissions and Reprints
    • Reviewing
  • Articles
    • OnlineFirst
    • Current Issue
    • Past Issues
    • CEBP Focus Archive
    • Meeting Abstracts
    • Progress and Priorities
    • Collections
      • COVID-19 & Cancer Resource Center
      • Disparities Collection
      • Editors' Picks
      • "Best of" Collection
  • For Authors
    • Information for Authors
    • Author Services
    • Best of: Author Profiles
    • Informing Public Health Policy
    • Submit
  • Alerts
    • Table of Contents
    • Editors' Picks
    • OnlineFirst
    • Citation
    • Author/Keyword
    • RSS Feeds
    • My Alert Summary & Preferences
  • News
    • Cancer Discovery News
  • COVID-19
  • Webinars
  • Search More

    Advanced Search

Letters to the Editor

Journals Should Publish All “Null” Results and Should Sparingly Publish “Positive” Results

John P.A. Ioannidis
John P.A. Ioannidis
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
DOI: 10.1158/1055-9965.EPI-05-0921 Published January 2006
  • Article
  • Info & Metrics
  • PDF
Loading

To the Editor: The editorial by Rebbeck et al. (1) is timely and important. Here, I share some thoughts on this debate. The X team (real but anonymous here) meets successfully most proposed criteria. X has published nine articles on mostly brand new (but also some replicated) gene-disease associations in journals with an average impact factor exceeding 10. All publications show statistically significant (“positive”) results; include fascinating arguments about biological plausibility and pathways; emphasize population structure (most “positives” pertain to “racial” subgroups); insist on heterogeneity and interactions; and have decent power for main effects, although it is interactions/subgroups that claim importance. I wish I knew what is post hoc versus a priori here. X team accurately represents, quite well, the research that currently dominates high-impact journals.

What is the credibility of this literature? I think very low. At a minimum, there are heavy reporting biases. With small effects and trillions of possible, mostly false, hypotheses, the chance that any team comes across nine “positive” and no “null” findings approaches 0% (2). This literature may be less credible than the small genetic association studies in Chinese language journals wherein 50% of reports show statistically significant results and the genetic effects exceed anything seen in the average English language literature (3).

Epidemiology has been criticized for nonreplication because it is a robust science where measurements are amenable to quantitative refutation. “Basic science” reasoning is often as cloudy, beautiful, and impossible to refute as poetry. Few promises made in basic science journals are fulfilled even 20 to 25 years later (4). Biological evidence is extremely interesting but we should try to measure better how (well) it works. Functional data are usually derived after epidemiologic hints are provided (5). We do not know what the concordance of epidemiology and biology really means (6). Why promote spurious concordance seeking?

I also feel uncomfortable about chasing heterogeneity, multigene models, and interactions as primary targets. I firmly believe in heterogeneity (7) but, in my limited experience, when one accumulates more and better evidence, heterogeneity often goes away and interactions disappear, leaving simple main effects.

The large majority of molecular epidemiology analytic results should be “null” (2, 8, 9), representing truly null findings. A credible scientific journal should publish all studies with “null” results provided they acknowledge their limitations. Conversely, such a journal should be cautious about publishing “positive” results, most of which are false. Independent replication is important and should be done by different teams, preferably by competitors. “Null” results should be published promptly in print in short versions, with more extensive details in web-based files. “Positive” results should be published equally promptly, but only on the web, pending independent replication; once refuted, the original article and the refutation could be printed as a single nice null report; the rare validated findings should appear in print with full details.

Even for simple main effects, we need large-scale evidence and a healthy environment where selective reporting is minimized. The Network of Networks, a HuGENet initiative, was recently launched as an effort to attain this goal (10). It is important to know who is doing what in one field, join forces across teams working in the same field, and use standard methods for phenotypes, genotypes, exposures, and analyses. If we firmly document main effects, we can then also pursue second- and third-order analyses.

References

  1. ↵
    Rebbeck TR, Martinez ME, Sellers TA, Shields PG, Wild CP, Potter JD. Genetic variation and cancer: improving the environment for publication of association studies. Cancer Epidemiol Biomarkers Prev 2004;13:1985–6.
    OpenUrlFREE Full Text
  2. ↵
    Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124.
    OpenUrlCrossRefPubMed
  3. ↵
    Pan Z, Trikalinos TA, Kavvoura FK, Lau J, Ioannidis JPA. Local literature bias in genetic epidemiology: an empirical evaluation of the Chinese literature. PLoS Med. In press.
  4. ↵
    Contopoulos-Ioannidis DG, Ntzani E, Ioannidis JP. Translation of highly promising basic science research into clinical applications. Am J Med 2003;114:477–84.
    OpenUrlCrossRefPubMed
  5. ↵
    Pharoah PD, Dunning AM, Ponder BA, Easton DF. The reliable identification of disease-gene associations. Cancer Epidemiol Biomarkers Prev 2005;14:1362.
    OpenUrlFREE Full Text
  6. ↵
    Rebbeck TR, Spitz M, Wu X. Assessing the function of genetic variants in candidate gene association studies. Nat Rev Genet 2004;5:589–97.
    OpenUrlPubMed
  7. ↵
    Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is not always enough. Lancet 1998;351:123–7.
    OpenUrlCrossRefPubMed
  8. ↵
    Wacholder S, Chanock S, Garcia-Closas M, El Ghormli L, Rothman N. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J Natl Cancer Inst 2004;96:434–42.
    OpenUrlAbstract/FREE Full Text
  9. ↵
    Sterne JA, Davey Smith G. Sifting the evidence—what's wrong with significance tests? BMJ 2001;322:226–31.
    OpenUrlFREE Full Text
  10. ↵
    Ioannidis JP, Bernstein J, Boffetta P, et al. A network of investigator networks in human genome epidemiology. Am J Epidemiol 2005;162:302–4.
    OpenUrlAbstract/FREE Full Text
PreviousNext
Back to top
Cancer Epidemiology Biomarkers & Prevention: 15 (1)
January 2006
Volume 15, Issue 1
  • Table of Contents

Sign up for alerts

View this article with LENS

Open full page PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for sharing this Cancer Epidemiology, Biomarkers & Prevention article.

NOTE: We request your email address only to inform the recipient that it was you who recommended this article, and that it is not junk mail. We do not retain these email addresses.

Enter multiple addresses on separate lines or separate them with commas.
Journals Should Publish All “Null” Results and Should Sparingly Publish “Positive” Results
(Your Name) has forwarded a page to you from Cancer Epidemiology, Biomarkers & Prevention
(Your Name) thought you would be interested in this article in Cancer Epidemiology, Biomarkers & Prevention.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Journals Should Publish All “Null” Results and Should Sparingly Publish “Positive” Results
John P.A. Ioannidis
Cancer Epidemiol Biomarkers Prev January 1 2006 (15) (1) 186; DOI: 10.1158/1055-9965.EPI-05-0921

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Journals Should Publish All “Null” Results and Should Sparingly Publish “Positive” Results
John P.A. Ioannidis
Cancer Epidemiol Biomarkers Prev January 1 2006 (15) (1) 186; DOI: 10.1158/1055-9965.EPI-05-0921
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • References
  • Info & Metrics
  • PDF
Advertisement

Related Articles

Cited By...

More in this TOC Section

  • Copy Number Variation and Ovarian Cancer Risk—Letter
  • Genome-wide Analysis of Common Copy Number Variation and Epithelial Ovarian Cancer Risk—Response
  • Shift Work, Chronotype, and Cancer Risk—Response
Show more Letters to the Editor
  • Home
  • Alerts
  • Feedback
  • Privacy Policy
Facebook   Twitter   LinkedIn   YouTube   RSS

Articles

  • Online First
  • Current Issue
  • Past Issues

Info for

  • Authors
  • Subscribers
  • Advertisers
  • Librarians

About Cancer Epidemiology, Biomarkers & Prevention

  • About the Journal
  • Editorial Board
  • Permissions
  • Submit a Manuscript
AACR logo

Copyright © 2021 by the American Association for Cancer Research.

Cancer Epidemiology, Biomarkers & Prevention
eISSN: 1538-7755
ISSN: 1055-9965

Advertisement