Misuse of Statistics: P-hacking and Data Dredging
Statistics & Research Integrity
Misuse of Statistics: P-hacking and Data Dredging
How p-hacking, data dredging, HARKing, and publication bias corrupt modern research — and what you can do about it.
The Problem
Why Statistical Misuse Is Corrupting Research Right Now
P-hacking and data dredging sit at the heart of the replication crisis across psychology, medicine, and the social sciences.
Misuse of statistics isn’t a fringe academic concern — it is embedded in the production of mainstream scientific knowledge. In 2015, the Open Science Collaboration published a landmark study in Science attempting to replicate 100 psychology experiments. Only 36% produced results consistent with the originals. Across medicine, nutrition science, and economics, the numbers told similar stories. The replication crisis had arrived, and at its heart was a simple, devastating problem: researchers were using statistical tools in ways that reliably produced false-positive findings.
P-hacking and data dredging are the central mechanisms behind this. They exploit a structural vulnerability in the most widely used framework for scientific inference — null hypothesis significance testing (NHST) — to manufacture the appearance of statistically significant results from data that, analyzed honestly, would yield nothing. Understanding how they work isn’t just academically interesting. If you’re a student writing a stats assignment, a professional evaluating a clinical trial, or a policy-maker relying on economic research, you’re operating in a landscape shaped by these practices. Need expert help navigating that landscape? Statistics homework help from top-rated experts is available when you need it.
~50%
of published psychology findings fail to replicate in independent tests
p < 0.05
The threshold driving p-hacking — but means a 1-in-20 false positive rate at baseline
97%
of psychology papers report at least one significant result — statistically impossible without bias
The American Statistical Association (ASA) issued a landmark statement in 2016 explicitly warning against misuse of p-values. In 2019 they went further, with a special issue of The American Statistician calling for researchers to move beyond “statistically significant” as a binary gate. In clinical medicine at institutions like Johns Hopkins Hospital, Mayo Clinic, and the National Health Service (NHS) in the UK, statistical misuse in published trials can mean treatments adopted on fraudulent evidence — or genuine treatments rejected on the basis of underpowered negative studies. The stakes could not be higher, and students learning statistics need to understand this context from day one.
Foundations
What Is a P-value? The Foundation That Gets Misused
Before you can understand p-hacking, you need a precise grip on what a p-value actually measures — because one of the core problems is that most people, including trained researchers, misunderstand it. A p-value is the probability of obtaining results at least as extreme as those observed, assuming the null hypothesis is true. That’s it. It does not measure the probability that the null hypothesis is true. It does not measure the probability that a finding will replicate. It does not measure the size or importance of an effect. These misinterpretations are not minor — they are the intellectual foundation on which p-hacking thrives.
What Is Null Hypothesis Significance Testing (NHST)?
Null hypothesis significance testing, developed through the competing frameworks of Ronald A. Fisher at Rothamsted Experimental Station in the UK and Jerzy Neyman and Egon Pearson at University College London in the early 20th century, works as follows. A researcher proposes a null hypothesis (usually that there is no effect or no difference). They collect data, compute a test statistic, and derive a p-value. If the p-value falls below a predetermined threshold — almost universally set at 0.05 — they reject the null hypothesis and declare a “statistically significant” result.
The 0.05 threshold was essentially arbitrary. Fisher himself noted it was a convenient cutoff, not a law of nature. Yet over decades it calcified into a binary pass/fail gate controlling publication, funding, and academic careers. When a single number below an arbitrary threshold becomes the ticket to publication, the incentive to manipulate analysis to get below that threshold becomes enormous — and that’s precisely what p-hacking does.
What Does a p-value of 0.05 Actually Mean?
A p-value of 0.05 means that, if the null hypothesis were true and you repeated the experiment an infinite number of times, you would get results this extreme or more extreme in 5% of those repetitions — purely by random chance. This directly implies that if researchers test a hypothesis that is actually false and set their threshold at 0.05, roughly one in twenty such tests will produce a “statistically significant” result just by random variation. Multiply that by the thousands of studies published annually, add the structural pressures of publication bias, and you start to see why the scientific literature can be saturated with false positives even when individual researchers are not consciously cheating.
“The p-value was never meant to be a standalone criterion for scientific truth. Treating it that way is the source of most of the damage.” — Andrew Gelman, Columbia University statistician
This is the foundation on which misuse of statistics rests. The p-value isn’t broken — it’s being asked to do things it was never designed to do. Understanding the difference between quantitative and qualitative data and the statistical tools appropriate for each is the first step in avoiding these traps in your own research.
Core Concept
P-hacking: What It Is, How It Works, and Why It’s Everywhere
P-hacking is the practice of running multiple statistical analyses on a dataset and reporting only those that yield a p-value below 0.05, without disclosing the analyses that didn’t. The term was popularized by Uri Simonsohn, Leif Nelson, and Joseph Simmons at the Wharton School of the University of Pennsylvania, who demonstrated in a 2011 paper in Psychological Science that researcher degrees of freedom — the many analytical choices available at each stage of a study — create enormous capacity for inflating false positive rates.
P-hacking doesn’t require intent to deceive. Researchers often engage in it without realizing they’re doing it. The problem is structural: the incentive system in academic science rewards significant results, and when researchers have flexibility in how they analyze data, confirmation bias naturally pushes them toward the choices that yield significance. The result is the same whether deliberate or not — false positives flood the literature.
What Are the Main Forms of P-hacking?
1
Selective Outcome Reporting
A study measures ten different outcomes but reports only the two that achieve statistical significance. This is also called outcome switching and is one of the most pervasive forms of p-hacking in clinical trial literature, specifically identified by AllTrials — a transparency campaign led by physician and author Ben Goldacre at the University of Oxford’s Centre for Evidence-Based Medicine.
2
Optional Stopping (Peeking)
Collecting data, running a significance test, continuing data collection if not significant, and stopping the moment significance is reached. A researcher who peeks at results multiple times can achieve a false positive rate of nearly 30% even when no true effect exists — far above the stated 5%.
3
Selective Exclusion of Participants
Removing participants deemed “outliers” after seeing that their inclusion prevents significance. A legitimate exclusion criterion is defined before data collection and applied consistently. An illegitimate one is applied selectively after seeing that certain participants’ data inconveniently keep the result non-significant.
4
Subgroup Mining
Testing whether an effect exists within many different subgroups and reporting only where significance was achieved. Testing indiscriminate subgroup after subgroup is a reliable way to generate false positives that illustrate exactly how easily the technique manufactures spurious findings.
5
Trying Multiple Analytical Models
Running the same data through multiple statistical models (ANOVA, regression, t-test, non-parametric alternatives) and reporting only the model that achieves significance. Each additional test performed on the same data inflates the family-wise error rate. Failing to apply corrections like the Bonferroni correction when running multiple tests is a core technical form of statistical misuse.
Important: P-hacking is not always intentional. The flexibility of statistical analysis means many researchers engage in it without realizing they’re doing so. This does not reduce its harm to the scientific literature — but it does shape what the solution needs to look like. Structural reforms, not just moral condemnation, are required.
Need Help With a Statistics Assignment?
Our expert statisticians help students across the US and UK understand regression, hypothesis testing, and research methods — from introductory courses to doctoral dissertations.
Order Statistics Help Log InCore Concept
Data Dredging: Fishing for Significance in the Same Pond
Data dredging — also called data fishing, data snooping, or variable dredging — is the practice of applying many statistical tests to a dataset without a pre-specified hypothesis, then presenting the significant findings as if they were the goal all along. Where p-hacking describes the manipulation of a single study’s analysis, data dredging describes a broader strategy of exploratory mining conducted in a confirmatory disguise.
The statistical problem is the same: when you run many tests, the probability that at least one will return a false positive grows rapidly. If you test 20 independent hypotheses using a 0.05 threshold, the probability of getting at least one false positive is approximately 64% (1 – 0.95²⁰). Test 100, and it’s 99.4%. Data dredging in a rich dataset with hundreds of variables can produce dozens of “significant” correlations through pure chance — and the researcher, without any record of the search process, can present the most interesting-looking ones as hypothesis-driven findings.
What Is the Difference Between Data Dredging and Exploratory Analysis?
This is a genuinely important distinction. Exploratory data analysis (EDA), developed systematically by John Tukey at Princeton University and Bell Labs, is a legitimate and valuable practice. The problem isn’t exploring data — it’s presenting exploratory findings as confirmatory ones. Properly conducted exploratory analysis says: “I found these interesting patterns that warrant further investigation.” Data dredging presents the same patterns as pre-specified hypothesis tests.
The solution is transparency in reporting, not the elimination of exploration. A study that honestly labels its findings as exploratory contributes legitimately to science. The same study whose findings are presented as pre-specified confirmatory tests is data dredging. The difference lies entirely in honest communication — something that pre-registration enforces structurally.
Real-World Examples of Data Dredging
One of the most cited examples involves a database of spurious correlations compiled by researcher Tyler Vigen, demonstrating associations like the near-perfect correlation between US per capita cheese consumption and deaths by bedsheet tangling. Both trend upward over the same period; there is no causal relationship.
More consequential examples exist in nutrition epidemiology. Researcher John Ioannidis at Stanford University’s Meta-Research Innovation Center (METRICS) has extensively documented how nutrition studies produce dramatic, often contradictory headlines — that wine causes cancer, then prevents it — largely because nutrition datasets contain thousands of variables and have historically lacked pre-registration norms.
“If you torture the data long enough, it will confess to anything.” — Ronald Coase, Nobel Laureate in Economics
Related Practices
HARKing, Cherry-Picking, and the Full Spectrum of Statistical Misuse
P-hacking and data dredging sit within a broader ecosystem of questionable research practices (QRPs). Understanding the full taxonomy clarifies why the problems are systemic rather than isolated. These practices often co-occur, compounding the distortion of the published record.
What Is HARKing (Hypothesizing After Results Are Known)?
HARKing was named and systematically described by Norbert Kerr at Michigan State University in a 1998 paper in Personality and Social Psychology Review. HARKing occurs when a researcher conducts an exploratory analysis, finds a significant result, and then writes the study up as if that result was the pre-specified hypothesis all along. The Methods section reads as if the analysis was confirmatory; the finding looks like a successful prediction rather than a post-hoc discovery.
Cherry-Picking Data
Cherry-picking refers to selectively citing or reporting data that supports a desired conclusion while omitting contradictory evidence. In clinical medicine, cherry-picking has had documented lethal consequences. GlaxoSmithKline was fined $3 billion by the US Department of Justice in 2012, partly for misrepresenting research data on the antidepressant Paxil — including failing to publish trials showing it was ineffective in adolescents.
What Is Publication Bias?
Publication bias is the tendency of journals to publish studies with statistically significant results preferentially over null or negative results, regardless of methodology quality. This creates the file drawer problem — articulated by Robert Rosenthal at Harvard University — where null results accumulate unpublished in researchers’ file drawers while significant results dominate the literature.
| Practice | Definition | Mechanism of Harm | Detection Method |
|---|---|---|---|
| P-hacking | Manipulating analysis until p < 0.05 | Inflates false positive rate per study | P-curve analysis; pre-registration |
| Data Dredging | Testing many variables without prior hypothesis | Spurious patterns appear significant | Bonferroni correction; registered reports |
| HARKing | Presenting post-hoc hypotheses as pre-specified | Disguises exploratory findings as confirmatory | Pre-registration; trial registries |
| Cherry-Picking | Selectively reporting favorable results | Literature distorted toward positives | Systematic reviews; all-trial registration |
| Publication Bias | Journals preferring significant results | Null results suppressed; file-drawer problem | Funnel plots; PROSPERO registry |
| Optional Stopping | Stopping data collection when significance reached | Drastically inflated false positive rate | Sequential analysis methods; pre-set n |
The Consequences
The Replication Crisis: What Statistical Misuse Has Done to Science
The replication crisis is the name given to the discovery, beginning roughly in the early 2010s, that large proportions of published findings across multiple scientific disciplines cannot be reproduced when independent researchers attempt to repeat the original experiments. It is not an isolated scandal but a systemic indictment of the statistical practices governing scientific publication for decades — practices dominated by p-hacking, data dredging, HARKing, and publication bias.
In psychology, the Open Science Collaboration‘s 2015 replication project found only 36–39% of 100 high-profile studies replicated successfully. In medicine, Ioannidis and colleagues found that when they attempted to replicate 49 highly cited clinical research claims, 16% were contradicted by subsequent evidence and another 16% found effects substantially smaller than originally claimed. In economics, the Open Economics project found a replication rate of around 60%.
Which Famous Studies Failed to Replicate?
Several landmark studies that became widely cited in textbooks failed replication. Social priming effects — including studies by John Bargh at Yale University suggesting that making people think about elderly people caused them to walk slower — failed to replicate in large pre-registered studies. The “power posing” research by Amy Cuddy, formerly of Harvard Business School, which became one of the most-watched TED Talks in history, saw its core hormonal claims fail independent replication.
Why does this matter for students? If you’re writing a research paper or literature review citing psychology, nutrition, management, or social science research, you’re drawing from a literature substantially shaped by statistical misuse. Knowing how to evaluate study quality — pre-registration status, sample size, effect size, replication record — is now a core academic literacy skill.
Is the Replication Crisis Limited to Psychology?
Emphatically no. The crisis is most visible in psychology because that field was first to undertake systematic self-examination, largely driven by Brian Nosek at the University of Virginia and his Center for Open Science. But equivalent problems exist in medicine, nutrition, economics, marketing science, management research, and education science. The common thread is not discipline — it’s the use of undisclosed researcher degrees of freedom combined with small samples, publication bias, and a threshold-based publication culture.
Writing a Research Methods Paper?
Our expert writers and statisticians can help you craft rigorous, evidence-based research methods sections — including proper handling of statistical tests, effect sizes, and pre-registration discussion.
Get Research Paper Help Log InPeople & Organizations
Key Researchers, Organizations, and Tools Shaping This Debate
Uri Simonsohn — Wharton / ESADE Business School
Uri Simonsohn, currently at ESADE Business School, is arguably the single most influential figure in making p-hacking a mainstream topic of concern. His 2011 paper on “false-positive psychology” directly demonstrated how researcher degrees of freedom allow essentially any dataset to be tortured into significance. His follow-up work developing the p-curve — a tool that tests whether a set of significant results shows the distribution expected from true effects versus p-hacking — gave the field its first broadly accessible diagnostic for statistical misuse.
John Ioannidis — Stanford University
John P. A. Ioannidis at Stanford School of Medicine published “Why Most Published Research Findings Are False” (2005, PLOS Medicine). Using probability theory, he demonstrated mathematically that under realistic research conditions the majority of claimed significant findings in many fields are likely false positives — anticipating the replication crisis by nearly a decade.
Brian Nosek and the Center for Open Science
Brian Nosek at the University of Virginia founded the Center for Open Science (COS) in 2013 with a mission to increase openness, integrity, and reproducibility in scientific research. COS operates the Open Science Framework (OSF), the most widely used pre-registration platform in academic research worldwide.
The American Statistical Association (ASA)
The American Statistical Association issued the first institutional professional statement on p-value misuse in 2016, clarifying that p-values do not measure the probability the hypothesis is true and that scientific conclusions should not be based on whether a p-value passes a threshold. In 2019, the ASA called for abandoning “statistical significance” as a binary concept.
Ben Goldacre and AllTrials — UK
Ben Goldacre, physician and author of Bad Science and Bad Pharma, has been the most prominent public communicator in the UK on clinical trial misuse. His AllTrials campaign advocates for all clinical trials to be registered before they begin and results published regardless of outcome.
Better Practice
Effect Sizes, Confidence Intervals, and What to Report Instead
If p-values used in isolation are the problem, what should researchers report instead? The answer isn’t to abandon statistical inference — it’s to report it more fully. The three essential complements to (or replacements for) bare p-values are effect sizes, confidence intervals, and Bayesian factors.
What Is an Effect Size?
An effect size is a standardized measure of the magnitude of a phenomenon or difference, independent of sample size. Where a p-value tells you how likely results this extreme are under the null hypothesis, an effect size tells you how big the effect actually is. The most common measures include Cohen’s d (for differences between two means), Pearson’s r (for correlations), and eta-squared (for ANOVA designs).
Jacob Cohen, a statistician at New York University, published “Statistical Power Analysis for the Behavioral Sciences” in 1969, establishing conventions still used today: a Cohen’s d of 0.2 is “small,” 0.5 is “medium,” and 0.8 is “large.” Effect sizes are now mandatory reporting requirements in Psychological Science, JAMA, and many other leading journals.
Confidence Intervals: The Information P-values Don’t Give You
A 95% confidence interval (CI) provides a range within which the true population parameter falls in 95% of cases if the sampling process is repeated infinitely. It gives you both the direction and the precision of an estimate — information that a bare p-value entirely lacks. The ASA now recommends that confidence intervals should accompany or replace p-values in most statistical reporting.
Bayesian Methods: A Fundamentally Different Approach
Bayesian statistics offers a fundamentally different inferential framework that sidesteps many p-hacking vulnerabilities. Instead of testing whether data are unlikely under the null hypothesis, Bayesian methods compute the Bayes factor: a direct comparison of how well the data are explained by the alternative hypothesis versus the null. Bayesian methods naturally penalize model complexity, don’t have the arbitrary stopping-rule problem of NHST, and produce posterior probability distributions rather than binary pass/fail decisions.
Solutions
How to Avoid P-hacking: Reforms, Tools, and Best Practices
Pre-registration: The Single Most Powerful Reform
Pre-registration means submitting a time-stamped, publicly available document specifying your hypothesis, sample size, data collection procedure, and analysis plan before collecting any data. It makes post-hoc hypothesis adjustment, outcome switching, and selective reporting immediately visible — anyone can compare the pre-registered plan against the published paper. The Open Science Framework (OSF) and ClinicalTrials.gov are the two most important pre-registration platforms.
A Nature Human Behaviour analysis of registered reports found that 44% produced null or negative results, compared to just 5–15% in conventionally published research. That difference reflects suppression of null results in the conventional system and inflation of positives through p-hacking.
Multiple Comparisons Correction
When a study tests multiple hypotheses, the family-wise error rate must be controlled. The most conservative method is the Bonferroni correction: divide the significance threshold (0.05) by the number of tests performed. Less conservative alternatives include the Benjamini-Hochberg procedure for controlling the false discovery rate (FDR), which is standard in genomics and neuroimaging.
Open Data and Reproducible Research
Sharing raw data and analysis code publicly — through platforms like OSF, GitHub, Dryad, or Zenodo — allows anyone to verify reported analyses. When another researcher can download the original dataset and run the analysis themselves, the scope for undetected p-hacking collapses dramatically.
How to Detect P-hacking in a Published Study
Several practical red flags signal possible p-hacking: p-values clustering suspiciously between 0.04 and 0.05; many outcome variables mentioned but few reported; post-hoc subgroup analyses presented as primary findings; no pre-registration; effect sizes implausibly large for the sample size; and every single test coming up significant — real data is rarely that cooperative.
| Reform | What It Addresses | Key Platform / Standard | Adopted By |
|---|---|---|---|
| Pre-registration | HARKing, outcome switching, optional stopping | OSF, ClinicalTrials.gov, ISRCTN | NIH, NHS, many journals |
| Registered Reports | Publication bias, p-hacking incentive | cos.io/rr, Nature Human Behaviour | 300+ journals as of 2025 |
| Open Data | Undetected manipulation, non-reproducibility | OSF, GitHub, Zenodo, Dryad | NIH, Wellcome Trust, EU Horizon |
| Effect Sizes + CI | Overemphasis on p-value significance | APA guidelines, ASA statement | Psychological Science, JAMA |
| Bonferroni / FDR correction | Multiple comparison inflation | R, SPSS, STATA packages | Standard in genomics, neuro, clinical trials |
| Bayesian methods | Binary significance threshold problems | JASP (free), Stan, brms in R | Nature Human Behaviour, growing adoption |
Struggling With Statistics or Research Methods?
Whether it’s regression, hypothesis testing, ANOVA, or interpreting effect sizes — our expert tutors are available 24/7 to help you submit with confidence.
Get Instant Help NowFor Students
What This Means for Your Coursework, Dissertations, and Career
The misuse of statistics debate might feel distant from undergraduate statistics homework or a master’s dissertation. It isn’t. The same practices operate at all levels of research, and the same principles of honest analysis apply whether you’re writing up a second-year research methods practical or a doctoral thesis at a Russell Group institution or Ivy League university.
How P-hacking and Data Dredging Affect Student Research
When students run data analyses, they face genuine analytical choices: which variables to include, whether to remove an outlier, which of several plausible statistical tests to use. Without awareness of how these choices affect results, it’s easy to unconsciously gravitate toward the choices that produce significant results — especially under deadline pressure.
The solution for students is the same as for professional researchers: decide your analysis plan before you analyze your data, document every analytical choice, and report all results — including null and unexpected ones. In many programs at UK universities including University of Manchester, King’s College London, and University of Edinburgh, pre-registration of student research projects is now formally encouraged or required.
Writing About Statistics in Essays and Reports
When you cite statistical findings in an essay — claims that coffee prevents Alzheimer’s, that mindfulness improves exam performance, that income inequality causes crime — you’re implicitly making a judgment about the quality of the underlying evidence. In the current landscape, citing a single significant study as if it were established fact is a critical thinking failure. Strong academic writing evaluates evidence quality: Was the study pre-registered? Has it been replicated? What is the effect size? Is there meta-analytic support?
Related Concept
Statistical Power, Sample Size, and Why Underpowered Studies Lie
Statistical power is the probability that a study will detect a true effect when one exists. A study with low statistical power — typically because the sample size is too small — isn’t just less likely to find real effects; it creates a specific problem that amplifies p-hacking damage. This is the winner’s curse: in underpowered research, the only way a study achieves statistical significance is if the data happen to show an effect larger than the true one, due to sampling variation.
Cohen (1962) surveyed published psychology studies and found average statistical power of around 0.46 — meaning studies had only a 46% chance of detecting a real medium-sized effect. In clinical trials regulated by the FDA and MHRA in the UK, sample size and power calculations are required by law. Free tools like G*Power make a priori power analysis straightforward for students.
For students: Power analysis should always be performed before data collection, not after. Post-hoc power calculations — run after seeing your data — are uninformative and frequently misleading. If your dissertation supervisor asks you to justify your sample size, a pre-specified power calculation is the correct answer. Without one, you cannot know whether a non-significant result means no effect exists or simply that your study was too small to detect it.
Frequently Asked
Frequently Asked Questions About P-hacking and Statistical Misuse
What exactly is p-hacking in simple terms?
P-hacking is when a researcher runs multiple statistical tests, tries different analytical approaches, or manipulates their data in various ways until they get a “statistically significant” result (p < 0.05) — and then reports that result without disclosing all the things they tried first. It’s like flipping a coin 100 times, finding three consecutive heads, and writing a paper claiming coins land heads in streaks. The conclusion looks statistically supported only because you don’t report the other 97 flips.
Is p-hacking illegal or unethical?
P-hacking exists on a spectrum. Deliberate p-hacking with intent to deceive constitutes research fraud — it violates the policies of virtually every university, funding agency, and journal. In clinical research, it can constitute regulatory fraud under FDA regulations or the UK’s Research Governance Framework. However, much p-hacking is unintentional, emerging from the natural flexibility of statistical analysis combined with unconscious confirmation bias. Intent does not eliminate harm, but it shapes what remedies are appropriate.
How common is p-hacking in academic research?
Surveys of researchers are concerning. A 2012 survey by Leslie John at Harvard Business School found that 58% of psychological scientists admitted to deciding whether to add more data after seeing whether results were significant; 35% reported unexpected findings as predicted; and 40% selectively reported studies. In medicine, analyses of published clinical trial data suggest outcome switching occurs in the majority of trials when pre-specified outcomes are compared to published ones.
What is the multiple comparisons problem?
The multiple comparisons problem arises when you test many hypotheses simultaneously. Each test carries a 5% chance of a false positive at α = 0.05. When you run 20 independent tests, the chance of at least one false positive rises to about 64%. Solutions include the Bonferroni correction (divide 0.05 by number of tests), the Benjamini-Hochberg False Discovery Rate procedure, or Bayesian methods that naturally handle multiple comparisons through prior specification.
What’s wrong with just using a stricter p-value threshold?
A widely discussed 2017 proposal in Nature Human Behaviour, signed by 72 prominent statisticians, suggested lowering the standard significance threshold from p < 0.05 to p < 0.005. This would reduce false positives substantially. But critics argue this doesn’t solve the underlying problem — p-hacking would continue; researchers would simply aim for 0.004 rather than 0.04. The solution isn’t a different number — it’s a different philosophy: report effect sizes, confidence intervals, and prior probabilities rather than relying on any single threshold.
How do I spot p-hacking when reading a study?
Watch for these red flags: (1) p-values that cluster suspiciously between 0.04 and 0.05; (2) many outcome variables mentioned but few reported; (3) post-hoc subgroup analyses presented as primary findings; (4) no pre-registration or trial registry number; (5) effect sizes that look implausibly large for a small sample; (6) all reported results are significant — real data rarely works that perfectly; (7) hypotheses that feel suspiciously tailored to exactly what the data showed. You can use the p-curve tool at p-curve.com to formally test whether a set of p-values shows statistical signs of p-hacking.
What is “researcher degrees of freedom”?
Coined by Simmons, Nelson, and Simonsohn, “researcher degrees of freedom” refers to the many legitimate-looking analytical choices researchers make during a study that each have the potential to influence results: which participants to include or exclude, which variables to control for, which statistical test to use, when to stop collecting data, how to code responses. A 2011 simulation showed that leveraging just four degrees of freedom across a study could inflate false positive rates from 5% to over 60%.
Can artificial intelligence be used to p-hack?
This is an increasingly important question. Machine learning algorithms that automatically search for predictive patterns are, structurally, data dredging machines — they find the combinations of variables that best predict outcomes in a dataset. Without proper train/test splits, cross-validation, and pre-specified evaluation metrics, ML-based research is extremely vulnerable to generating overfit findings that look significant but don’t generalize. The same pre-registration and effect size reporting principles that apply to conventional statistics apply equally to machine learning research.
What is the file drawer problem in statistics?
The file drawer problem, described by Robert Rosenthal at Harvard, refers to the tendency for null results to go unpublished — sitting in researchers’ file drawers — while significant results get published. This creates systematic bias in the published literature because readers see only a non-representative sample of all studies conducted. Meta-analyses drawing on published literature therefore overestimate true effects. The solution is trial registration, journal policies that publish negative results, and registered reports.
Does pre-registration actually stop p-hacking?
Pre-registration does not make p-hacking impossible — a researcher could simply ignore their pre-registration. But it creates accountability: anyone can compare the registered plan to the published paper and identify deviations. Registered Reports go further by removing the incentive entirely — publication is already assured regardless of outcome, so there’s no point in p-hacking. Empirically, registered reports produce dramatically higher rates of null results (around 44%) compared to conventional publication (5–15%), confirming they genuinely reduce inflation caused by p-hacking and publication bias.