Misuse of Statistics: P-hacking and Data Dredging

Why Statistical Misuse Is Corrupting Research Right Now

P-hacking and data dredging sit at the heart of the replication crisis across psychology, medicine, and the social sciences.

Misuse of statistics isn't a fringe academic concern — it is embedded in the production of mainstream scientific knowledge. In 2015, the Open Science Collaboration published a landmark study in Science attempting to replicate 100 psychology experiments. Only 36% produced results consistent with the originals. Across medicine, nutrition science, and economics, the numbers told similar stories. The replication crisis had arrived, and at its heart was a simple, devastating problem: researchers were using statistical tools in ways that reliably produced false-positive findings.

P-hacking and data dredging are the central mechanisms behind this. They exploit a structural vulnerability in the most widely used framework for scientific inference — null hypothesis significance testing (NHST) — to manufacture the appearance of statistically significant results from data that, analyzed honestly, would yield nothing. Understanding how they work isn't just academically interesting. If you're a student writing a stats assignment, a professional evaluating a clinical trial, or a policy-maker relying on economic research, you're operating in a landscape shaped by these practices. Need expert help navigating that landscape? Statistics homework help from top-rated experts is available when you need it.

The American Statistical Association (ASA) issued a landmark statement in 2016 explicitly warning against misuse of p-values. In 2019 they went further, with a special issue of The American Statistician calling for researchers to move beyond "statistically significant" as a binary gate. In clinical medicine at institutions like Johns Hopkins Hospital, Mayo Clinic, and the National Health Service (NHS) in the UK, statistical misuse in published trials can mean treatments adopted on fraudulent evidence — or genuine treatments rejected on the basis of underpowered negative studies. The stakes could not be higher, and students learning statistics need to understand this context from day one.

What Is a P-value? The Foundation That Gets Misused

Before you can understand p-hacking, you need a precise grip on what a p-value actually measures — because one of the core problems is that most people, including trained researchers, misunderstand it. A p-value is the probability of obtaining results at least as extreme as those observed, assuming the null hypothesis is true. That's it. It does not measure the probability that the null hypothesis is true. It does not measure the probability that a finding will replicate. It does not measure the size or importance of an effect. These misinterpretations are not minor — they are the intellectual foundation on which p-hacking thrives.

What Is Null Hypothesis Significance Testing (NHST)?

Null hypothesis significance testing, developed through the competing frameworks of Ronald A. Fisher at Rothamsted Experimental Station in the UK and Jerzy Neyman and Egon Pearson at University College London in the early 20th century, works as follows. A researcher proposes a null hypothesis (usually that there is no effect or no difference). They collect data, compute a test statistic, and derive a p-value. If the p-value falls below a predetermined threshold — almost universally set at 0.05 — they reject the null hypothesis and declare a "statistically significant" result.

The 0.05 threshold was essentially arbitrary. Fisher himself noted it was a convenient cutoff, not a law of nature. Yet over decades it calcified into a binary pass/fail gate controlling publication, funding, and academic careers. When a single number below an arbitrary threshold becomes the ticket to publication, the incentive to manipulate analysis to get below that threshold becomes enormous — and that's precisely what p-hacking does. Students writing research methods papers need to explain this machinery clearly; our scientific method essay writing guide walks through the logic step by step.

What Does a p-value of 0.05 Actually Mean?

A p-value of 0.05 means that, if the null hypothesis were true and you repeated the experiment an infinite number of times, you would get results this extreme or more extreme in 5% of those repetitions — purely by random chance. This directly implies that if researchers test a hypothesis that is actually false and set their threshold at 0.05, roughly one in twenty such tests will produce a "statistically significant" result just by random variation. Multiply that by the thousands of studies published annually, add the structural pressures of publication bias, and you start to see why the scientific literature can be saturated with false positives even when individual researchers are not consciously cheating.

This is the foundation on which misuse of statistics rests. The p-value isn't broken — it's being asked to do things it was never designed to do. Understanding the difference between quantitative and qualitative data and the statistical tools appropriate for each is the first step in avoiding these traps in your own research.

P-hacking: What It Is, How It Works, and Why It's Everywhere

P-hacking is the practice of running multiple statistical analyses on a dataset and reporting only those that yield a p-value below 0.05, without disclosing the analyses that didn't. The term was popularized by Uri Simonsohn, Leif Nelson, and Joseph Simmons at the Wharton School of the University of Pennsylvania, who demonstrated in a 2011 paper in Psychological Science that researcher degrees of freedom — the many analytical choices available at each stage of a study — create enormous capacity for inflating false positive rates.

P-hacking doesn't require intent to deceive. Researchers often engage in it without realizing they're doing it. The problem is structural: the incentive system in academic science rewards significant results, and when researchers have flexibility in how they analyze data, confirmation bias naturally pushes them toward the choices that yield significance. The result is the same whether deliberate or not — false positives flood the literature.

What Are the Main Forms of P-hacking?

P-hacking manifests in several specific analytical choices that researchers make during a study. Recognizing these is essential for anyone reading or producing statistical research.

Data Dredging: Fishing for Significance in the Same Pond

Data dredging — also called data fishing, data snooping, or variable dredging — is the practice of applying many statistical tests to a dataset without a pre-specified hypothesis, then presenting the significant findings as if they were the goal all along. Where p-hacking describes the manipulation of a single study's analysis, data dredging describes a broader strategy of exploratory mining conducted in a confirmatory disguise.

The statistical problem is the same: when you run many tests, the probability that at least one will return a false positive grows rapidly. If you test 20 independent hypotheses using a 0.05 threshold, the probability of getting at least one false positive is approximately 64% (1 – 0.95²⁰). Test 100, and it's 99.4%. Data dredging in a rich dataset with hundreds of variables can produce dozens of "significant" correlations through pure chance — and the researcher, without any record of the search process, can present the most interesting-looking ones as hypothesis-driven findings.

What Is the Difference Between Data Dredging and Exploratory Analysis?

This is a genuinely important distinction. Exploratory data analysis (EDA), developed systematically by John Tukey at Princeton University and Bell Labs, is a legitimate and valuable practice. The problem isn't exploring data — it's presenting exploratory findings as confirmatory ones. Properly conducted exploratory analysis says: "I found these interesting patterns that warrant further investigation." Data dredging presents the same patterns as pre-specified hypothesis tests.

The solution is transparency in reporting, not the elimination of exploration. A study that honestly labels its findings as exploratory contributes legitimately to science. The same study whose findings are presented as pre-specified confirmatory tests is data dredging. The difference lies entirely in honest communication — something that pre-registration enforces structurally. Students working on research papers should understand how to conduct research transparently from the very beginning of their project.

Real-World Examples of Data Dredging

One of the most cited examples involves a database of spurious correlations compiled by researcher Tyler Vigen, demonstrating associations like the near-perfect correlation between US per capita cheese consumption and deaths by bedsheet tangling. Both trend upward over the same period; there is no causal relationship. Yet if a researcher reported this with a p-value without disclosing the data-dredging process, it would look like a genuine finding.

More consequential examples exist in nutrition epidemiology. Researcher John Ioannidis at Stanford University's Meta-Research Innovation Center (METRICS) has extensively documented how nutrition studies produce dramatic, often contradictory headlines — that wine causes cancer, then prevents it — largely because nutrition datasets contain thousands of variables and have historically lacked pre-registration norms. His 2005 paper "Why Most Published Research Findings Are False" in PLOS Medicine remains one of the most downloaded scientific articles ever written. Understanding regression analysis as the backbone of predictive modeling helps you evaluate when published regression results can actually be trusted.

HARKing, Cherry-Picking, and the Full Spectrum of Statistical Misuse

P-hacking and data dredging sit within a broader ecosystem of questionable research practices (QRPs). Understanding the full taxonomy clarifies why the problems are systemic rather than isolated. These practices often co-occur, compounding the distortion of the published record.

What Is HARKing (Hypothesizing After Results Are Known)?

HARKing was named and systematically described by Norbert Kerr at Michigan State University in a 1998 paper in Personality and Social Psychology Review. HARKing occurs when a researcher conducts an exploratory analysis, finds a significant result, and then writes the study up as if that result was the pre-specified hypothesis all along. The Methods section reads as if the analysis was confirmatory; the finding looks like a successful prediction rather than a post-hoc discovery.

HARKing is corrosive because it converts exploratory findings into the currency of confirmatory science. A genuine exploratory finding carries appropriate uncertainty — it suggests an avenue for further investigation. A HARKed finding is presented as confirmed knowledge. When subsequent researchers attempt to replicate it and fail, the apparent "surprise" of non-replication is actually the system correctly identifying that the original finding was noise. Understanding how the art of persuasion shapes academic writing helps you recognize when a paper is framing findings more confidently than the underlying analysis warrants.

Cherry-Picking Data

Cherry-picking refers to selectively citing or reporting data that supports a desired conclusion while omitting contradictory evidence. In published research, this takes the form of citing only the studies that confirm a hypothesis in a literature review, reporting only favorable subgroup results, or selecting time periods in longitudinal data that happen to show the desired trend.

In clinical medicine, cherry-picking has had documented lethal consequences. GlaxoSmithKline was fined $3 billion by the US Department of Justice in 2012, partly for misrepresenting research data on the antidepressant Paxil — including failing to publish trials showing it was ineffective in adolescents. Selective reporting of favorable trial outcomes while suppressing negative ones is cherry-picking at industrial scale, enabled by the same statistical incentive structures driving p-hacking in academic labs.

What Is Publication Bias?

Publication bias is the tendency of journals to publish studies with statistically significant results preferentially over null or negative results, regardless of methodology quality. This creates the file drawer problem — articulated by Robert Rosenthal at Harvard University — where null results accumulate unpublished in researchers' file drawers while significant results dominate the literature. Meta-analyses drawing from a publication-biased literature aggregate that bias rather than correcting it. Funnel plot asymmetry, a technique developed by statistician Colin Begg, detects publication bias in meta-analytic datasets by testing whether small studies show disproportionate clustering among positive results. This kind of meta-analytic awareness is now expected at graduate programs at Oxford, Cambridge, Harvard, and MIT.

The Replication Crisis: What Statistical Misuse Has Done to Science

The replication crisis is the name given to the discovery, beginning roughly in the early 2010s, that large proportions of published findings across multiple scientific disciplines cannot be reproduced when independent researchers attempt to repeat the original experiments. It is not an isolated scandal but a systemic indictment of the statistical practices governing scientific publication for decades — practices dominated by p-hacking, data dredging, HARKing, and publication bias.

In psychology, the Open Science Collaboration's 2015 replication project found only 36–39% of 100 high-profile studies replicated successfully. In medicine, Ioannidis and colleagues found that when they attempted to replicate 49 highly cited clinical research claims, 16% were contradicted by subsequent evidence and another 16% found effects substantially smaller than originally claimed. In economics, the Open Economics project found a replication rate of around 60%. In preclinical cancer biology, the Reproducibility Project: Cancer Biology led by the Center for Open Science in Charlottesville, Virginia, failed to replicate many key findings that had formed the basis of drug development programs.

Which Famous Studies Failed to Replicate?

Several landmark studies that became widely cited in textbooks failed replication. Social priming effects — including studies by John Bargh at Yale University suggesting that making people think about elderly people caused them to walk slower — failed to replicate in large pre-registered studies. The "power posing" research by Amy Cuddy, formerly of Harvard Business School, which became one of the most-watched TED Talks in history, saw its core hormonal claims fail independent replication. The glucose and self-control literature, subjected to a large multilab replication effort by the Psychological Science Accelerator, found no support for the main effect.

Is the Replication Crisis Limited to Psychology?

Emphatically no. The crisis is most visible in psychology because that field was first to undertake systematic self-examination, largely driven by Brian Nosek at the University of Virginia and his Center for Open Science. But equivalent problems exist in medicine, nutrition, economics, marketing science, management research, and education science. The common thread is not discipline — it's the use of undisclosed researcher degrees of freedom combined with small samples, publication bias, and a threshold-based publication culture. Where those conditions exist, replication problems follow.

Key Researchers, Organizations, and Tools Shaping This Debate

Understanding the misuse of statistics landscape requires knowing the people and organizations who have defined, documented, and driven reform of these practices. These are not peripheral figures — they are at the heart of contemporary scientific debate in the United States and the United Kingdom.

Uri Simonsohn — Wharton / ESADE Business School

Uri Simonsohn, currently at ESADE Business School (formerly at UPenn's Wharton School), is arguably the single most influential figure in making p-hacking a mainstream topic of concern. His 2011 paper on "false-positive psychology" directly demonstrated how researcher degrees of freedom allow essentially any dataset to be tortured into significance. His follow-up work developing the p-curve — a tool that tests whether a set of significant results shows the distribution expected from true effects versus p-hacking — gave the field its first broadly accessible diagnostic for statistical misuse. The p-curve tool is free at p-curve.com and is now used in systematic reviews at Psychological Science and Nature Human Behaviour.

John Ioannidis — Stanford University

John P. A. Ioannidis at Stanford School of Medicine, where he directs the Meta-Research Innovation Center (METRICS), published "Why Most Published Research Findings Are False" (2005, PLOS Medicine). Using probability theory, he demonstrated mathematically that under realistic research conditions the majority of claimed significant findings in many fields are likely false positives. His work preceded the replication crisis by nearly a decade and anticipated its empirical findings precisely.

Brian Nosek and the Center for Open Science

Brian Nosek at the University of Virginia founded the Center for Open Science (COS) in 2013 with a mission to increase openness, integrity, and reproducibility in scientific research. COS operates the Open Science Framework (OSF), the most widely used pre-registration platform in academic research worldwide. Nosek coordinated the landmark Reproducibility Project: Psychology that produced the 2015 replication findings in Science and was named one of Nature's ten people who mattered in science in 2015.

The American Statistical Association (ASA)

The American Statistical Association, based in Alexandria, Virginia, issued the first institutional professional statement on p-value misuse in 2016, clarifying that p-values do not measure the probability the hypothesis is true and that scientific conclusions should not be based on whether a p-value passes a threshold. In 2019, the ASA's special journal issue called for abandoning "statistical significance" as a binary concept. This guidance is now referenced in statistics curricula across the US and is central to understanding how sampling methods interact with significance testing.

Ben Goldacre and AllTrials — UK

Ben Goldacre, physician and author of Bad Science and Bad Pharma, has been the most prominent public communicator in the UK on clinical trial misuse. His AllTrials campaign — co-founded with Sense About Science and BMJ — advocates for all clinical trials to be registered before they begin and results published regardless of outcome. The campaign has secured commitments from major pharmaceutical companies and policy changes at the European Medicines Agency and NHS England.

Effect Sizes, Confidence Intervals, and What to Report Instead

If p-values used in isolation are the problem, what should researchers report instead? The answer isn't to abandon statistical inference — it's to report it more fully. The three essential complements to (or replacements for) bare p-values are effect sizes, confidence intervals, and Bayesian factors.

What Is an Effect Size?

An effect size is a standardized measure of the magnitude of a phenomenon or difference, independent of sample size. Where a p-value tells you how likely results this extreme are under the null hypothesis, an effect size tells you how big the effect actually is. The most common measures include Cohen's d (for differences between two means), Pearson's r (for correlations), and eta-squared (for ANOVA designs).

Jacob Cohen, a statistician at New York University, published "Statistical Power Analysis for the Behavioral Sciences" in 1969, establishing conventions still used today: a Cohen's d of 0.2 is "small," 0.5 is "medium," and 0.8 is "large." His 1994 paper "The Earth Is Round (p < .05)" in American Psychologist remains one of the most powerful critiques of p-value culture ever written by a statistician. Effect sizes are now mandatory reporting requirements in Psychological Science, JAMA, and many other leading journals.

Confidence Intervals: The Information P-values Don't Give You

A 95% confidence interval (CI) provides a range within which the true population parameter falls in 95% of cases if the sampling process is repeated infinitely. It gives you both the direction and the precision of an estimate — information that a bare p-value entirely lacks. A p = 0.04 result with a confidence interval spanning from essentially zero to a massive effect size tells a very different story from one with a tight interval around a moderate effect. Reformers including the ASA now recommend that confidence intervals should accompany or replace p-values in most statistical reporting. Understanding logistic regression properly is a related essential skill for students working with confidence intervals in practice.

Bayesian Methods: A Fundamentally Different Approach

Bayesian statistics — developed into modern practice by Harold Jeffreys, Bruno de Finetti, and more recently Andrew Gelman at Columbia University — offers a fundamentally different inferential framework that sidesteps many p-hacking vulnerabilities. Instead of testing whether data are unlikely under the null hypothesis, Bayesian methods compute the Bayes factor: a direct comparison of how well the data are explained by the alternative hypothesis versus the null. Bayesian methods naturally penalize model complexity, don't have the arbitrary stopping-rule problem of NHST, and produce posterior probability distributions rather than binary pass/fail decisions. For students curious about where to begin, regularization in machine learning uses closely related Bayesian-flavored concepts.

How to Avoid P-hacking: Reforms, Tools, and Best Practices

The response to the p-hacking and data dredging problem has generated a significant and coherent reform movement. These reforms are being adopted at the level of journals, funding agencies, and universities. Students who understand them are better equipped to conduct research honestly and to evaluate the research they read.

Pre-registration: The Single Most Powerful Reform

Pre-registration means submitting a time-stamped, publicly available document specifying your hypothesis, sample size, data collection procedure, and analysis plan before collecting any data. It makes post-hoc hypothesis adjustment, outcome switching, and selective reporting immediately visible — anyone can compare the pre-registered plan against the published paper. The Open Science Framework (OSF) and ClinicalTrials.gov, operated by the US National Library of Medicine (part of NIH), are the two most important pre-registration platforms. In the UK, the ISRCTN registry serves a similar function for clinical trials. Pre-registration is now mandatory for many federally funded studies in the US and for all clinical trials in the UK NHS system.

A Nature Human Behaviour analysis of registered reports — where journals agree to publish studies regardless of the result, based on pre-specified designs — found that 44% produced null or negative results, compared to just 5–15% in conventionally published research. That difference reflects suppression of null results in the conventional system and inflation of positives through p-hacking.

Multiple Comparisons Correction

When a study tests multiple hypotheses, the family-wise error rate must be controlled. The most conservative method is the Bonferroni correction: divide the significance threshold (0.05) by the number of tests performed. If you run 10 tests, your per-test threshold becomes 0.005. Less conservative alternatives include the Benjamini-Hochberg procedure for controlling the false discovery rate (FDR), which is standard in genomics and neuroimaging at the Broad Institute of MIT and Harvard and the UK Biobank. Students running multiple tests in any statistics course should apply correction methods and understand why — understanding statistical calculations in Excel often precedes more advanced correction work.

Open Data and Reproducible Research

Sharing raw data and analysis code publicly — through platforms like OSF, GitHub, Dryad, or Zenodo — allows anyone to verify reported analyses. When another researcher can download the original dataset and run the analysis themselves, the scope for undetected p-hacking collapses dramatically. Science, Nature, and PLOS ONE all now have data sharing policies; many NIH and Wellcome Trust-funded studies are required to share data upon publication. The norm of treating data as a private researcher asset is actively shifting.

How to Detect P-hacking in a Published Study

Several practical red flags signal possible p-hacking. P-values that cluster suspiciously between 0.04 and 0.05 are a statistical fingerprint of selective reporting. Other red flags: many outcome variables measured but only a few reported; no pre-registration or trial registration; unusually large effect sizes in small samples (the winner's curse); post-hoc subgroup analyses presented as primary findings; and a paper where every single test is significant. Real data is messy — papers where everything works perfectly rarely reflect honest analysis. For students, mastering academic writing in research papers includes understanding which sources merit citation and how to evaluate study quality critically.

What This Means for Your Coursework, Dissertations, and Career

The misuse of statistics debate might feel distant from undergraduate statistics homework or a master's dissertation. It isn't. The same practices operate at all levels of research, and the same principles of honest analysis apply whether you're writing up a second-year research methods practical or a doctoral thesis at a Russell Group institution or Ivy League university.

How P-hacking and Data Dredging Affect Student Research

When students run data analyses, they face genuine analytical choices: which variables to include, whether to remove an outlier, which of several plausible statistical tests to use. Without awareness of how these choices affect results, it's easy to unconsciously gravitate toward the choices that produce significant results — especially under deadline pressure. Understanding common student mistakes in academic work includes recognizing when your analytical choices might be shaping your results rather than your results shaping your analysis.

The solution for students is the same as for professional researchers: decide your analysis plan before you analyze your data, document every analytical choice, and report all results — including null and unexpected ones. In many programs at UK universities including University of Manchester, King's College London, and University of Edinburgh, pre-registration of student research projects is now formally encouraged or required. In the US, programs at Cornell, Princeton, and Duke have embedded open science practices into undergraduate research training.

Writing About Statistics in Essays and Reports

When you cite statistical findings in an essay — claims that coffee prevents Alzheimer's, that mindfulness improves exam performance, that income inequality causes crime — you're implicitly making a judgment about the quality of the underlying evidence. In the current landscape, citing a single significant study as if it were established fact is a critical thinking failure. Strong academic writing evaluates evidence quality: Was the study pre-registered? Has it been replicated? What is the effect size? Is there meta-analytic support? For essays drawing heavily on empirical research, understanding how to build a proper argumentative essay includes handling the quality of your statistical evidence critically. If your statistics coursework hasn't covered these foundations, expert statistics homework support can fill in the gaps.

Statistical Power, Sample Size, and Why Underpowered Studies Lie

Statistical power is the probability that a study will detect a true effect when one exists. A study with low statistical power — typically because the sample size is too small — isn't just less likely to find real effects; it creates a specific problem that amplifies p-hacking damage. This is the winner's curse: in underpowered research, the only way a study achieves statistical significance is if the data happen to show an effect larger than the true one, due to sampling variation. Published small studies therefore systematically overestimate true effect sizes.

Cohen (1962) surveyed published psychology studies and found average statistical power of around 0.46 — meaning studies had only a 46% chance of detecting a real medium-sized effect. In clinical trials regulated by the FDA and MHRA (Medicines and Healthcare products Regulatory Agency) in the UK, sample size and power calculations are required by law. Free tools like G*Power (available from the University of Düsseldorf) make a priori power analysis straightforward for students. Understanding applied social statistics includes knowing how to plan an adequately powered study from the outset.

Frequently Asked Questions About P-hacking and Statistical Misuse