p-values and Significance Levels (α)
Statistics Student Guide
p-Values and Significance Levels (α)
Everything you need to master hypothesis testing — what p-values actually measure, how significance levels (α) work, Type I & II errors, statistical power, one- vs two-tailed tests, and the Bonferroni correction — with real examples and exam-ready explanations.
The Core Concept
What Is a p-Value?
The p-value is the probability of observing data at least as extreme as your sample results, assuming the null hypothesis is true. That single sentence contains everything — and yet it is routinely misread, misquoted, and misapplied in published research, textbooks, and lecture halls alike. Getting this definition exactly right is the foundation of understanding every hypothesis test you will ever run. Statistics assignment help almost always begins here, because every other confusion about hypothesis testing flows from a misunderstanding of this number.
Let’s be very concrete. Suppose you are testing whether a new teaching method improves exam scores. The null hypothesis (H₀) says it makes no difference. You run your experiment, collect data, compute a test statistic, and get p = 0.03. This means: if the teaching method truly made no difference, the probability of seeing a difference at least as large as the one you observed, just by random sampling chance, is only 3%. That is strong evidence against H₀ — but not proof that the method works. The distinction matters enormously.
0.05
The most common significance threshold — a 5% acceptable risk of a false positive
≠
P-value does NOT equal the probability the null hypothesis is true — the most critical misconception to avoid
1925
Year Ronald Fisher introduced the 0.05 threshold — rooted in convention, not mathematical law
What Does a p-Value Actually Measure?
The p-value measures how surprising your data would be if the null hypothesis were true. It is a conditional probability — conditional on H₀ being true. It says nothing, directly, about whether H₀ is true. It says nothing about the size or importance of the effect. It says nothing about whether your results will replicate. It is a single piece of probabilistic evidence in a much larger picture. According to StatPearls (NCBI), statistical significance does not automatically imply clinical significance — a distinction that matters enormously in medicine and psychology.
Here is a common analogy that clarifies the logic. Imagine you flip a coin 20 times and get 17 heads. You hypothesize the coin is fair (H₀: probability of heads = 0.5). The p-value asks: if this coin were truly fair, how likely is it to get 17 or more heads in 20 flips, just by chance? If that probability is very small, your data are inconsistent with the fair-coin hypothesis. You might then conclude the coin is biased. Notice you did not compute “the probability the coin is fair” — that would require a Bayesian approach. The p-value works entirely within the frequentist framework, where H₀ is either true or not, and the p-value is about the data, not the hypothesis.
“The p-value is not the probability of the null hypothesis being true. It never was, and treating it as such is one of the most consequential statistical errors in modern science.” — Common framing in research methods courses at Stanford, Oxford, and University of Toronto.
The Formula: How p-Values Are Calculated
p-Values do not come from a single formula — they depend on the test statistic and the probability distribution used. For different tests, you compute a test statistic and then find the area under the appropriate distribution curve that is as extreme or more extreme than that value.
For a z-test (large sample, known population σ):
Test statistic: z = (x̄ − μ₀) / (σ / √n)
p-value (two-tailed) = 2 × P(Z ≥ |z|) = 2 × (1 − Φ(|z|))
For a t-test (small sample, unknown σ):
Test statistic: t = (x̄ − μ₀) / (s / √n) [with df = n − 1]
p-value = area under t-distribution beyond |t|
For a chi-squared test:
Test statistic: χ² = Σ [(Observed − Expected)² / Expected]
p-value = P(χ² ≥ observed χ²) — always one-tailed
In practice, software does this for you. Excel’s T.TEST(), Python’s scipy.stats.ttest_ind(), R’s t.test(), and SPSS all compute p-values automatically. But knowing what the software is computing — the area in the tail of a distribution — is essential for setting up the test correctly and interpreting its output accurately.
What is a Good p-Value?
There is no universally “good” p-value — only whether it is below or above your pre-set significance level α. A p-value of 0.049 and one of 0.001 both lead to rejection of H₀ at α = 0.05, but the latter provides far stronger evidence against H₀. Always think of the p-value as one piece of evidence, not a verdict.
Setting the Bar
What Is the Significance Level (α) and How Do You Choose It?
The significance level (α) is the threshold you set — before collecting any data — to decide when you will reject the null hypothesis. It represents the maximum probability of committing a Type I error you are willing to accept: concluding that an effect exists when it actually does not. If your p-value falls at or below α, you declare the result statistically significant and reject H₀. This is not a subjective judgment call made after seeing your results. It is a pre-committed decision rule that defines the logical structure of your test.
The most common significance level is α = 0.05, which accepts a 5% chance of a false positive. In high-stakes fields like medicine, pharmacology, and particle physics, stricter levels (α = 0.01 or α = 0.001) are standard. Exploratory social science research sometimes uses α = 0.10. The choice of α should always be justified by the consequences of each error type.
Why Is α = 0.05 the Default?
The 0.05 threshold traces back to Ronald Fisher — the British statistician who essentially invented modern hypothesis testing while working at Rothamsted Experimental Station in the UK in the 1920s. In his 1925 text Statistical Methods for Research Workers, Fisher described 0.05 as a convenient threshold — roughly 2 standard deviations from the mean in a standard normal distribution — and suggested it as a starting point for judging whether results were worth further investigation. He later clarified that he never intended it as a rigid rule, and that different fields should use different thresholds based on context. The 0.05 level persists today largely by scientific inertia.
Fisher’s own words on 0.05: He described it as a level at which a “scientifically minded person” might choose to investigate further — not as an absolute criterion of truth. He explicitly argued that a single significant result was insufficient evidence; replication was essential. This nuance is frequently lost in undergraduate statistics courses, where the binary reject/fail-to-reject framework can oversimplify the actual logic of inference.
Common Significance Levels and When to Use Them
α = 0.10 (10%): Exploratory research, pilot studies, social science
Higher power, higher Type I error risk
α = 0.05 (5%): Standard threshold — most academic research, business analytics
Balances Type I and Type II error risk
α = 0.01 (1%): High-stakes research — medical trials, policy evaluation
Strong evidence required; lower Type II error tolerable
α = 0.001 (0.1%): Particle physics, genome-wide association studies (GWAS)
Near-zero false positive tolerance; massive sample sizes typical
Significance Level vs P-Value: The Key Distinction
Significance Level (α)
- Set before data collection
- A fixed decision threshold
- Chosen by the researcher based on field norms and error tolerance
- Represents acceptable Type I error rate
- Does not depend on data
- Example: “We will use α = 0.05 for this study”
P-Value
- Calculated from data after collection
- Changes with every new dataset
- Computed by statistical software or test tables
- Represents probability of data given H₀ is true
- Compared to α to make a decision
- Example: “Our t-test produced p = 0.032”
This distinction is not merely semantic. Researchers who look at their data and then choose α to make their results significant are engaging in a practice known as p-hacking or researcher degrees of freedom — a major contributor to the replication crisis in psychology, nutrition science, and social science. Ethical statistical practice requires that α be pre-registered before data collection, precisely to prevent this from happening.
Struggling With Hypothesis Testing?
Our statistics experts explain p-values, significance levels, and every test from scratch — with step-by-step solutions and fast delivery.
Get Statistics Help Now Log InThe Full Framework
Hypothesis Testing: The Step-by-Step Process
Hypothesis testing is the formal statistical procedure that connects p-values and significance levels into a coherent decision-making framework. Every test — from a simple t-test comparing two group means to a complex ANOVA or chi-squared test — follows the same logical structure. Once you understand it deeply, you can apply it to any situation you encounter in research, assignments, or professional work.
1
State H₀ and H₁
The null hypothesis (H₀) assumes no effect, no difference, or no relationship. The alternative hypothesis (H₁) is what you aim to provide evidence for. Be specific. H₀: μ = 100 (mean IQ equals 100). H₁: μ ≠ 100 (two-tailed) or H₁: μ > 100 (one-tailed). The hypothesis must be stated before looking at data.
2
Set the Significance Level α
Choose your threshold — 0.05, 0.01, or 0.001 — based on the consequences of Type I and Type II errors. Document this choice before collecting or analyzing data. Pre-registration of α is standard practice in clinical trials and increasingly in social and behavioral science.
3
Select the Correct Test
Match the test to your data type and research design. One-sample t-test for comparing a sample mean to a known value. Two-sample t-test for comparing two independent groups. Paired t-test for before/after or matched pairs data. Chi-squared for categorical variables. ANOVA for multiple group comparisons. Each test has specific assumptions about distributions, independence, and variance.
4
Collect Data and Compute the Test Statistic
Gather your sample data. Compute the test statistic — z, t, F, χ², etc. — using the appropriate formula. This statistic converts your sample result into a standardized value that can be located on a probability distribution. Statistical software (R, SPSS, Python, Stata, Excel) handles this step, but you need to verify the inputs and check assumptions.
5
Find the p-Value
Find the probability of observing a test statistic as extreme as yours under H₀. For a two-tailed z-test with z = 2.1: p = 2 × P(Z ≥ 2.1) = 2 × 0.018 = 0.036. This is the area in the tail(s) of the distribution. Software computes this automatically; understanding what it represents is the critical skill.
6
Compare p to α and Decide
If p ≤ α: Reject H₀ — result is statistically significant. If p > α: Fail to reject H₀ — insufficient evidence against H₀. Never say “accept H₀” — failing to reject is not the same as proving H₀ true. The result is called non-significant, not “proven null.”
7
Report Effect Size and Confidence Interval
The p-value tells you whether an effect is detectable; the effect size tells you how large it is. Report Cohen’s d for t-tests, η² for ANOVA, r for correlation, odds ratios for logistic regression. Include a 95% confidence interval for the estimated parameter. These together give the full inferential picture.
A Complete Worked Example
A university researcher wants to test whether students using a new study app score higher on a statistics exam than the national average of 70%. She samples 36 students, finds a mean of 74%, with a sample standard deviation of 12%.
H₀: μ = 70 (the app makes no difference)
H₁: μ > 70 (one-tailed: the app improves scores)
α = 0.05
Test statistic (one-sample t-test):
t = (x̄ − μ₀) / (s / √n)
t = (74 − 70) / (12 / √36)
t = 4 / (12 / 6)
t = 4 / 2 = 2.00
Degrees of freedom: df = n − 1 = 35
p-value (one-tailed): P(t₃₅ ≥ 2.00) ≈ 0.027
Decision: p = 0.027 < α = 0.05 → Reject H₀
Conclusion: There is statistically significant evidence that students using
the app score higher than the national average (t(35) = 2.00, p = 0.027).
Effect size (Cohen’s d):
d = (74 − 70) / 12 = 0.33 → small-to-medium effect
What Is the Critical Region?
The critical region (also called the rejection region) is the set of test statistic values that would lead you to reject H₀. It corresponds directly to the significance level: at α = 0.05 for a two-tailed z-test, the critical region is all z-values with |z| > 1.96. At α = 0.01, the threshold is |z| > 2.576. The boundary between the critical and non-critical regions is called the critical value. If your computed test statistic falls in the critical region, your p-value is below α, and you reject H₀.
Critical Values for Common Tests (Two-Tailed, α = 0.05)
z-test: |z| > 1.96
t-test: |t| > t_{α/2, df} (look up in t-table by degrees of freedom)
χ²-test: χ² > χ²_{α, df} (one-tailed by convention)
F-test: F > F_{α, df1, df2} (look up in F-table)
Error and Power
Type I Errors, Type II Errors, and Statistical Power
Every hypothesis test can end in one of four outcomes — two correct decisions and two errors. Understanding these four possibilities is not just an exam topic. It is the foundation for understanding why the significance level and sample size choices you make before a study matter so much.
| H₀ Is Actually True | H₀ Is Actually False | |
|---|---|---|
| Reject H₀ | Type I Error (False Positive) Probability = α |
Correct Decision ✓ Probability = Power = 1 − β |
| Fail to Reject H₀ | Correct Decision ✓ Probability = 1 − α |
Type II Error (False Negative) Probability = β |
What Is a Type I Error?
A Type I error is a false positive — you reject H₀ when it is actually true. You conclude there is an effect when there is none. The probability of a Type I error equals your significance level α. If α = 0.05, there is a 5% chance you will incorrectly reject a true null hypothesis, even when everything in the study is done correctly. This is not a flaw in the method — it is an acknowledged, controlled risk.
Type I Error in Practice
A pharmaceutical company tests a drug with no real effect. Running the trial at α = 0.05, there is a 5% chance the trial shows a “significant” result by chance alone. If 20 pharmaceutical companies test 20 ineffective drugs at α = 0.05, on average 1 of those 20 drugs will appear to work by statistical accident — even if none of them do. This is precisely why replication matters, and why stricter α levels are required for regulatory approval of medical therapies.
What Is a Type II Error?
A Type II error is a false negative — you fail to reject H₀ when it is actually false. An effect genuinely exists, but your test misses it. The probability of a Type II error is denoted β (beta). Common acceptable values for β range from 0.10 to 0.20. Type II errors are particularly costly in medical screening (missing a real disease) and safety engineering (failing to detect a structural flaw).
What Is Statistical Power?
Statistical power = 1 − β. It is the probability of correctly detecting a real effect when one actually exists. Higher power means fewer missed discoveries. The conventional minimum for adequate power is 0.80 (80%).
Power depends on four interrelated factors:
- Sample size (n): Larger samples → more power. The single most controllable factor.
- Effect size: Larger effects are easier to detect. A drug that reduces blood pressure by 20 mmHg is easier to detect than one that reduces it by 2 mmHg.
- Significance level α: Higher α → more power (but more Type I error risk). Lowering α reduces power.
- Variability (σ): Lower variability → more power. Controlled experiments are more powerful.
Power Analysis (for a one-sample t-test):
Required n ≈ [(z_α + z_β) × σ / δ]²
Where:
z_α = critical z-value for significance level (1.645 for α=0.05 one-tailed)
z_β = critical z-value for power (0.842 for 80% power)
σ = population standard deviation (estimated)
δ = minimum effect size you want to detect (μ₁ − μ₀)
Example: Detecting a 5-point IQ difference (σ=15, α=0.05 two-tailed, 80% power):
n ≈ [(1.96 + 0.842) × 15 / 5]² = [2.802 × 3]² = 8.406² ≈ 71 participants
The Tradeoff: α vs Power
Here is the fundamental tension in hypothesis testing: reducing α to minimize Type I errors simultaneously reduces power and increases Type II errors. There is no setting that eliminates both. The only way to reduce both errors simultaneously is to increase sample size.
The multiple testing problem: Running many tests at α = 0.05 inflates the probability of at least one Type I error. Three tests give a 14% family-wise error rate. Ten tests give 40%. This is how researchers inadvertently generate false positives through data exploration — try enough analyses and something will appear significant by chance. The Bonferroni correction and its variants address this directly.
Test Direction
One-Tailed vs Two-Tailed Tests: When to Use Each
Every hypothesis test directs its evidence — either in one direction or two. This choice shapes how the critical region is distributed and, crucially, how easy it is to achieve statistical significance. Getting this choice wrong distorts your results and undermines the validity of your conclusions.
Two-Tailed Tests
A two-tailed test tests for effects in either direction: the parameter is significantly larger OR significantly smaller than the null value. The critical region is split equally between both tails of the distribution. At α = 0.05, each tail gets α/2 = 0.025. This is the default, and it is the right choice in most situations — when you genuinely do not have a strong, pre-specified directional prediction.
Two-tailed at α = 0.05:
Critical values: z < −1.96 OR z > 1.96
p-value = 2 × P(Z ≥ |z_observed|)
Example: Observed z = 2.10
p = 2 × P(Z ≥ 2.10) = 2 × 0.018 = 0.036 → Reject H₀ at α = 0.05 ✓
One-Tailed Tests
A one-tailed test directs all evidence toward one tail — either the left (testing if the parameter is significantly smaller) or the right (testing if it is significantly larger). The full α is concentrated in one tail, making it easier to achieve significance in that direction — but only in that direction.
One-tailed (right-tail) at α = 0.05:
Critical value: z > 1.645
p-value = P(Z ≥ z_observed)
Example: Observed z = 1.70
One-tailed p = P(Z ≥ 1.70) = 0.045 → Reject H₀ ✓
Two-tailed p = 2 × 0.045 = 0.089 → Fail to reject H₀ ✗
Same data, different decisions based on the directional specification.
This example illustrates why the test direction is a critical choice — and why it must be made before seeing the data. A researcher who computes a two-tailed p = 0.09, decides it is “almost significant,” and switches to a one-tailed test post-hoc to achieve p = 0.045 is engaging in p-hacking. Use one-tailed tests only when: (a) you have a strong prior theoretical reason to expect the effect only in one direction, and (b) an effect in the opposite direction would be substantively meaningless or impossible.
“The choice between one- and two-tailed tests should be made as part of the study design, based on the scientific question — not post-hoc based on what makes the result ‘significant.'” — Standard guidance in APA Publication Manual (7th edition).
Multiple Comparisons
Multiple Testing Problem and the Bonferroni Correction
When you run a single hypothesis test at α = 0.05, you accept a 5% chance of a false positive. But when you run multiple tests on the same dataset, the probability of getting at least one false positive across all tests grows — fast. This is the multiple testing problem, and it is one of the most common sources of spurious findings in modern research.
Why Multiple Testing Inflates False Positives
P(at least one false positive) = 1 − (1 − α)^k
k = 1: 1 − (0.95)¹ = 0.050 (5% — as expected)
k = 5: 1 − (0.95)⁵ = 0.226 (22.6%!)
k = 10: 1 − (0.95)¹⁰ = 0.401 (40%!)
k = 20: 1 − (0.95)²⁰ = 0.642 (64%!)
At 20 tests, there is a 64% chance of at least one false positive
— even if none of the null hypotheses are false.
The Bonferroni Correction
The simplest and most widely known correction is the Bonferroni correction. The idea is straightforward: divide your desired family-wise error rate by the number of tests to get the adjusted α for each individual test.
Bonferroni Correction:
α_adjusted = α / k
Example: You run 5 post-hoc comparisons after ANOVA. α = 0.05.
α_adjusted = 0.05 / 5 = 0.01
Each individual test must achieve p ≤ 0.01 to be declared significant.
The family-wise error rate is controlled at approximately 5%.
For GWAS with k = 1,000,000 tests:
α_adjusted = 0.05 / 1,000,000 = 5 × 10⁻⁸
This is the genome-wide significance threshold.
The Bonferroni correction is conservative — it tends to reduce power substantially, increasing Type II errors. When many tests are correlated, it over-corrects. Alternative corrections like the Benjamini-Hochberg procedure (which controls the False Discovery Rate rather than the family-wise error rate) offer a better power-error tradeoff for large-scale testing problems.
When Is the Bonferroni Correction Required?
Apply the Bonferroni correction whenever you conduct multiple hypothesis tests on the same data and want to control the risk of any false positive. This includes: post-hoc comparisons after a one-way ANOVA, testing multiple outcomes in a clinical trial, testing the same hypothesis across multiple subgroups, and any exploratory analysis where many statistical tests are conducted simultaneously.
Need Help With Multiple Comparisons or ANOVA?
Our statistics experts handle Bonferroni corrections, post-hoc tests, power analysis, and every aspect of hypothesis testing — with clear explanations.
Start an Order Login to AccountWhat p-Values Are NOT
The Most Dangerous Misconceptions About p-Values
The misinterpretation of p-values is not just a student problem. It has driven flawed conclusions in peer-reviewed journals across medicine, psychology, economics, and nutrition for decades. The American Statistical Association issued a formal statement on p-values in 2016 specifically because misuse had become so widespread.
Misconception 1: “p < 0.05 proves the alternative hypothesis is true”
Wrong. A significant p-value means your data are inconsistent with H₀ at your chosen α level. It does not prove H₁. A single study with p = 0.03 is suggestive, not conclusive. Evidence accumulates through replication and meta-analysis — not through a single p-value crossing a threshold.
Misconception 2: “p = 0.05 means there is a 5% probability the null hypothesis is true”
Wrong — this is the most common and most consequential misreading. The p-value is P(data | H₀ true), not P(H₀ true | data). The latter is a posterior probability, which requires Bayesian methods and a prior probability for H₀. The frequentist p-value makes no probabilistic statement about H₀ being true or false.
Misconception 3: “A larger p-value means more evidence for H₀”
Wrong. Failing to reject H₀ is not the same as evidence that H₀ is true. A non-significant result could mean: (a) H₀ really is true, (b) the study lacked power to detect a real effect, or (c) the effect is real but smaller than what the study was designed to detect.
Misconception 4: “p = 0.049 and p = 0.051 are meaningfully different”
Wrong. The 0.05 threshold is a convention, not a law of nature. The difference in evidence between p = 0.049 and p = 0.051 is infinitesimal. Many journals now require reporting of exact p-values and effect sizes so readers can evaluate the full strength of evidence.
Misconception 5: “Statistical significance implies practical significance”
Wrong — and enormously important in applied work. With a large enough sample, even a trivially small effect becomes statistically significant. Always evaluate effect size (Cohen’s d, r², odds ratio) alongside p-values to assess whether a finding matters, not just whether it exists.
The ASA’s stance on p-values (2016): The American Statistical Association stated that scientific conclusions should not be based only on whether a p-value passes a specific threshold. Decisions should be based on the totality of evidence, including effect sizes, confidence intervals, study design, prior evidence, and replication. Reporting p < 0.05 as the sole criterion for a finding’s validity is, in their words, an obstacle to scientific progress.
Beyond p-Values
Effect Sizes and Confidence Intervals: The Full Picture
No serious modern statistics course teaches p-values in isolation. The p-value tells you whether an effect is detectable given your sample size and significance threshold. The effect size tells you how big the effect is. The confidence interval tells you the plausible range for the true parameter. Together, they give you the complete inferential picture.
Common Effect Size Measures
| Test | Effect Size Measure | Small | Medium | Large | Interpretation |
|---|---|---|---|---|---|
| t-test (two groups) | Cohen’s d | 0.2 | 0.5 | 0.8 | Standardized mean difference in SD units |
| ANOVA | η² (eta-squared) | 0.01 | 0.06 | 0.14 | Proportion of variance explained |
| Correlation | r (Pearson’s) | 0.10 | 0.30 | 0.50 | Strength of linear relationship |
| Chi-squared | Cramér’s V | 0.10 | 0.30 | 0.50 | Association between categorical variables |
| Logistic regression | Odds Ratio | 1.5 | 2.5 | 4.0 | Ratio of odds of outcome between groups |
| Linear regression | R² / f² | 0.02 | 0.15 | 0.35 | Proportion of variance in outcome explained |
Confidence Intervals and Their Relationship to p-Values
A confidence interval (CI) for a parameter θ gives a range of plausible values for the true parameter, based on your sample data. A 95% CI means: if you repeated this study many times, 95% of the intervals constructed would contain the true parameter value.
95% CI for a one-sample mean:
CI = x̄ ± t_{α/2, df} × (s / √n)
Example: x̄ = 74, s = 12, n = 36, df = 35, t₀.₀₂₅,₃₅ ≈ 2.030
CI = 74 ± 2.030 × (12/6)
CI = 74 ± 2.030 × 2
CI = 74 ± 4.06
95% CI: (69.94, 78.06)
Interpretation: The interval does NOT include 70 (the null value) →
consistent with rejecting H₀ at α = 0.05. The CI and the test agree.
This is the key relationship: a 95% CI that does not include the null value corresponds exactly to a two-tailed test with p ≤ 0.05. The CI gives additional information: not just whether the null is rejected, but where the true effect likely lies and how precisely it was estimated.
Real-World Context
p-Values and Significance Levels in Practice: Across Disciplines
P-values and significance levels appear in virtually every research domain that uses quantitative data. Knowing how different fields apply these concepts — and how their conventions differ — gives you both broader understanding and the ability to read research across disciplines.
In Medicine and Clinical Trials
Clinical trials use rigorous hypothesis testing frameworks with pre-registered protocols. The FDA in the US and EMA in Europe require randomized controlled trials (RCTs) to demonstrate significance at α = 0.05 (often 0.01 for confirmatory trials) before approving a drug or medical device. But significance alone is insufficient — clinical significance, measured by effect size and absolute risk reduction, is equally required. A drug that reduces mortality by 0.001% might achieve p < 0.0001 in a large enough trial but still not justify widespread clinical adoption.
In Psychology and Social Science
Psychology has been at the epicenter of the replication crisis precisely because the field relied too heavily on p < 0.05 as the sole criterion for publishable findings, often with small, underpowered samples. The Open Science Collaboration replicated 100 psychology studies in 2015 and found that fewer than 40% produced significant results the second time. This catalyzed widespread reform, including pre-registration, larger samples, reporting of effect sizes, and meta-analysis.
In Economics and Finance
Econometricians use p-values in regression analyses, difference-in-differences designs, instrumental variables, and causal inference frameworks. A p-value of 0.05 is the conventional threshold, but economic interpretation focuses heavily on the magnitude and direction of estimated coefficients — not just their significance. A wage regression that shows a statistically significant gender wage gap is important, but the 95% confidence interval for the gap size is what drives policy interpretation.
In Data Science and A/B Testing
Technology companies like Google, Meta, Amazon, and Netflix run thousands of A/B tests — randomized experiments comparing two versions of a product, page, or algorithm — every year. Each test is, at its core, a hypothesis test with a p-value and significance threshold. The practical stakes are enormous: a feature that appears to improve click-through rate by 0.5% across hundreds of millions of users is statistically detectable but may or may not justify the engineering cost and user experience change.
Statistics Assignment Due Soon?
From hypothesis testing and p-values to effect sizes and confidence intervals — our experts deliver complete, step-by-step solutions fast.
Get Help Now Log InExam Mastery
How to Answer p-Value and Hypothesis Testing Questions on Exams
Hypothesis testing questions follow predictable patterns in statistics exams at every level — from introductory courses at community college to graduate qualifying exams at research universities. Knowing these patterns means you can answer confidently and completely, even under time pressure.
The Non-Negotiable Checklist
- Always state H₀ and H₁ explicitly. Do not assume the marker knows what you mean. Write them as formal mathematical statements — e.g., H₀: μ₁ = μ₂ vs H₁: μ₁ ≠ μ₂.
- Always state α before computing anything. Even if the question specifies α = 0.05, write it down. It signals to the marker that you understand the decision framework.
- Identify the correct test and state its assumptions. t-test requires approximately normal population or large n. Chi-squared requires expected cell counts ≥ 5. Failure to check assumptions costs marks at postgraduate level.
- Show the test statistic formula with values substituted. Never just write the final number — show the computation step by step.
- State the decision rule precisely. “Reject H₀ if p ≤ 0.05” or “Reject H₀ if |t| > 1.96.” Then apply it to your computed value.
- Write a conclusion in context. Not just “reject H₀” — but “there is sufficient evidence at the 5% significance level to conclude that the mean exam score differs from 70%.”
- Include effect size if the question asks for practical significance.
Quick Reference: Which Test to Use
One quantitative variable, known σ, compare to fixed value → z-test
One quantitative variable, unknown σ, compare to fixed value → one-sample t-test
Two independent groups, quantitative → independent samples t-test
Two related groups (before/after, matched pairs) → paired t-test
Three or more groups, quantitative → one-way ANOVA
Two categorical variables → chi-squared test of independence
One categorical variable, compare to expected → chi-squared goodness-of-fit
Quantitative predictor → outcome relationship → linear regression
Binary outcome, quantitative or categorical predictors → logistic regression
Never write “accept H₀”. Failing to reject the null hypothesis is not the same as proving it. The correct language is “fail to reject H₀” or “there is insufficient evidence to reject H₀.” Writing “accept H₀” suggests you believe you have proven the null — and will cost you marks in every statistics course at every university in the US and UK.
Frequently Asked
Frequently Asked Questions About p-Values and Significance Levels
What is a p-value in statistics?
A p-value is the probability of observing data at least as extreme as your sample results, given that the null hypothesis (H₀) is true. It ranges from 0 to 1. A small p-value — typically ≤ 0.05 — means your data would be very unlikely under H₀, providing evidence to reject it. A large p-value means your data are reasonably consistent with H₀. Critically, the p-value does NOT measure the probability that H₀ is true, and it does not measure the probability that your result occurred by chance alone — both are common and consequential misinterpretations.
What is the significance level (α) and how do I choose it?
The significance level α is the threshold you set before an experiment for rejecting the null hypothesis. It equals the maximum acceptable Type I error rate — the probability of falsely detecting an effect that does not exist. Common values: 0.05 for most academic and business research; 0.01 or 0.001 for high-stakes fields like medicine and genetics. Choose α based on the consequences of each error type. If a false positive would be very costly (e.g., approving an ineffective drug), use a stricter α.
What is the difference between a p-value and a significance level?
The significance level (α) is set by the researcher before data collection — it is a fixed decision threshold. The p-value is calculated from the data after the experiment. You compare them to make a decision: if p ≤ α, the result is statistically significant and you reject H₀; if p > α, you fail to reject H₀. Think of α as the bar you set in advance, and the p-value as whether your data clear that bar. Critically, α never changes; the p-value is a function of your specific data and varies each time the experiment is run.
What are Type I and Type II errors in hypothesis testing?
A Type I error is a false positive: you reject H₀ when it is actually true. Its probability equals α. A Type II error is a false negative: you fail to reject H₀ when it is actually false. Its probability is denoted β. These two errors are inversely related: reducing α (fewer false positives) increases β (more missed effects). The only way to reduce both errors simultaneously is to increase sample size. Statistical power (1 − β) is the probability of correctly detecting a true effect, and most studies aim for power ≥ 0.80 (80%).
Why is α = 0.05 used as the standard significance level?
The 0.05 threshold was popularized by statistician Ronald Fisher in his 1925 book Statistical Methods for Research Workers. Fisher described it as a convenient round number — approximately 2 standard deviations from the mean in a normal distribution — that represented a reasonable threshold for declaring results worth investigating further. He never intended it as a rigid law of nature. The 0.05 level persists today largely by tradition, not mathematical necessity.
Does a p-value below 0.05 prove my hypothesis is correct?
No — and this is one of the most important misconceptions to overcome. A p-value below 0.05 means your data are inconsistent with the null hypothesis at the 5% level. It does not prove the alternative hypothesis is true. It does not rule out all other explanations. It does not mean the effect is large or practically important. A single significant p-value provides evidence — not proof. Strong scientific conclusions require replication, meta-analysis, large effect sizes, and theoretical coherence.
What is statistical power and why does it matter?
Statistical power (1 − β) is the probability that a hypothesis test correctly rejects a false null hypothesis — i.e., detects a real effect when one truly exists. Low power means you will frequently miss real effects (Type II errors). Most researchers target power ≥ 0.80 (80%). Power depends on sample size (more participants = more power), effect size (larger effects are easier to detect), significance level α (higher α = more power), and data variability (less noise = more power).
What is the Bonferroni correction and when should I use it?
The Bonferroni correction adjusts the significance level when you conduct multiple hypothesis tests on the same data. Without correction, the probability of at least one false positive grows with every additional test (e.g., 10 tests at α = 0.05 gives a 40% false positive rate). The correction: divide α by the number of tests (α_adjusted = 0.05 / k). Use it when running multiple post-hoc comparisons after ANOVA, testing multiple outcomes in a study, or running many simultaneous tests. Note: Bonferroni is conservative and reduces power; the Benjamini-Hochberg FDR correction may be more appropriate for large-scale testing.
What is the difference between one-tailed and two-tailed tests?
A two-tailed test detects effects in either direction (greater than or less than the null value), distributing α/2 in each tail. It is appropriate when you have no specific directional prediction. A one-tailed test concentrates all of α in one tail, making it easier to detect an effect in one specific direction. Use one-tailed tests only when you have a strong, pre-specified theoretical reason to expect the effect to go only one way. Always specify directionality before data collection. Switching from two-tailed to one-tailed after seeing results to achieve significance is p-hacking.
Can I use p-values alone to interpret my research results?
No. The American Statistical Association, APA, and most major journals now explicitly require that p-values be accompanied by effect sizes and confidence intervals. A p-value tells you whether an effect is detectable at your significance level — not how large it is, how practically important it is, or how certain your estimate is. Always report all three: p-value, effect size, and confidence interval.