Most students do not struggle with the arithmetic of statistics. They struggle with the choice: should this data go through a t-test, a chi-square, an ANOVA, or something else? Pick the wrong test and the number you report, however carefully computed, answers a question nobody asked. This guide gives you a repeatable way to choose, explains the assumptions behind each test, and works two examples end to end.
It is written for engineering, science and management students, and for anyone analysing survey or experiment data for a project. You will not need software to follow the logic, although you will want a calculator or spreadsheet for the final numbers.
Start with three questions, not with the test
Before you look at a list of tests, answer these in order. They narrow the field to one or two candidates.
- What is the goal? Are you comparing groups, checking an association between two variables, or predicting one variable from another?
- What type of data is the outcome? Continuous numbers (marks, time, voltage), ordered categories (rating 1 to 5), or unordered categories (yes or no, brand A or B)?
- How are the groups related? Independent groups (different people in each), or paired (the same people measured twice, or matched pairs)?
Only after that do you check assumptions such as approximate normality and similar spread. Most wrong choices come from skipping the first three questions and jumping straight to the t-test, which is the only one many students remember.
The decision table
| Goal and data | Usual test | If assumptions fail |
|---|---|---|
| One sample mean vs a known value | One-sample t-test | Wilcoxon signed-rank test |
| Two independent groups, numeric outcome | Independent (Welch) t-test | Mann-Whitney U test |
| Same subjects measured twice | Paired t-test | Wilcoxon signed-rank test |
| Three or more independent groups, numeric outcome | One-way ANOVA | Kruskal-Wallis test |
| Two categorical variables | Chi-square test of independence | Fisher’s exact test (small counts) |
| Relationship between two numeric variables | Pearson correlation | Spearman rank correlation |
| Predict a numeric outcome | Linear regression | Transform data or use robust methods |
The “if assumptions fail” column holds the non-parametric alternatives. They work on ranks rather than raw values, so they cope better with outliers and skewed data, at the cost of slightly lower power when the data really are normal.
What the common assumptions actually mean
Assumptions sound scarier than they are. Here is what each one asks of you in practice.
- Independence. One observation should not influence another. Measuring the same student five times and treating them as five students breaks this, and no test can fix it afterwards.
- Approximate normality. For t-tests and ANOVA, the data (or the differences, for a paired test) should be roughly bell-shaped. With 30 or more observations per group, moderate departures matter much less because of the central limit theorem. With very small samples, look at a histogram or a normal probability plot before trusting the result.
- Similar spread. Classic t-tests and ANOVA assume the groups have similar variances. The Welch version of the t-test does not, which is why many textbooks and software now use it by default.
- Adequate counts. The chi-square test relies on expected counts in each cell being reasonably large, a common rule of thumb being at least 5. Below that, use Fisher’s exact test.
Worked example 1: comparing two groups with a t-test
A lecturer tries two revision methods. Five students use Method A and five different students use Method B. Their test scores are:
| Method | Scores | Mean | Variance |
|---|---|---|---|
| A | 78, 82, 85, 90, 75 | 82.0 | 34.5 |
| B | 70, 74, 80, 72, 76 | 74.4 | 14.8 |
Choose the test. The goal is to compare two groups, the outcome is numeric, and the groups are independent (different students). That points to an independent two-sample t-test, and because the variances differ, the Welch version is the safe choice.
Compute. The standard error of the difference is the square root of (34.5 ÷ 5 + 14.8 ÷ 5), which is the square root of 9.86, about 3.14. The difference in means is 82.0 − 74.4 = 7.6. The test statistic is t = 7.6 ÷ 3.14 ≈ 2.42. The Welch degrees of freedom work out to about 6.9.
Interpret. With roughly 7 degrees of freedom, the two-sided critical value at the 5% level is about 2.36, and our t of 2.42 is just past it, giving a p-value of roughly 0.046. So the data suggest a difference between the methods, but only just. With five students per group, a single different score could flip the conclusion, so the honest summary is “some evidence of a difference, and the study is too small to be confident.” That sentence is worth far more in a report than “p < 0.05, significant."
Worked example 2: two categories with a chi-square test
A team tests whether a redesigned sign-up page raises conversions. Of 100 visitors to the old page, 25 sign up. Of 100 visitors to the new page, 40 sign up.
| Signed up | Did not | Total | |
|---|---|---|---|
| New page | 40 | 60 | 100 |
| Old page | 25 | 75 | 100 |
| Total | 65 | 135 | 200 |
Choose the test. Both variables are categorical (page version, signed up or not), so this is a chi-square test of independence on a 2×2 table.
Compute. Expected count for each cell is row total × column total ÷ grand total. For the “new page, signed up” cell that is 100 × 65 ÷ 200 = 32.5. The other expected counts are 67.5, 32.5 and 67.5. Each cell’s contribution is (observed − expected)² ÷ expected, and every cell is 7.5 away from its expected value, so the contributions are 56.25 ÷ 32.5 ≈ 1.73 and 56.25 ÷ 67.5 ≈ 0.83, twice each. The chi-square statistic is therefore 2 × (1.73 + 0.83) ≈ 5.13.
Interpret. With one degree of freedom, the 5% critical value is 3.84, so 5.13 is significant, with a p-value of about 0.02. All expected counts exceed 5, so the test is valid. Remember that significance is not the same as importance: report the 15-point difference in conversion (40% against 25%) alongside the p-value.
Paired or independent: the most common slip
If the same ten students take a test before and after a course, the two columns of marks are not independent samples. Each student’s “after” score is linked to their own “before” score. A paired t-test works on the ten differences and removes the person-to-person variation, which usually makes it far more sensitive. Running an independent t-test on paired data throws that advantage away, and running a paired test on independent groups is simply wrong. Ask yourself, “can I draw a line from each value in one group to a specific value in the other?” If yes, it is paired.
More than two groups: why not just run several t-tests?
Suppose you compare four teaching methods. Running a t-test on every pair means six tests. Each test has a 5% chance of a false alarm, and across six tests the chance of at least one false alarm climbs to roughly 26%. One-way ANOVA tests all four group means in a single step. If it is significant, you follow up with a post-hoc test such as Tukey’s HSD, which adjusts for the multiple comparisons. If the data are skewed or ordinal, use Kruskal-Wallis instead.
Correlation, regression and what they do not tell you
Pearson’s r measures the strength of a straight-line relationship between two numeric variables, from −1 to +1. Spearman’s rank correlation does the same on ranks and handles curved but consistently rising or falling patterns. Neither proves that one variable causes the other. Ice-cream sales and drowning incidents rise together because both depend on hot weather, not because one drives the other. Regression goes one step further and gives an equation to predict the outcome, but the same warning about causation applies unless the data came from a designed experiment.
Sample size and power: decide before you collect
A test can only detect a difference your sample is large enough to reveal. Power is the probability of detecting a real effect of a given size. A common target is 80% power at a 5% significance level. Small samples often give “not significant” results simply because the study was underpowered, which is not the same as “no effect.” Plan your sample size before collecting data, and report effect sizes and confidence intervals, not just p-values.
A quick checklist before you run any test
- Write the question as a sentence: “Is X different between A and B?” or “Is X related to Y?”
- Identify the outcome type and whether groups are independent or paired.
- Plot the data. A histogram or box plot reveals outliers and skew in seconds.
- Choose the test, check its assumptions, and name the non-parametric backup.
- Report the effect size, a confidence interval, the sample size and the p-value together.
For quick number checks such as means, standard deviations and unit handling while you work, the free Free Technology Calculators app is handy, and the Free Technology Calculators guide lists everything it covers. Students in engineering streams may also like our article on free calculators for engineering students and the guide to unit conversions and dimensional analysis. More tools are on the products page.
Frequently asked questions
What does a p-value of 0.05 actually mean?
It is the probability of seeing a result at least this extreme if there were truly no effect. It is not the probability that your hypothesis is true, and it does not measure the size or importance of the effect.
Do I always have to test for normality?
With larger samples (roughly 30 or more per group) t-tests are fairly robust. With small samples, look at the plot and think about whether the data plausibly come from a bell-shaped process. If in doubt, run the non-parametric version too and see whether the conclusion agrees.
One-tailed or two-tailed?
Use a two-tailed test unless you decided, before seeing the data, that only one direction matters and the other would be meaningless. Switching to one-tailed after the fact to get a smaller p-value is not honest practice.
Can I use a t-test on ratings from 1 to 5?
Many researchers do when scales are averaged across several items, but single ordinal ratings are better analysed with Mann-Whitney or Wilcoxon tests, which do not treat the gaps between points as equal.
Disclaimer: This article is educational material and does not replace advice from your instructor, supervisor or a qualified statistician. Worked numbers are illustrative; verify results with statistical software before submitting academic or professional work.