A mentee in a public health training programme stopped me after a workshop last month. She had two studies on the same intervention open on her laptop. One reported a large effect with a tiny p-value. The other found nothing. Her question was simple: 'Which one do I trust?' I used to answer that question by reciting definitions. Now I start with the part most statistics fundamentals explanations skip: what the number was actually trying to tell us.
Statistics is not a machine you feed data into and receive truth. It is a set of tools for measuring uncertainty. Once that shift happens, the rest of the fundamentals become easier to keep straight.
The sample and the population
Almost every statistical claim starts with a sample standing in for a larger population. A trial of 2,000 patients is meant to say something about millions. A poll of 1,000 voters is meant to estimate a country's mood. Scientists can rarely measure everyone, so they measure a subset and accept a trade: the sample introduces uncertainty in exchange for practicality.
What makes a sample trustworthy is how it is chosen. A random sample gives every eligible person or unit a known chance of being included. That is why national polling organisations can interview around 1,000 people and get reasonably close to the national vote share, while an online poll with 200,000 self-selected respondents can be badly wrong. The larger number feels more precise, but the selection process matters more than size.
The difference between population and sample also explains two terms that get confused. The standard deviation measures how spread out individual values are within a set of data. The standard error measures how much a sample estimate, such as a mean, would vary from sample to sample. A huge standard deviation tells you the people or measurements differ a lot. A small standard error tells you the estimate is fairly stable. Both can be true at once.
Why a p-value is smaller than you think
A p-value is one of the most misused numbers in science. It is not the probability that the result happened by chance. It is not the probability that the null hypothesis is false. The American Statistical Association spelled this out in a widely cited 2016 statement: a p-value is the probability of seeing data at least as extreme as the observed data, assuming there is no real effect or difference.
Suppose a clinical trial finds a difference in blood pressure between two groups and reports p = 0.03. That means: if the two treatments truly had no difference, you would expect to see a difference this large in about 3% of identical experiments. It is a statement about the data under an assumed world, not a direct statement about the world itself.
The common threshold of 0.05 is a convention, not a border between true and false. R.A. Fisher introduced it in the 1920s as a rough cut-off for further investigation, and it has hardened into a ritual. A result at p = 0.049 and one at p = 0.051 are nearly the same evidence, but many journals treat the first as publishable and the second as unremarkable. The American Statistical Association's statement on p-values warns against making decisions on bright-line thresholds alone.
Because 0.05 is a gate, scientists can unconsciously or deliberately push results under it. They might add participants until significance appears, test several outcome measures, report only the most promising, or run many subgroup analyses. This cluster of behaviours is called p-hacking. A 2016 article in Nature documented how routine the misuse of p-values had become across disciplines.
The consequence is not abstract. When effects are small and samples are flexible, a p-value below 0.05 can be routine even when there is no true effect. This helps explain why many high-profile findings have failed to replicate. A study might be statistically significant and still be wrong.
Confidence intervals and the size of the effect
A confidence interval gives a range of plausible values for the true effect. If a study estimates that a new teaching method improves scores by 4 points with a 95% confidence interval from 1 to 7 points, the researchers are saying the data are compatible with effects as small as 1 point and as large as 7. The interval does not mean there is a 95% chance the true value is between 1 and 7. It means that if the same procedure were repeated many times, about 95% of the calculated intervals would capture the true value.
The interval conveys precision. A narrow interval means the estimate is stable. A wide interval means the study cannot pin the effect down. If the interval includes zero, the result may be compatible with no effect at all, even if a p-value is less than 0.05.
Effect size addresses a different question: not whether a difference exists, but whether it matters. A blood pressure drug might lower systolic pressure by 0.8 mmHg, and with a sample of 20,000 people that tiny effect could have a p-value below 0.001. But a clinician would probably not consider half a point clinically meaningful. Another treatment might lower blood pressure by 8 mmHg with more uncertainty and be far more important. Reporting effect size keeps statistical significance from being confused with practical significance.
Correlation, causation, and the confounder
Two variables moving together does not mean one caused the other. Ice cream sales and drowning deaths rise together in summer, but buying ice cream does not make people drown. Heat is the confounder: a third variable that affects both.
Scientists use regression to estimate a relationship while adjusting for measured confounders. A regression model says, in effect: after accounting for age, smoking status, and other recorded variables, what is the remaining association? The key word is measured. If an important confounder is not in the data, regression cannot fix it. That is why observational studies can suggest associations but rarely prove causation on their own.
Randomised controlled trials work because random assignment breaks the link between treatment and confounders. The flip of a coin decides who gets the intervention, so any differences that emerge are more plausibly caused by the treatment itself. This is the foundation of modern evidence-based medicine. The Seeing Theory project from Brown University offers visual explanations of these ideas, including probability, randomisation, and regression.
A different kind of probability
Most school statistics are built on frequentist logic: probability is the long-run frequency of events. Bayesian statistics treats probability as a degree of belief that gets updated as new evidence arrives. You start with a prior belief, multiply it by the likelihood of the new data, and produce a revised belief called the posterior.
This approach handles rare events more intuitively. Consider a screening test for a condition that affects 1 in 1,000 people. The test is 95% sensitive and 95% specific. If a person tests positive, many people assume the chance of having the condition is 95%. It is closer to 2%. The reason is the false positives: 5% of the 999 healthy people is about 50 positive results, while the test finds fewer than one of the 1 true cases. The rare condition means most positive results are wrong.
Bayesian methods underpin spam filters, weather forecasts, election models, and some modern clinical trial designs. They do not replace frequentist methods; they answer a related but different question with different machinery.
Reading a number without being fooled
These fundamentals translate into a short checklist. I use it whenever a headline, tweet, or preprint makes a statistical claim.
- What is the sample, and how was it selected? A volunteer sample or an online opt-in is less reliable than a random sample.
- What exactly is being measured? A change in cholesterol is not the same as a change in heart attacks.
- How big is the effect, not just the p-value? Ask for the confidence interval and the real-world magnitude.
- Who paid for the study, and was the analysis plan registered before the data arrived?
- Could an unmeasured third variable explain the association?
None of these questions requires a statistics degree. They require slowing down long enough to ask them. That pause is the real fundamental skill: treating a number as a claim to be examined rather than a verdict to be accepted.
If you want one small step this month, pick a health or policy headline that reports a percentage, find the original study or summary, and check whether it gives a confidence interval or effect size. You will quickly see which stories are telling you the number and which are telling you the range of doubt.
Photo by Sharad Bhat on Unsplash