Academic Jobs - Home of Higher Ed Logo

negative binomial distribution: discovery and impact in probability theory and statistics

10 min read0 vues

At a glance

The negative binomial distribution began as a question about trials in a French gambling manual, became the standard fix for overdispersed counts in 1920, and now underpins superspreading models and RNA-seq pipelines. A close look at how it got there, what its dispersion parameter actually means, and where a good fit can still mislead you.

Mathematical equations are written on a white page
Photo by Bozhin Karaivanov on Unsplash

Six hundred and forty-seven women, five weeks, one factory floor. Major Greenwood and Udny Yule put the accident counts into the Journal of the Royal Statistical Society in 1920, and the table has been photocopied into statistics courses ever since: 447 women had no accident, 132 had one, 42 had two, 21 had three, and five had four or more.

Poisson's law, the default model for rare independent events, gets the middle roughly right and both tails wrong. It expects about 406 women in the zero column, which is close, but 189 in the one-accident column where 132 were observed, and fewer than eight women with three or more accidents against the 26 who actually recorded them. The sample mean was 0.46. The sample variance was 0.69. Poisson insists those numbers are equal.

Greenwood and Yule stopped pretending the workers were interchangeable. If each woman's accident rate is itself drawn from a gamma distribution, and the counts are Poisson around that personal rate, the aggregate is the negative binomial distribution: a count model with two parameters instead of one, a mean and a dispersion term that lets the variance run ahead of the mean without moving the centre.

That same distribution now sits inside the tools epidemiologists use to describe superspreading, bioinformaticians use to call differentially expressed genes, and actuaries use to price motor cover. Its history runs from a seventeenth-century gambling dispute through a shell factory in the First World War to single-cell sequencing, and one property has followed it the whole way. It is the distribution you reach for when your counts are lumpier than Poisson allows.

Two Histories, One Family

The algebra came first, and it came from cards and dice. Blaise Pascal mentioned the distribution in work published posthumously in 1679, and Pierre Rémond de Montmort gave it a proper treatment in the second edition of his Essay d'analyse sur les jeux de hasard in 1713, working out the probability of the number of trials needed to reach a fixed number of successes. That waiting-time version still carries the name Pascal distribution, and when a single success is required it collapses into the geometric distribution every first-year course teaches.

Two centuries later the same probability mass function turned up from the opposite direction, as a mixture. Greenwood and Yule's 1920 derivation, and the urn model that Florian Eggenberger and George Pólya published in 1923, both arrive at the negative binomial by letting a Poisson rate vary across individuals rather than by counting trials to a target.

The two readings produce identical formulas, which is convenient and occasionally misleading. In one you're waiting for successes in a sequence of Bernoulli trials. In the other you're watching a population that is heterogeneous in a way you cannot observe directly. Most modern users are doing the second, whether they say so or not.

With mean μ and dispersion k, the variance is μ plus μ² divided by k. Let k run to infinity and you recover the Poisson; set k equal to 1 and you get the geometric. Everything in between describes counts that are overdispersed but not hopeless. The name itself is purely algebraic: the mass function can be written as a binomial coefficient with a negative upper index, so it falls out of expanding a binomial to a negative power. The textbook parameters are a target count and a per-trial probability, which tells you where the formula spent its first two centuries.

A Good Fit Is Not a Mechanism

Overdispersion, variance greater than the mean, is the tell, and people spotted it early. William Sealy Gosset, writing as Student in Biometrika in 1907, counted yeast cells in a haemocytometer and found more spread than Poisson sampling could explain. Greenwood and Yule's fit did the same job on the accident data: a chi-square statistic of about 4 for the negative binomial against roughly 100 for Poisson on the same table.

Accidents in five weeksWomen observedExpected under PoissonExpected under negative binomial
0447406.3445.9
1132189.0134.9
24244.044.0
3216.814.7
430.85.0
5 or more20.12.5

What the fit cannot tell you is why the counts are lumpy. A 1943 review of contagious distributions made the point sharply: a Pólya or negative binomial distribution fits beautifully whether the underlying process is genuinely contagious or the population is simply inhomogeneous in a way the observer cannot see. Separating those explanations takes more than a marginal distribution. You need the correlation between events in successive time periods, or covariates that account for the heterogeneity, or both.

Ronald Fisher's 1941 note in the Annals of Eugenics is the paper that moved the negative binomial into routine practice, and its subject was estimation rather than application. With two parameters, the method of moments is inefficient for the exponent, so Fisher calculated the theoretical efficiency that tells a user whether a full maximum-likelihood fit is worth the arithmetic. His worked example was tick counts on sheep, which is as good an illustration as any of how unglamorous the founding data of applied statistics tend to be.

That framing still holds. The sample mean is an efficient estimate of μ almost regardless of how the data arrived. The dispersion parameter is where the arguments happen, where software packages disagree, and where a careless default can quietly change your conclusions.

Superspreading Is Overdispersion With a Public Health Budget

The epidemiological version arrived in 2005, when Joseph Lloyd-Smith and colleagues published a model in Nature in which each infected person has an individual reproductive number drawn from a gamma distribution with mean R0 and dispersion k. Poisson processes with gamma-distributed intensity produce negative binomial offspring, so the number of secondary cases per person is negative binomial with the same two parameters. The paper fitted SARS, measles, smallpox, monkeypox and pneumonic plague data, and the negative binomial beat the homogeneous alternatives across the board.

For the Singapore SARS outbreak the estimated k was about 0.16. Under that fit, roughly 73 per cent of cases infected fewer than one other person on average, while about 6 per cent infected more than eight: the clinical shorthand is that most people barely transmit and a few transmit a lot. As k goes to infinity the distribution becomes Poisson and transmission is uniform; at k equal to 1 it is geometric. Small k means extreme heterogeneity, and k in outbreaks is small.

A 2023 reanalysis in Scientific Reports refitted the same model across sixteen secondary-case datasets and put median k estimates between 0.03 for MERS-CoV and 0.85 for SARS-CoV-2, with SARS-CoV-2 the most variable of the pathogens examined. That range, not a single number, is the honest summary of how much individual variation in infectiousness these outbreaks showed.

The policy consequence is not intuitive, which is why the 2005 paper mattered. For a fixed R0, more heterogeneity lowers the chance that any single introduction sparks an epidemic and raises the chance that a spark becomes an explosion. Control aimed at individuals, such as isolation, increases heterogeneity, so it can extinguish an outbreak with less total intervention than a blanket measure applied to everyone.

Genomics Runs on It

Sequencing made the negative binomial infrastructure. In 2007 Mark Robinson and Gordon Smyth proposed modelling digital gene expression counts with a negative binomial and moderating the level of overdispersion across genes using conditional weighted likelihood, a method that stayed usable even with the smallest number of libraries. That work became edgeR, and the same error model sits inside DESeq2. Both fit a negative binomial generalised linear model, estimate a mean-variance trend across genes, and shrink each gene's dispersion toward that trend. Shrinkage is what lets a lab compare groups with three replicates each, which is roughly where sequencing budgets landed for a decade.

In single-cell data the pattern repeats with a wrinkle. Analyses of a wide range of datasets found clear overdispersion in every one, and found that the overdispersion parameter varied substantially across datasets, which argues against any fixed default. That is why the variance-stabilising tools built for single-cell work estimate the parameter empirically rather than assuming it.

And the caveat from 1943 resurfaces in new clothes. A 2026 paper in PLOS Computational Biology compared the negative binomial against the Beta-Poisson distribution predicted by the two-state model of gene expression and found that the negative binomial approximates simulated data well only within an intermediate band of kinetic parameters, a band that widens with lower mean expression, more technical noise and larger samples. A good negative binomial fit, in other words, does not demonstrate transcriptional bursting. Burst parameters recovered from such fits carried large relative errors even after technical noise was corrected, although ranking genes by burst frequency stayed reliable.

A postdoc in a neighbouring lab, call her Dr. M, learned the practical version of that lesson the hard way. Three replicates a group, a standard pipeline, and a list of roughly 1,800 genes at a 5 per cent false discovery rate; when the experiment was repeated with six replicates a group, the list fell by about half and the dispersion estimates walked toward the fitted trend. Nothing was wrong with the software. The dispersion had simply been under-estimated, and every downstream p-value inherited the error.

Claims, Crashes and the Cost of a Wrong Variance

Actuarial work is where the distribution earns its keep in cash. Claim counts per policy are dominated by zeros and long on the right tail, so the equidispersion assumption in Poisson regression fails almost immediately. A worked example published in The Actuary in 2024, using a portfolio of claim counts, put the Akaike information criterion near 141,700 for the Poisson model and near 129,800 for the negative binomial on the same data, with zero-inflated variants improving on both. The gap of roughly 12,000 AIC points is not a subtle statistical preference; it's a model that misses the shape of the data.

Road safety took the same route. Negative binomial regression, often called Poisson-gamma in that literature, is the standard error structure for the safety performance functions used in before-and-after evaluations of engineering treatments, and for the empirical Bayes method that corrects for regression to the mean when you audit an accident blackspot. Ezra Hauer's 2001 paper showed that the overdispersion parameter should scale with the length of road section under study. Assume one shared dispersion parameter across a network and the likelihood gets dominated by very short segments, while the empirical Bayes safety estimates shift materially. A bookkeeping choice about one parameter changes which junctions get rebuilt.

Scientific equations and abstract glowing forms against a dark background

Photo by Brecht Corbeel on Unsplash

What This Means for Your Lab

Count data arrive whether or not you ordered them: colonies per plate, reads per gene, events per patient per year, defects per batch. The negative binomial is the sensible default for that shape of data, and the misuse is almost always the same. Someone fits it, reports a coefficient, and treats the dispersion parameter as plumbing rather than as a finding.

  • Report the dispersion estimate alongside the mean, with an interval. A k of 0.4 and a k of 4 describe very different populations, and readers cannot judge your model without the number.
  • Check the variance-to-mean ratio before choosing anything. With a few thousand observations a ratio of 1.4 to 1.6 is a real signal; with forty observations the same ratio may be noise, and the Poisson model will do less damage than an unstable dispersion estimate.
  • If heterogeneity is the scientific question, model it directly with covariates on the variance or a random effect on the rate, instead of reading it off a marginal fit after the fact.
  • When the sample is small, say in the methods section that your dispersion is borrowed from a trend across features. Shrinkage is a legitimate trade, but it is a trade, and reviewers are entitled to know it happened.

The concrete next step takes about ten minutes. Take the last count dataset you analysed, fit the Poisson and the negative binomial to the same table, and print both dispersion estimates next to the log-likelihoods. If the second model moves your conclusion, you have found the paragraph your discussion section was missing. If it doesn't, you now have a sentence of justification you can defend.

  • negative binomial distribution
  • overdispersion
  • probability theory
  • count data
  • statistics history
Portrait de Dr. Nathan Harlow
A propos de l'auteur

Dr. Nathan HarlowVoir auteur

Academic Jobs In House Author

Dr. Nathan Harlow is a Contributing Writer at AcademicJobs, with expertise in STEM education and research methodologies. Holding a PhD in Physics from Stanford University, he has 18 years of experience in academic resea…

Discussions

Sort par :

Soyez le premier à commenter cet article !

vous

Vous serez invité à vous connecter avant de publier votre commentaire.

Browse par faculté

Browse par sujet

Frequently Asked Questions

📊What is the negative binomial distribution in simple terms?

It's a probability distribution for counts of events where the variance is allowed to be larger than the mean. If you know the average number of events per unit, a Poisson model fixes the spread too. The negative binomial adds a second parameter, usually written k or theta, that widens the distribution without shifting its centre. It's the standard choice for accident counts, insurance claims, sequencing reads and secondary infections, all of which are lumpier than Poisson allows.

🎲Who discovered the negative binomial distribution?

The algebra goes back to Blaise Pascal, who mentioned it in work published posthumously in 1679, and to Pierre Rémond de Montmort, whose 1713 Essay d'analyse sur les jeux de hasard treated the number of trials needed to reach a fixed number of successes. The modern statistical use arrived in 1920, when Major Greenwood and Udny Yule fitted it to accident counts from a munitions factory and explained the extra spread by variation in individual risk.

🔢Why is it called the negative binomial distribution?

The name is algebraic, not descriptive. The probability mass function can be written as a binomial coefficient with a negative upper index, so the distribution falls out of expanding a binomial expression raised to a negative power. Ronald Fisher's 1941 paper in the Annals of Eugenics uses the negative-index form directly. Alternative names, including Pascal distribution and Pólya distribution, honour the people who derived it from waiting times and from urn schemes respectively.

📈How is the negative binomial different from the Poisson distribution?

Poisson assumes the variance equals the mean. The negative binomial assumes variance equals the mean plus the squared mean divided by the dispersion parameter, so the two agree only when that parameter is very large. In practice the Poisson is the special case, not the rival model. If the variance-to-mean ratio in your data sits around 1.3 or higher across thousands of observations, the Poisson will understate uncertainty and overstate significance.

📉What does the dispersion parameter k or theta mean?

It measures how much the underlying rates vary between units. Small values mean extreme heterogeneity: a handful of units account for most of the events. Large values mean the population behaves almost uniformly and the model converges on Poisson. In outbreak analysis a k around 0.1 to 0.2 is the signature of superspreading; in RNA-seq work the equivalent parameter is conventionally called theta and is estimated per gene, then shrunk toward a fitted trend.

🦠Why is the negative binomial used to model superspreading?

Because individual infectiousness varies, and the simplest way to represent that variation is a gamma distribution of individual reproductive numbers, which mixes with Poisson transmission to give a negative binomial offspring distribution. Lloyd-Smith and colleagues set out this model in Nature in 2005 and found it fitted SARS, measles and smallpox data better than homogeneous alternatives. For the Singapore SARS outbreak the estimated dispersion was around 0.16, meaning most cases infected almost nobody while a small minority infected many.

🧬How is the negative binomial used in RNA-seq analysis?

Read counts per gene are overdispersed relative to Poisson because of biological variability between samples. edgeR, DESeq2 and related tools fit a negative binomial generalised linear model, estimate the mean-variance relationship across all genes, and shrink each gene's dispersion estimate toward that trend. The shrinkage is what makes experiments with three replicates per group analysable, though it also means the dispersion you report is partly borrowed information.

🧪Is a good negative binomial fit evidence of a mechanism?

No. A negative binomial distribution can arise from genuine contagion, from unobserved heterogeneity between units, or from a mixture of both, and the marginal distribution cannot tell them apart. Statisticians were pointing this out in the 1940s. A 2026 study in PLOS Computational Biology showed the same ambiguity in single-cell genomics: good fits occur across a wide range of gene-expression kinetics without any requirement for transcriptional bursting.

⚙️What is negative binomial regression?

It's a generalised linear model for counts with a log link and negative binomial errors, used when the outcome is overdispersed. Coefficients are usually reported as incidence rate ratios. It's the standard tool in epidemiology for rates per person-year, in insurance for claim frequency, and in road safety for crash counts per segment, where it is often called Poisson-gamma regression.

🧭How much data do you need to estimate the dispersion parameter?

More than most people assume. The mean is well estimated even in modest samples, but the dispersion parameter needs enough units and enough repeat observations to separate real heterogeneity from sampling noise. Rules of thumb vary by field; in genomics the workaround is to pool information across thousands of features, and in outbreak analysis the workaround is to report intervals rather than point estimates because k is hard to pin down from finite data.

💰Where else is the negative binomial used outside epidemiology and biology?

Insurance claim counts, road crash frequency, patent and publication counts, customer purchase frequencies, and quality control defect counts all show up overdispersed. In a 2024 worked example published in The Actuary, a negative binomial model improved the Akaike information criterion by roughly 12,000 points over Poisson on the same claim data. Highway safety agencies use the same error structure to evaluate whether an engineering change actually reduced crashes.

📚Are the Pascal, Pólya and negative binomial distributions the same thing?

They describe the same family arrived at by different routes. The Pascal version counts trials or failures before a fixed number of successes in independent Bernoulli trials. The Pólya-Eggenberger version comes from an urn scheme in which drawing a coloured ball makes that colour more likely next time, which produces genuine contagion. The Poisson-gamma mixture, used by Greenwood and Yule in 1920, produces the same formula from unobserved heterogeneity instead. Same algebra, three stories.