Skip to content

Negative binomial regression

A count regression with a log-linked conditional mean and explicit extra-Poisson variation: NB2 conventions, coefficient interpretation and diagnostic limits.

Negative binomial regression models a non-negative integer response using a negative binomial distribution whose conditional mean depends on explanatory variables. A quant might use it for daily operational-incident counts when a Poisson regression gives a plausible average but too little variation around that average. The regression describes how expected frequency changes; it does not model the monetary size of each incident.

Specify the mean and the variance

A common version is NB2 with a log link. Write YiY_i for the count, xi\mathbf{x}_i for observed predictors and ei>0e_i>0 for a known exposure, such as operating hours:
log⁡μi=xi⊤β+log⁡ei,E[Yi∣xi,ei]=μi,Var(Yi∣xi,ei)=μi+αμi2,α>0.\begin{aligned} \log\mu_i &= \mathbf{x}_i^\top\boldsymbol{\beta}+\log e_i,\\ \mathbb{E}[Y_i\mid\mathbf{x}_i,e_i]&=\mu_i,\\ \text{Var}(Y_i\mid\mathbf{x}_i,e_i)&=\mu_i+\alpha\mu_i^2,\qquad \alpha>0. \end{aligned}
The exposure term is an offset: its coefficient is fixed at one. Thus μi=eiexp⁡(xi⊤β)\mu_i=e_i\exp(\mathbf{x}_i^\top\boldsymbol{\beta}) is an expected count, while exp⁡(xi⊤β)\exp(\mathbf{x}_i^\top\boldsymbol{\beta}) is the expected event rate per exposure unit. Including this offset assumes counts scale proportionally with exposure at fixed predictors. Equal-exposure observations can use ei=1e_i=1.
The extra term αμi2\alpha\mu_i^2 allows conditional variance above the Poisson value μi\mu_i. This is overdispersion relative to Poisson, after accounting for the specified predictors and exposure. Comparing the raw sample variance with the raw sample mean mixes this question with differences in fitted means across observations.
The name alone is insufficient: NB1, under another common convention, uses Var(Yi∣xi,ei)=μi+αμi\text{Var}(Y_i\mid\mathbf{x}_i,e_i)=\mu_i+\alpha\mu_i. Its extra variation grows linearly rather than quadratically with the mean. With μ=4\mu=4 and a numerical dispersion value α=0.5\alpha=0.5, NB1 gives variance 66, whereas NB2 gives 1212. A dispersion estimate cannot be transferred between these models just because both call it “alpha.” statsmodels documents both conventions.

A forecast with mean four

Consider a specified synthetic NB2 model for one operating day, at fixed predictors and exposure:
μ=4,α=0.5,Var(Y∣x,e)=4+0.5(42)=12.\mu=4,\qquad \alpha=0.5,\qquad \text{Var}(Y\mid\mathbf{x},e)=4+0.5(4^2)=12.

To compute probabilities, convert to the size–probability parameterisation:

k=1α=2,p=kk+μ=13.k=\frac{1}{\alpha}=2,\qquad p=\frac{k}{k+\mu}=\frac13.

Here the probability mass is

P(Y=y∣x,e)=Γ(y+k)Γ(k)y!pk(1−p)y,y=0,1,2,….\mathbb{P}(Y=y\mid\mathbf{x},e) =\frac{\Gamma(y+k)}{\Gamma(k)y!}p^k(1-p)^y, \qquad y=0,1,2,\ldots.
This is the failures-before-kk-successes convention when kk is an integer, as in R's negative-binomial documentation. It is not the total-trials convention, whose support starts at kk. In regression, kk may be any positive real number: “size” is a distributional parameter, not necessarily a literal success target, and operational incidents need no Bernoulli stopping interpretation.

At zero, the mass simplifies to

P(Y=0∣x,e)=pk=19≈11.11%.\mathbb{P}(Y=0\mid\mathbf{x},e)=p^k=\frac19\approx11.11\%.
A Poisson distribution with the same mean has variance 44 and zero probability e−4≈1.83%e^{-4}\approx1.83\%. Matching the mean therefore does not match either the chance of a quiet day or the upper tail.
Specified synthetic counts share mean 4: NB2 with alpha 0.5 has variance 12 and zero probability 11.11%, while Poisson has variance 4 and zero probability 1.83%. A log-link coefficient log 2 doubles the NB2 mean to 8 and raises variance to 40 at fixed dispersion.
Specified synthetic counts share mean 4: NB2 with alpha 0.5 has variance 12 and zero probability 11.11%, while Poisson has variance 4 and zero probability 1.83%. A log-link coefficient log 2 doubles the NB2 mean to 8 and raises variance to 40 at fixed dispersion.
Original comparison of specified conditional count laws, not fitted data. Filled bars show NB2 and outlined bars show Poisson; the lower panel holds exposure and dispersion fixed while changing one covariate.

Read coefficients as ratios

With the log link, increasing xjx_j by one unit multiplies the conditional mean by exp⁡(βj)\exp(\beta_j), holding all other predictors and exposure fixed. A coefficient βj=log⁡2\beta_j=\log 2 changes a mean of 44 to 88. It does not add two events. At fixed α=0.5\alpha=0.5, the new NB2 variance is 8+0.5(82)=408+0.5(8^2)=40; dispersion affects the predictive spread even though it does not appear in the mean equation.

This interpretation concerns the specified regression relationship. A coefficient is not automatically a causal effect. Interactions or nonlinear predictor terms also mean that the effect of changing an original input must be calculated from the full mean equation.

For ordinary independent observations, fitting commonly maximises the negative-binomial likelihood over regression coefficients and dispersion. Check what the chosen fitting routine actually estimates: MASS glm.nb estimates its additional size parameter, while the statsmodels NB2 family documentation describes an ancillary alpha treated as given. Supplying a family with a dispersion value is not the same operation as estimating that dispersion.

Extra variance is a modelling choice

One NB2 construction is a Poisson count conditional on an unobserved multiplier UiU_i, with intensity μiUi\mu_iU_i, where UiU_i is gamma distributed with shape 1/α1/\alpha and scale α\alpha. Then EUi]=1\mathbb{E}U_i]=1 and Var(Ui)=α\text{Var}(U_i)=\alpha, producing the NB2 variance by the [law of total variance. This is a useful heterogeneity story, not evidence that a real dataset's hidden intensity follows a gamma law. The Loss Distribution Approach, §5, derives the mixture and connects frequency to aggregate losses. Its public preview is open; the full lesson requires membership.

A fitted dispersion parameter does not repair every count-data problem:

  • A wrong conditional mean: omitted predictors, incorrect exposure or a wrong functional form can appear as overdispersion. Recheck the mean specification.
  • Dependence: repeated counts from the same desk or serially clustered incident days require a dependence model or suitable inference. Merely replacing the one-observation distribution leaves cross-observation dependence unspecified.
  • Excess zeros: NB2 already predicts more zeros than Poisson in this example. If zeros still exceed a credible fitted count model, a structural-zero or hurdle mechanism may be relevant; many zeros alone do not establish either mechanism.
Brooks et al. discuss separate model components for dispersion, correlated observations and structural zeros. Check fitted probabilities, residual patterns and predictive performance against the data-generating setting before interpreting a larger dispersion as a complete solution.

Sources

Last updated
  • count data
  • regression
  • overdispersion
  • negative binomial
  • log link
Keep connecting the ideas
← Back to the glossaryBack to top ↑