Firth correction and complete separation: Definition & Examples

Firth correction is a likelihood-based method that reduces the leading asymptotic bias of logistic-regression coefficient estimates. It also gives finite estimates when complete separation makes ordinary maximum likelihood diverge, provided the design matrix has full column rank. In a sparse credit dataset, a feature that perfectly divides repayments from non-repayments can otherwise produce an enormous fitted coefficient that reflects the sample's geometry rather than a reliably measured effect.

Why the ordinary fit runs away

In logistic regression, independent binary outcomes have model probabilities

p_i=\frac{1}{1+e^{-x_i^\top\beta}}.

The feature vector $x_i$ includes a constant if the model has an intercept. Complete separation means there is a direction in coefficient space giving a strictly positive score to every observed success and a strictly negative score to every observed failure. Increasing the magnitude along that direction keeps improving the ordinary likelihood: at least one coefficient has no finite maximum-likelihood estimate. Quasi-complete separation has at least one observation on the separating boundary and at least one strict correctly signed inequality, with no observations on the wrong side; it can cause the same divergence.

A large returned coefficient or a convergence message is therefore insufficient evidence that a finite optimum exists. Kosmidis and Firth (2021), §1 describe this failure and its implications for Wald inference.

The adjustment depends on information

Write the ordinary log-likelihood as $\ell(\beta)$ . For binary logistic regression, the expected Fisher information is

I(\beta)=X^\top W(\beta)X, \qquad W(\beta)=\operatorname{diag}\{p_i(1-p_i)\}.

Firth fitting maximises

\ell_F(\beta)=\ell(\beta)+\frac12\log\det I(\beta).

The added term is the log of the Jeffreys-prior density in the coefficient parameterisation. Firth's original result removes the first-order asymptotic bias term for the canonical coefficients; it does not promise exact unbiasedness in a finite sample. See Firth (1993), §§3.1 and 3.3.

For full-column-rank

X

, the information determinant tends to zero as coefficients escape to infinity, making the penalty tend to minus infinity. The corrected maximum is finite even under separation: this is Kosmidis and Firth's finiteness result, §2.1. With redundant columns, the determinant is already zero; this formula cannot identify their separate coefficients. Resolve that rank deficiency first.

Five repayments, no observed non-repayments

Consider a fictional toy cohort of five loans, all repaid within a stipulated horizon. Code repayment as one. Assume five independent Bernoulli trials with a common repayment probability

p

, and fit only an intercept

\eta=\log\{p/(1-p)\}

. This simplification ignores loan heterogeneity and dependence.

The ordinary log-likelihood is $\ell(\eta)=5\log p$ . Its derivative is $5(1-p)>0$ for every finite $\eta$ . The probability estimate approaches the boundary $p=1$ as $\eta\to+\infty$ ; there is no finite ordinary intercept estimate.

Here $I(\eta)=5p(1-p)$ , so the corrected objective becomes

\ell_F(\eta) =5\log p+\frac12\log\{5p(1-p)\} =\frac{11}{2}\log p+\frac12\log(1-p)+\frac12\log 5.

Differentiation with respect to $\eta$ gives

\frac{d\ell_F}{d\eta}=\frac{11}{2}-6p=0, \qquad \widehat p_F=\frac{11}{12}\approx0.916667, \qquad \widehat\eta_F=\log 11\approx2.397895.

In a fictional five-repayment cohort, ordinary log likelihood rises toward zero as the intercept tends to infinity. The Firth objective peaks at intercept 2.397895, with fitted repayment probability 11/12, or 91.67 percent.

Original likelihood curves for the fictional intercept-only cohort. The ordinary objective has a supremum at an infinite intercept; the Firth objective has a finite maximum. Vertical values include the displayed penalty constant.

The result looks like adding half a success and half a failure. That shortcut holds for this single-probability model. It is not a rule to add half an observation to every cell in a regression with predictors: the general adjustment depends on the design and fitted leverage, as Firth's §3.3 explains.

Finite does not mean automatically trustworthy

Parameterisation matters. In this example, transforming the Jeffreys prior to a density with respect to

p

gives the Beta

(1/2,1/2)

prior. After five successes, the posterior density in

p

is proportional to

p^{9/2}(1-p)^{-1/2}

and diverges at one: it has no interior mode. The Firth probability

11/12

comes from maximising the penalised objective in

\eta

and then transforming the estimate. Modes depend on the density's coordinate system, even though Jeffreys' prior measure is invariant.

Bias reduction has a target. Removing the leading bias term for coefficients does not make the transformed probability exactly unbiased or empirically calibrated. A finite repayment estimate from five observations still carries substantial uncertainty. For the separate predictive task, the membership lesson Probability calibration, available through a public preview, explains how held-out outcome frequencies test the probability interpretation.

Estimation is not causal identification. Firth correction changes the estimation objective. It does not establish that a borrower's feature causes repayment, resolve omitted confounding, or make a tiny cohort representative of a future lending population.

Sources

Firth (1993), Bias reduction of maximum likelihood estimates, §§3.1 and 3.3: canonical-parameter Jeffreys penalty, binomial logit example and design-dependent adjustments.
Kosmidis and Firth (2021), Jeffreys-prior penalty, finiteness and shrinkage in binomial-response generalized linear models, §§1–2: separation, full-rank finiteness theorem and inference limitations.

Why the ordinary fit runs away

The adjustment depends on information

Five repayments, no observed non-repayments

Finite does not mean automatically trustworthy

Sources

Related terms

Logistic regression fitting and SVM loss comparison