Generative and discriminative models: Definition & Examples

Generative models describe a distribution of the data; discriminative models learn a prediction from observed inputs directly. For classification, the distinction is usually between modelling the joint distribution of features and labels, and modelling the label given the features. A quant predicting issuer default can use either approach. A quant trying to simulate new issuer features needs something more than a default-probability classifier.

What each model takes responsibility for

Write $X$ for the features and $Y$ for the class label. A generative classifier models

p(x,y)=p(x\mid y)p(y).

It describes how common each class is and how features vary within that class. It then uses Bayes’ theorem to obtain the conditional probability needed for prediction:

p(y\mid x)=\frac{p(x\mid y)p(y)}{\sum_c p(x\mid c)p(c)}.

The sum runs over the possible classes, and the denominator must be positive. Here $p$ denotes probability mass for discrete features or density for continuous features. These are model distributions, not guarantees that fitted probabilities match reality.

A probabilistic discriminative model fits

p(y\mid x)

directly. Other discriminative algorithms learn a score or decision boundary without producing probabilities. Neither approach needs to specify the distribution of

X

. This classical distinction appears in Ng and Jordan’s comparison of naive Bayes and logistic regression.

Approach	Typical example	Main modelling commitment
Generative classification	Naive Bayes	Class priors and feature distributions conditional on each class; naive Bayes additionally assumes conditional independence of features within a class
Generative classification	Linear discriminant analysis	Gaussian class-conditional features with a shared covariance matrix
Discriminative classification	Logistic regression	A conditional class probability with a linear predictor on the log-odds scale

The output shape alone does not identify the training approach. Shared-covariance Gaussian discrimination also implies a logistic conditional probability. Its parameters are fitted through a joint model, however, and generally differ from parameters obtained by directly fitting logistic regression. Ng’s CS229 notes, §1.3, explain why the Gaussian assumptions are stronger.

One default prediction, two routes

Consider a fictional model of an issuer over a specified one-year horizon. Let

Y=1

mean default during that year and

X=1

mean a binary alert observed at its start. Assume:

The default prior is $\mathbb{P}(Y=1)=0.20$ .
The alert occurs in 80% of default cases: $\mathbb{P}(X=1\mid Y=1)=0.80$ .
It occurs in 10% of non-default cases: $\mathbb{P}(X=1\mid Y=0)=0.10$ .

These are illustrative assumptions, not market estimates. The generative model assigns joint probabilities of $0.20(0.80)=0.16$ to default with an alert and $0.80(0.10)=0.08$ to non-default with an alert. Hence

\mathbb{P}(Y=1\mid X=1)=\frac{0.16}{0.16+0.08}=\frac{2}{3}.

The 80% alert rate is not the answer: it conditions in the opposite direction. In 1,000 cases with exactly the assumed proportions, 160 of the 240 alerted cases default. Without an alert, 40 of 760 cases default, giving $1/19$ .

Fictional 1000 issuer cases: 160 default with an alert, 40 default without, 80 non-default with an alert and 720 neither. Conditional default probabilities are two thirds with an alert and one nineteenth without.

Original fictional model. Left: all joint case counts and probabilities. Right: conditional default probabilities derived from those counts; a discriminative model could estimate them directly.

A discriminative model could learn those two conditional probabilities directly from labelled cases. Their agreement with the generative calculation would not establish which training approach produced them. Nor would those conditional outputs alone specify how frequently alerts occur. Recovering the joint distribution additionally requires a marginal model for $X$ .

Classification is not data synthesis

“Generative” also covers models that learn a distribution of observations without a class label. A model of issuer features could sample synthetic issuers; a generative classifier could sample a class first and then features conditional on that class. The toy model above can generate binary alert/default pairs, not realistic balance sheets or market paths.

Explicit likelihood evaluation is not required for every generative model. A generative adversarial network learns a sampling mechanism through competition between a generator and a discriminator. As the original GAN paper describes, its discriminator distinguishes training observations from generated observations. Having that discriminative component does not turn the whole generator into a direct default classifier.

The practical test is the requested output: a label or conditional probability, or a new observation? A model can serve more than one role, but producing plausible synthetic data and predicting labels well are separate achievements.

Which assumptions can the data support?

Generative classifiers can benefit from strong distributional structure when labelled samples are limited. That structure can also be wrong. Ng and Jordan demonstrate different sample-size regimes for naive Bayes and its discriminative counterpart under their stated setup; their result is not a universal ranking of every generative and discriminative model.

For credit or financial-text classification, compare future-period prediction errors and probability calibration, with features available at the decision time. For synthesis, examine whether generated observations preserve the dependencies and constraints required by the intended use. A valid probability distribution need not be an adequate financial model.

To work through the models, Generative Classifiers: Gaussian Discriminant Analysis and Naive Bayes derives both class-conditional constructions. Logistic Regression and Maximum Likelihood develops the direct conditional objective. Both have public previews; the full lessons require membership.

Sources

Andrew Y. Ng and Michael I. Jordan (2001), On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes, Abstract and §1: joint versus conditional learning and the scope of the comparison.
Andrew Ng, CS229 Lecture Notes: Generative Learning Algorithms, opening discussion, §§1.2–1.3 and §2: Bayes classification, shared-covariance Gaussian models and naive Bayes.
Ian Goodfellow and co-authors (2014), Generative Adversarial Nets, Abstract: a generator captures a data distribution while a discriminator distinguishes real from generated samples.

What each model takes responsibility for

One default prediction, two routes

Classification is not data synthesis

Which assumptions can the data support?

Sources

Related terms

Bayes' theorem

Classification

Conditional probability

Linear discriminant analysis

Logistic regression fitting and SVM loss comparison

Naive Bayes classifier