Support vector machine: Definition & Examples

A support vector machine (SVM) is a supervised learning method that fits a decision boundary by balancing a wide separation between classes against penalties for observations that violate that separation. For a quant classifying future return directions or credit events, the useful distinction is between getting a historical label right and placing it comfortably on the right side of the boundary. Neither establishes that the rule will work on future data.

The classification version is the focus here. Support vector regression is a related method for numerical targets, with a different loss.

A correct prediction can still incur loss

For two classes, encode the observed label as $y\in\{-1,+1\}$ . A linear SVM uses the score

f(x)=w^\top x+b.

Here $x$ is the feature vector, $w$ contains fitted weights, and $b$ is an intercept. Predict $+1$ for a positive score and $-1$ for a negative score; a score of zero requires a stated tie rule. The decision boundary is $f(x)=0$ .

The functional margin of an observation is

m=yf(x)

. A positive margin means the score has the correct sign. The standard hinge loss asks for more than this:

\ell(y,f(x))=\max\{0,1-yf(x)\}.

Consider the stipulated, dimensionless score

f(x)=x

and two observations whose actual label is

+1

. This score is given to illustrate the loss; it has not been fitted to financial data.

Feature and score $x=f(x)$	Functional margin $yf(x)$	Predicted class	Hinge loss
0.5	0.5	$+1$	$\max(0,1-0.5)=0.5$
2	2	$+1$	$\max(0,1-2)=0$

Both classifications are correct, but only the second meets the unit-margin requirement. A loss of $0.5$ is not a probability of error, and a score of $2$ is not a probability at all.

Hinge loss for the fixed dimensionless score f(x)=x and label +1. Observation A at x=0.5 has margin 0.5 and loss 0.5. Observation B at x=2 has margin 2 and loss zero. The shaded interval between scores zero and one is correctly classified but still penalised.

Original fixed-score illustration, without model fitting or market observations. Filled dots mark the two examples. The dashed lines distinguish the decision boundary from the unit-margin threshold.

The unit threshold is a score normalisation. Multiplying both

w

and

b

by a positive constant preserves the boundary and class predictions while changing the functional margins. For

w\ne0

, the label-signed geometric distance is

yf(x)/\lVert w\rVert

, which is unchanged by that rescaling. The Functional and Geometric Margins lesson develops this geometry and the hard-margin optimisation; its full text requires membership, with a public preview.

Hard separation or paid violations

A hard-margin classifier minimises

\tfrac12\lVert w\rVert^2

subject to

y_i f(x_i)\ge1

for every training observation. For separable data, this chooses the widest separating margin in the chosen feature geometry. If the classes cannot be separated there, the constraints are infeasible. Even separable data can contain an outlier that makes insisting on perfect separation unhelpful.

A soft-margin classifier allows violations. One standard linear formulation is

\min_{w,b}\;\frac12\lVert w\rVert^2 +C\sum_{i=1}^{n}\max\{0,1-y_i(w^\top x_i+b)\}, \qquad C>0.

The intercept is unpenalised in this formulation. The norm term supplies regularisation; the sum measures margin violations. At fixed data and scaling, increasing $C$ puts more weight on reducing those violations relative to shrinking the weights. It does not guarantee greater test accuracy.

Compare objectives before comparing parameter values. If the loss is instead averaged, the equivalent form is

\frac{\lambda}{2}\lVert w\rVert^2+ \frac1n\sum_{i=1}^{n}\ell(y_i,f(x_i)), \qquad \lambda=\frac{1}{Cn}.

Multiplying this expression by

Cn

recovers the preceding sum-loss objective, so they have the same minimisers under these conventions. A numerical regularisation setting is not portable without checking loss normalisation, penalty and sample weights. The Soft-Margin SVMs lesson continues with slack variables and the dual solver; membership is required for the full lesson, with a public preview.

Why “support vectors”, and where kernels enter

In the usual dual representation, support vectors are training observations with nonzero dual coefficients. They contribute directly to the fitted score. In a soft-margin model they can include observations inside the margin or on the wrong side of the decision boundary, not just points touching a clean separating gap. The two observations in the illustration cannot be declared support vectors: no training problem was solved.

A kernel supplies inner products in a transformed feature space,

K(x,z)=\langle\phi(x),\phi(z)\rangle

, without necessarily constructing the transformation explicitly. A linear boundary in that space can be nonlinear in the original inputs. A kernel changes the model's geometry; it does not remove overfitting or turn its score into a calibrated probability.

For financial forecasting, fit feature scaling on training data alone, use features available at each decision time, and respect when labels become observable. Select the kernel and regularisation through an appropriate time-aware validation design. Class imbalance, changing market regimes and transaction costs still matter after classification. Turning an SVM score into a trade size requires a separate, justified rule; interpreting it as an event probability requires a separately fitted and validated calibration step.

Sources

Andrew Ng, CS229: Support Vector Machines, §§2–4, 6–8: signed scores, margins, support vectors, kernels and the non-separable case. The local CS229 handout collection contains the same sections.
Afshine Amidi and Shervine Amidi, CS229 Supervised Learning Cheatsheet — Support Vector Machines: hinge-loss and kernel definitions.

A correct prediction can still incur loss

Hard separation or paid violations

Why “support vectors”, and where kernels enter

Sources

Related terms

Overfitting

Supervised learning