Logistic regression fitting and SVM loss comparison: Definition & Examples

Logistic regression models the probability of a binary outcome by applying a sigmoid to a linear score. Fitting it means estimating the score's coefficients from labelled observations, usually by minimising regularised Bernoulli negative log-likelihood. A linear support vector machine can use the same form of score, but its usual hinge-loss objective rewards separation by a margin. That distinction matters when a quant needs a probability of a positive outcome rather than just a trading signal's class or rank.

A probability model before a decision rule

For features $x\in\mathbb{R}^d$ and label $Y\in\{0,1\}$ , write

f(x)=b+w^\top x,\qquad p(x)=\mathbb{P}(Y=1\mid x)=\sigma(f(x))=\frac{1}{1+e^{-f(x)}}.

These are model probabilities. The model assumes linear log-odds:

\log\frac{p(x)}{1-p(x)}=b+w^\top x.

Holding other features fixed, increasing feature

j

by one unit multiplies the model odds by

e^{w_j}

. It does not add

w_j

probability points. A nonlinear, preselected feature such as a squared return can enter

x

; “linear” then describes the score in those features. Stanford's logistic-regression notes develop the conditional model and its likelihood.

Consider a fictional specified model, not coefficients estimated from market data:

p(x)=\sigma(-1+0.8x).

Here $x$ is a signal standardised using training-period statistics. At $x=0$ , $p=0.268941$ ; at $x=2$ , $p=0.645656$ . A cutoff of $0.5$ changes the predicted class at $x=1.25$ , where the score is zero. Choosing another cutoff changes the action boundary while leaving the fitted probability curve intact.

A fictional sigmoid probability curve gives p equal to 0.268941 at feature x equal to 0 and 0.645656 at x equal to 2. The probability 0.5 boundary occurs at x equal to 1.25.

Fictional specified coefficients. The curve models probability; the marked 0.5 cutoff is a separate decision convention.

What fitting actually minimises

For observations

(x_i,y_i)

, assume conditionally independent Bernoulli outcomes under the model. With

p_i=p(x_i)

, the log-likelihood is

\ell(b,w)=\sum_{i=1}^n\left[y_i\log p_i+(1-y_i)\log(1-p_i)\right].

Maximum likelihood maximises this sum. Equivalently, one commonly minimises

J_{\mathrm{log}}(b,w) =\frac{1}{n}\sum_{i=1}^n \left[\log(1+e^{f_i})-y_i f_i\right] +\frac{\lambda}{2}\lVert w\rVert^2, \qquad f_i=b+w^\top x_i,\quad\lambda\ge0.

The bracketed quantity is binary log loss, written without first computing logarithms of probabilities near zero. Software should evaluate the softplus term $\log(1+e^{f_i})$ with a numerically stable implementation.

The gradient for the slope is

\nabla_w J_{\mathrm{log}} =\frac{1}{n}\sum_i(p_i-y_i)x_i+\lambda w,

and the intercept gradient is the mean of

p_i-y_i

. There is generally no closed-form coefficient formula. Gradient methods, Newton methods and quasi-Newton solvers iteratively reduce the objective. Each observation's second derivative with respect to its score is

p_i(1-p_i)>0

: the objective is convex in

(b,w)

for fixed features and a nonnegative quadratic penalty. A finite stationary minimum is global; convexity alone does not promise a finite solution, unique coefficients or automatic numerical convergence. The official logistic-regression documentation describes penalties and solver choices; loss averaging and penalty conventions must match when comparing implementations.

Separation is the important exception. If one hyperplane strictly separates every training label, scaling that separator drives unregularised log loss towards zero while coefficients grow without bound. There is no finite unregularised maximum-likelihood estimate. Quasi-separation can cause related problems. An L2 penalty constrains this growth; in the displayed convention it penalises slopes and leaves the intercept unpenalised. Both classes must be represented: a one-class training sample can still send that unpenalised intercept to infinity. Check convergence and coefficient behaviour, as the statsmodels pitfalls documentation emphasises.

Change the loss, change the fitting incentives

To compare classifiers, recode labels as

s_i=2y_i-1\in\{-1,+1\}

and define the signed functional margin

m_i=s_i f_i

. Positive means correctly classified under the zero-score boundary; negative means incorrectly classified. This margin is a score, not a geometric distance: for nonzero

w

, signed distance to the hyperplane is

m_i/\lVert w\rVert

The two pointwise losses are

L_{\mathrm{log}}(m)=\log(1+e^{-m}),\qquad L_{\mathrm{hinge}}(m)=\max(0,1-m).

For a positive label with $f=1$ , the logistic loss is $0.313262$ , while hinge loss is zero. Logistic loss continues falling as a correct score grows. Hinge loss becomes flat once $m\ge1$ : pushing a point farther past the margin supplies no additional reduction in that point's loss.

Logistic and hinge loss plotted against signed functional margin from minus 2 to 3. At margin 1 logistic loss is 0.313262 and hinge loss is zero; hinge stays zero beyond margin 1.

Original comparison of pointwise losses at common signed scores. The curves do not compare fitted forecasting accuracy or economic performance.

A soft-margin linear SVM minimises, in one mean-loss convention,

J_{\mathrm{svm}}(b,w) =\frac{1}{n}\sum_i\max(0,1-s_i f_i) +\frac{\lambda}{2}\lVert w\rVert^2.

The quadratic term controls the trade-off between coefficient size and margin violations. The familiar objective

\frac12\lVert w\rVert^2+C\sum_i L_{\mathrm{hinge}}(m_i)

is equivalent here when

\lambda=1/(nC)

for

C>0

. A shared numerical penalty setting is not a fair tuning procedure across loss functions. Also check the actual estimator: squared hinge is a different loss. See the official SVM formulation and the membership lesson Soft-Margin SVMs and the SMO Algorithm for the slack-variable construction; its public page is a preview.

Probability, calibration and a trading label

Applying

\sigma(f)

to an arbitrary SVM score produces a number between zero and one, not an established probability. Platt scaling instead learns a sigmoid's parameters from held-out scores and labels. Logistic regression already has a probability model, but that does not ensure empirical calibration under misspecification, strong shrinkage or a changed market regime. On new observations, predictions near

0.7

should correspond to positive outcomes about 70% of the time if they are calibrated. The official calibration guide explains this distinction. The membership lesson Probability calibration, with a public preview, develops the checks before sizing.

For a concrete research design, suppose features are frozen after exchange close on trading day

t

. Define

Y_t=1

when the long return from the next opening price to the following day's opening price exceeds a stipulated 5 basis point round-trip cost, and zero otherwise. This is a fictional label definition; the cost is fixed before looking at outcomes, and the label matures only at the second opening price. The example probability model above could target that event, but no evidence here says it predicts it successfully.

Fit scaling, imputation and feature selection only inside each training window, then apply the learnt transformations to later observations. Tune penalties, calibration and cutoffs using time-respecting validation; training labels must have matured before the validation prediction time, and overlapping outcome intervals need appropriate separation. Keep the final test period outside all these choices. This applies the train-only preprocessing rule to a financial information timeline.

Even a calibrated probability that the cost-adjusted return is positive does not give its expected size or the loss when it is negative. A $0.5$ classification cutoff therefore does not by itself identify a profitable trade, and the stipulated cost is already in this label rather than an additional deduction from its probability.

Practice: a class is not an action

A different application has a credible positive-outcome probability of $0.58$ . A false positive costs 3 loss units, a false negative costs 2, and correct decisions cost zero. Which action minimises expected loss?

Answer. A positive action costs

3(1-0.58)=1.26

in expectation. A negative action costs

2(0.58)=1.16

, so choose negative. The positive-action cutoff is

3/(3+2)=0.60

, with indifference at equality. A probability above

0.5

identifies the more likely class, but these asymmetric error costs justify a different action. These are stipulated decision losses, not a model of trading payoffs.

Sources

Stanford CS229, Spring 2026 notes, §2.1: sigmoid model and Bernoulli likelihood.
scikit-learn: Logistic regression: regularised fitting and solvers.
statsmodels: Pitfalls: nonexistent estimates and incomplete convergence.
scikit-learn: Support vector machines: margin objective and score/probability distinction.
scikit-learn: Probability calibration: reliability and held-out calibration.
scikit-learn: Common pitfalls: fitting preprocessing within the training split.

A probability model before a decision rule

What fitting actually minimises

Change the loss, change the fitting incentives

Probability, calibration and a trading label

Practice: a class is not an action

Sources

Related terms

Bernoulli distribution

Maximum likelihood estimation

Support vector machine