Soft actor-critic: Definition & Examples

Soft actor-critic (SAC) is an off-policy, model-free reinforcement-learning algorithm that trains a stochastic policy to favour both reward and policy entropy. An actor chooses actions; critics estimate their soft values. For a quant testing continuous execution sizes or portfolio adjustments in a simulator, this gives a way to learn a distribution of actions while reusing past transitions. It does not establish that exploratory trades are safe or profitable. Haarnoja et al. introduced SAC in 2018.

What “soft” changes

At a state $s$ , the policy balances estimated value $Q(s,a)$ against its concentration:

V^\pi(s)=\mathbb{E}_{a\sim\pi(\cdot\mid s)}[Q^\pi(s,a)-\alpha\log\pi(a\mid s)].

The second term is an entropy contribution in expectation. Natural logarithms give entropy in nats; temperature

\alpha>0

expresses its weight in reward units per nat. The critic already includes future entropy through its soft Bellman recursion. Entropy therefore changes the value being learned, rather than merely adding random noise to a greedy action. The original paper, §3–4, develops this maximum-entropy policy evaluation and improvement.

Two choices make the trade-off visible

Consider fixed, fictional action values $Q_A=2$ and $Q_B=0$ , in arbitrary reward units, with $\alpha=1$ . Let $p$ be the probability of action A. The score is

F(p)=2p-p\log p-(1-p)\log(1-p),

using the continuous endpoint convention $0\log0=0$ . Differentiating gives $F'(p)=2+\log((1-p)/p)$ . Its unique interior maximum satisfies

p=\frac{e^2}{e^2+1}=0.880797078,\qquad 1-p=0.119202922.

The corresponding accounting is:

Quantity	Value
Expected Q	1.761594156 reward units
Entropy	0.365333855 nats
Expected Q plus weighted entropy	2.126928011 reward units
Always choose A: expected Q and soft score	2 reward units

The larger soft score is not extra trading profit: part of it is an entropy bonus. The maximum also equals

\log(e^2+1)

. With fixed values, lowering temperature concentrates probability on A; increasing it moves the probabilities towards equality. Multiplying both Q values and temperature by the same positive factor preserves these probabilities. Changing reward units without changing temperature changes the optimisation problem.

Synthetic two-action objective with Q values 2 and 0 and temperature 1. The maximum occurs at probability 0.880797 for the higher-Q action: expected Q 1.761594 plus entropy 0.365334 gives 2.126928. Always choosing that action gives score 2.

An original finite-action illustration: the soft optimum sacrifices some expected Q to gain entropy. These curves do not simulate continuous SAC training.

From that picture to continuous actions

The two-action calculation illustrates the objective; continuous SAC normally fits a parameterised density instead of explicitly normalising a list of action values.

In the common version without a separate value network, draw transitions

(s,a,r,s',d)

from a replay buffer and a fresh next action

a'\sim\pi_\phi(\cdot\mid s')

. A critic target is

y=r+\gamma(1-d)\left[\min_{i=1,2}Q_{\bar\theta_i}(s',a')-\alpha\log\pi_\phi(a'\mid s')\right].

Here $d=1$ marks genuine termination, not an arbitrary data cutoff. Each critic fits this target. Slowly averaged target parameters $\bar\theta_i$ stabilise bootstrapping; taking the smaller critic estimate limits overestimation. It does not produce a statistical lower confidence bound. The actor minimises

J_\pi(\phi)=\mathbb{E}_{s\sim\mathcal D,\,a\sim\pi_\phi} \left[\alpha\log\pi_\phi(a\mid s)-\min_{i=1,2}Q_{\theta_i}(s,a)\right],

usually differentiating through a reparameterised action sample. Replayed states support off-policy learning; the actor's sampled actions come from its current policy. Spinning Up's SAC equations explain this target and policy loss.

A continuous density is not an action probability. Its entropy is differential entropy: it can be negative and depends on action coordinates. For Gaussian samples

u

mapped to normalised actions

a=\tanh u

, the correct density obeys

\log\pi(a\mid s)=\log\mu(u\mid s)-\sum_j\log(1-\tanh^2u_j),

where

\mu

is the pre-squash Gaussian density. Omitting this Jacobian correction changes the entropy calculation. Further rescaling into physical action units needs its own change-of-variables adjustment and a consistent entropy target. See Algorithms and Applications, Appendix C.

Which SAC version?

The ICML 2018 algorithm included a separate learned state-value network and target value network, alongside twin critics. Algorithms and Applications, initially posted in December 2018 and revised in January 2019, removed that extra value network and introduced automatic temperature adjustment from an average-entropy constraint. Automatic adjustment is an extension, not a defining requirement of every SAC implementation. It still requires choosing an entropy target in specified action coordinates.

For financial experiments, report reward-only performance separately from the training objective, and state whether evaluation samples actions or uses a deterministic action. Replay does not turn a fixed historical price record into an environment with credible responses to unseen orders. Fees, impact, inventory constraints and information timing must belong to the experiment before interpreting its returns.

For the underlying sequential-decision framework, Markov Decision Processes and Value/Policy Iteration develops states, rewards and Bellman recursions. A public preview is available; the full lesson requires membership.

Sources

Haarnoja et al. (2018), Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, §§3–4 and Algorithm 1.
Haarnoja et al., Soft Actor-Critic Algorithms and Applications, §§4–6 and Appendix C; v2 dated 29 January 2019.
OpenAI, Spinning Up: Soft Actor-Critic, Key Equations and Pseudocode.

What “soft” changes

Two choices make the trade-off visible

From that picture to continuous actions

Which SAC version?

Sources

Related terms

Reinforcement learning