Parameter-efficient fine-tuning (PEFT) adapts a pretrained model to a target task by training a small subset of its parameters or adding a small trainable component, while keeping most or all base weights fixed. For a quant team adapting a language model to financial disclosures, it can make separate task adaptations cheaper to train and store. It does not establish that the resulting text scores measure the intended financial concept.
The useful question is where the task-specific change enters the computation. A low-rank update, a hidden-feature adapter and a learned attention prefix can all leave the base checkpoint unchanged, but they give the optimiser different ways to influence predictions.
Three ways to steer the same base
A weight-update branch. Low-rank adaptation (LoRA) learns two small matrices whose product modifies a selected weight matrix. In a simplified unscaled form, the layer computes : the base matrix stays fixed and learn the task-specific change. The linked article explains rank and scaling in detail. LoRA is one PEFT method, rather than another name for the whole family. Hu et al., §4.1
An inserted feature transformation. A bottleneck adapter sends a hidden vector down to a narrower dimension, applies a nonlinearity, projects back up and adds a residual connection. A simplified bias-free adapter is
Here compresses features, expands them and is a nonlinear activation. Unlike a linear LoRA update, the nonlinear branch generally cannot be absorbed into a single fixed weight matrix. Houlsby et al.'s original Transformer design inserts two adapters per block and also trains task-specific layer-normalisation parameters and a classification head. An adapter-only toy count therefore understates that complete configuration. Houlsby et al., §2.1 and Figure 2
A learned attention context. Prefix tuning learns continuous vectors that act as extra context for attention. In the autoregressive formulation, subsequent tokens can attend to learned prefix keys and values at each layer while the language model remains frozen. These are learned numerical representations, not necessarily embeddings of real words. Prefix tuning reaches into attention layers; merely typing an instruction before a document performs no parameter training. Li and Liang also distinguish it from learning embeddings only at the input layer. Li and Liang, §4.1–4.2
Illustrative local counts for different insertion units. The mechanisms change different parts of the computation; the numbers are not a performance or peak-memory comparison.
Count the insertion, then count the model
For an illustrative hidden width of 1,024, count only the stated components:
| Local component | Count | Trainable scalars |
|---|---|---|
| LoRA on one square matrix, rank 8 | 16,384 | |
| One adapter, bottleneck 16, no biases | 32,768 | |
| One layer's direct prefix, 20 positions, separate key and value widths of 1,024 | 40,960 |
These are different units of insertion. A model may adapt several projection matrices in each block, place multiple adapters per block, or attach prefixes across many layers. Add any trainable head, biases or normalisation parameters to the actual inventory. The table does not show that LoRA is the best method or that its whole-model memory consumption is smallest.
The prefix row counts directly learned key/value entries. Li and Liang use an additional neural network to generate the prefix during training, then retain only the resulting prefix for deployment. That training parameterisation has a different count from the stored entries shown here. Li and Liang, §4.3
Frozen weights still participate
“Frozen” means the optimiser does not update those weights. Forward passes still use the base model, and gradients can still propagate through frozen operations to reach earlier trainable components. Freezing base weights can remove their parameter-gradient buffers and optimiser states; it does not eliminate activation memory. Sequence length, batch size, precision, checkpointing and insertion locations all affect peak training memory.
Deployment costs differ too. A compatible LoRA update can be merged into its base matrix for inference; a nonlinear adapter remains an extra computation, while a prefix adds attention context. Measure the actual configured system rather than treating the trainable-parameter fraction as a latency estimate. Hu et al., §3–4.1
For a financial-tone experiment, keep the checkpoint and tokenizer fixed while comparing adaptation choices on the same labelled data and evaluation rule. Record the trainable components, their locations, and the checkpoint version alongside each saved adaptation. A small task-specific file depends on that base: attaching it to a different checkpoint need not reproduce the model.
Choose adaptation settings on validation data and reserve a later test period for the final assessment. A smaller trainable component can still overfit labels, and freezing a model does not repair future information already present in its inputs or pretraining corpus. The SelfMadeQuant Transformers lesson develops the financial-text validation context (public preview; full lesson requires membership).
Sources
- Houlsby et al. (2019), Parameter-Efficient Transfer Learning for NLP, §2–2.1: adapter architecture, insertion points and additional trainable components.
- Li and Liang (2021), Prefix-Tuning: Optimizing Continuous Prompts for Generation, §3–4.3: per-layer prefix activations, frozen language model and training reparameterisation.
- Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models, §2–4.1: low-rank updates, adapter overhead and merging for inference.