<-- home

LoRA adapters: one base model, many skills

March 18, 2025

Imagine you bought a giant encyclopedia. It represents a massive AI model, like GPT-4 or Llama 3. You want to customize it so it becomes an expert on your company’s internal rules. That customization is fine-tuning.

1. What is LoRA, and why is it needed?

Full fine-tuning rewrites the encyclopedia and saves another giant copy. Three specialties can mean three complete model checkpoints.

LoRA (Low-Rank Adaptation) keeps the original model frozen and learns a slim packet of edits called an adapter. At each selected layer:

W is the frozen pretrained matrix. The adapter contains two much smaller trainable matrices, A and B. Rank r controls the narrow bottleneck between them.

LoRA works when the task-specific change fits inside this smaller subspace. It is effective often, but not guaranteed; rank and target layers still need evaluation.

2. Is training a LoRA adapter cheaper?

For a 4096 x 4096 matrix, full fine-tuning can update 16,777,216 parameters. A rank-16 adapter learns 131,072: 0.78% as many for that matrix.

Across 32 layers with LoRA on q_proj and v_proj, that illustrative rank-16 adapter is about 16 MiB in 16-bit storage. A 7-billion-parameter 16-bit base is about 14 GB: the saved adapter is roughly 800x smaller.

Training memory does not fall 800x because activations and the base weights remain. The saving comes from keeping gradients and optimizer state only for A and B.

Figure 1. Step through training. Compare full fine-tuning with LoRA, then follow what is computed, traversed, stored, and updated in each layer.
Training mode
16,777,216 full-matrix parameters 131,072 rank-16 adapter parameters 0.78% trainable
computed gradient traversed updated frozen
Three sequential layers with parallel LoRA paths Each activation enters a layer, splits through frozen W and trainable A then B, sums, and becomes the input activation for the next layer. The final activation produces the loss. h0 Layer 1 waiting W frozen A1 B1 + h1 Layer 2 waiting W frozen A2 B2 + h2 Layer 3 waiting W frozen A3 B3 + h3 objective loss

LoRA mode: W is computed but frozen; A/B are trainable. Start with the forward pass.

Illustrative memory comparison per layer Bars are scaled within each row. Values are explanatory, not model-wide measurements.
Memory Full fine-tuning LoRA Why it matters
Weights 2 MBW 2.016 MBW + A/B The frozen base still has to be resident.
Gradients 2 MBW gradients 16 KBA/B gradients 125x less gradient memory.
Optimizer 4 MBW state 32 KBA/B state 125x less optimizer memory.
Activations 64 KBsaved for backward 64 KBsaved for backward No saving in this simplified example.
Trainable state 6 MBgradients + optimizer 48 KBgradients + optimizer 125x smaller
Total shown 8.064 MB 2.128 MB 3.8x smaller overall

There is no universal “training is N times cheaper” number. Sequence length, batch size, quantization, checkpointing, and target modules still dominate the real GPU bill.

3. Is serving LoRA cheaper?

The operational win is sharing one base. One hundred 16 MiB adapters add about 1.6 GB of artifacts; one hundred separate 14 GB checkpoints would be about 1.4 TB.

Figure 2. One base, many named adapters. A router selects the small update used for each request.
Choose an illustrative request
request My password reset link expired. What should I do?
loaded adapters
support active
finance idle
base only idle
active: support
Three-layer inference with a selected LoRA adapter A request activation passes through three sequential layers. In each layer, the frozen base path and selected adapter path are added. Base-only mode removes the adapter path. The final activation produces a token. h0 Layer 1 waiting W frozen A1 B1 + h1 Layer 2 waiting W frozen A2 B2 + h2 Layer 3 waiting W frozen A3 B3 + h3 generated token

The request uses W + delta W_support. The finance adapter stays loaded but inactive.

shared base14 GB loaded adapters32 MiB active adapter16 MiB KV cachegrows per token

The implementation idea is small:

# Pseudocode: train independent adapters from one frozen base.
base = load_model("base", frozen=True)
for name, data in tasks:
    adapter = train_lora(base, data, rank=16)
    save_adapter(name, adapter)

# Serve one base with several named adapters.
server = load_model("base")
server.load_adapter("support")
server.load_adapter("finance")

for batch in requests:
    names = [route(request) for request in batch]
    responses = server.generate(batch, adapter_names=names)

Deployment Strategies in Practice

When putting LoRA into production, you have a few ways to manage how adapters interact with the base model:

  • Switching: Activating one specific adapter for an entire request or a homogeneous batch.
  • Mixed-Adapter Batching: Assigning different adapters to individual sequences within the same batch. Production-scale implementations typically use specialized serving engines such as vLLM or LoRAX.
  • Composition: Mathematically blending multiple adapters together dynamically, rather than just routing requests between them.
  • Merging: Permanently baking the adapter’s weights into the base model. This removes the separate adapter-layer inference overhead but destroys the ability to hot-swap adapters on the fly.

LoRA makes specialization dramatically smaller. Training still traverses the base, and serving still computes the base, but many specialists can share that one expensive foundation.

References

  1. LoRA: Low-Rank Adaptation of Large Language Models
  2. Hugging Face PEFT LoRA reference
  3. PEFT checkpoint format