LoRA fine-tuning: teaching the model to give the label you need

The next system wants one word. The model gives it a paragraph.

Imagine a support queue where each report needs a label such as CERTIFICATE or STORAGE. The model has the report, yet returns an explanation where the queue expects a short answer. More information is not necessarily what is missing. The behavior may need practice.

Fine-tuning continues a model’s training with examples of the behavior you want. One method, LoRA, lets you keep the original model weights fixed and train a smaller addition instead. You can save that addition as an adapter, load it beside the base model, and compare the answers with and without it.

That is a different job from supplying current facts. If the assistant needs today’s procedure, retrieving that document is usually more direct than training yesterday’s procedure into the model. Here the question is whether a model can learn to give a useful label more consistently. Four invented examples will show where a LoRA change lives; unfamiliar reports will have to tell us whether it earns its place.

A change beside the original calculation

Follow the report “Certificate expired on the public endpoint” as the model processes it. A selected calculation uses the base model’s weights as usual. LoRA adds another route alongside it: two smaller learned arrays of numbers, called matrices A and B, produce an update, which is scaled and added to the base result. Training changes this extra route while leaving the original weights alone.

The name LoRA means low-rank adaptation. “Low-rank” describes a restriction on the changes that route can express. A tiny invented calculation makes it easier to see. Set the scale factor to one. Let A take the input [2, 1] and subtract the second value from the first, leaving 1. Let B turn that number into the update [3, 2]. Give the same A and B the input [4, 2] and the update becomes [6, 4].

The amount has changed, but the update still points along the same [3, 2] direction. This route has rank one: it cannot independently choose every possible two-number update. The script later uses rank four, rather than this toy calculation. The idea carries over: an adapter learns a constrained change, not a miniature copy of all the base weights. Full fine-tuning, by comparison, can update every model parameter.

Saving A and B also explains why an adapter needs its compatible base checkpoint and configuration. It saves the addition, not the original calculation that the addition belongs to. With the base fixed, loading the adapter adds the update; removing an unmerged adapter restores that layer’s base computation. We can give the same report to both arrangements and look at what changes.

A small adapter file can still require a substantial training machine. The base model must be in memory, along with the intermediate values made while the report passes through it, called activations, and the optimizer’s working state for calculating updates. The weights guide follows that underlying training process. File size tells us what was saved, not everything the machine had to hold while making it.

Four reports, one saved adapter

The first run is deliberately small enough to follow from beginning to end. It uses four synthetic reports and the checkpoint described in the SmolLM2 model card. PEFT supplies the adapter setup and saving. This is a documentation-based exercise, not Nate’s original training recipe or a measured improvement.

Use an isolated practice machine with supported Python and PyTorch, space for the checkpoint download and enough memory for training. The code stays on the CPU to avoid GPU setup. Even this small model can take time and considerable memory; if the machine runs out, stop rather than raising shared limits on a serving system.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install torch transformers peft
python -m pip freeze > requirements-used.txt

Save the next script as lora-smoke.py and run python lora-smoke.py. It downloads the base model, takes four training steps, saves an adapter, then reloads it with the recorded base revision. The saved dependency list and run record belong beside the adapter: they describe the setup that produced it.

Before the loop, look at the first pair. The input is “Certificate expired on the public endpoint.” The desired answer is “CERTIFICATE.” The script places the report after a classification instruction and before “Class:”, then appends that answer and an end token. The model has a particular continuation to learn, right down to where it should stop.

import json
from pathlib import Path
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, PeftModel

base = 'HuggingFaceTB/SmolLM2-135M-Instruct'
torch.manual_seed(7)
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base)
revision = model.config._commit_hash
model = get_peft_model(model, LoraConfig(
    task_type='CAUSAL_LM', r=4, lora_alpha=8,
    target_modules=['q_proj', 'v_proj'], lora_dropout=0.0,
))
model.print_trainable_parameters()
# Synthetic smoke data, not a representative training collection.
train = [
    ('Certificate expired on the public endpoint.', 'CERTIFICATE'),
    ('The disk cannot create files because no inodes remain.', 'STORAGE'),
    ('TLS negotiation reports an expired certificate.', 'CERTIFICATE'),
    ('The volume has exhausted available space.', 'STORAGE'),
]

def prompt(text):
    return ('Classify the report as CERTIFICATE, STORAGE or UNKNOWN.\n'
            'Report: ' + text + '\nClass:')

optimizer = torch.optim.AdamW(
    [p for p in model.parameters() if p.requires_grad], lr=1e-4)
model.train()
for text, answer in train:
    prefix = tokenizer(prompt(text), add_special_tokens=False)['input_ids']
    suffix = tokenizer(' ' + answer + tokenizer.eos_token,
                       add_special_tokens=False)['input_ids']
    ids = torch.tensor([prefix + suffix])
    labels = ids.clone()
    labels[:, :len(prefix)] = -100
    optimizer.zero_grad()
    loss = model(input_ids=ids, attention_mask=torch.ones_like(ids),
                 labels=labels).loss
    loss.backward()
    optimizer.step()
    print('training loss', float(loss.detach()))

out = Path('certificate-adapter-smoke')
model.save_pretrained(out)
tokenizer.save_pretrained(out)
(out / 'run.json').write_text(json.dumps({
    'base': base, 'revision': revision, 'seed': 7,
    'purpose': 'synthetic four-step smoke test, no efficacy claim',
}, indent=2))
# Reload the exact base revision with the saved adapter.
reloaded_base = AutoModelForCausalLM.from_pretrained(base, revision=revision)
reloaded = PeftModel.from_pretrained(reloaded_base, out)
reloaded.eval()
inputs = tokenizer(prompt('The certificate validity period has ended.'),
                   return_tensors='pt', add_special_tokens=False)
with torch.no_grad():
    output = reloaded.generate(**inputs, max_new_tokens=8, do_sample=False,
                               pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0, inputs['input_ids'].shape[1]:],
                       skip_special_tokens=True))

The report and instructions remain visible during training, but the code sets their labels to -100. That excludes them from the loss, the score used to measure prediction error. We want to score the answer continuation, including its end token, rather than reward the model for copying the question. For the first pair, the comparison asks how well it predicted the supplied “CERTIFICATE” continuation.

The backward calculation works out how that error responds to the trainable parameters, and the optimizer takes one update step. In this script, the trainable additions sit on the selected query and value projections, two transformations within attention. The base weights remain fixed. Then the loop repeats for the other three report-and-label pairs.

Training and generation both use the same plain completion format here. Switching the application to chat messages would require consistent preparation and a new evaluation. Otherwise we would be practicing one way of asking and judging another.

After saving, the script loads the recorded base revision and attaches the adapter. It asks about a differently worded report, “The certificate validity period has ended,” and prints the generated continuation. Read what actually appears, including a wrong label or extra prose. A lower training loss would mean a better fit to the practice examples, not a promise about this answer.

Even a perfect “CERTIFICATE” here would be an encouraging first look at a very familiar problem. The sentence is close to the training material. The reports waiting in a real queue may be much less accommodating. Training was not executed for this guide, so no output or improvement is claimed.

The next report must really be new

Suppose two reports describe the same certificate incident in different words. Put one into training and the other into the final test, and the test now rewards familiarity with that case. The filenames may say “train” and “test”; the underlying incident has crossed the line between them.

A useful collection needs reports you have permission to use, reviewed labels, and separate training, development and final-test sets divided by the underlying case. Keep paraphrases and related records together. Give reports outside the known classes a meaningful UNKNOWN label instead of making every problem choose between certificates and storage.

Before training, save the base setup’s answers to the held-out questions. Now there is something concrete for the adapter to improve on. For each report, examine the class, the required format, appropriate uncertainty and how it handles unrelated requests. Choose thresholds before looking at final-test answers. A tidy label that sends a report to the wrong team may be more costly than an otherwise harmless formatting difference.

Development cases are where settings can change. The final test waits for the candidate you select. Keep revisiting that final set to repair each mistake and it becomes another development set, whatever its folder is called. Retain failures as well as successes so the result still describes the work the model struggled with.

The reports themselves deserve a read before any training starts. Conflicting labels give the model conflicting lessons. Private information can survive training, too; possession of the resulting checkpoint does not grant permission to distribute the material it learned from.

Does the label help the queue?

Give the base and adapted model the same unseen work, keeping the prompt, decoding settings, runtime, supplied evidence and hardware constraints steady. If those also change, you can compare the two complete setups, but the whole difference cannot be credited to LoRA.

If representative unseen reports do not improve, keep the working base setup. Loading the original checkpoint and prompt without the adapter gives you a clear way back. Keep them separately rather than merging the experiment into your only baseline.

We started with a queue asking for one word and a model inclined to say more. The saved adapter lets us try a change without confusing it with the model we started from. The decision to keep it comes from the unfamiliar reports: the right class, in the right format, compared with what the base already did. That is the moment the training exercise can become useful to the people waiting for their reports to reach the right place.