LLM Fine-Tuning: From Transfer Learning to Domain Adaptation

Introduction

In 2026, the landscape of large language models is defined by abundance. Thousands of pre-trained models are available for download — Llama 3, Qwen 2.5, DeepSeek V3, Mistral, Phi-4, Gemma, and dozens more. Each one has been trained on trillions of tokens at enormous compute cost. And yet, nearly every production deployment requires additional training. The reason is simple: general-purpose models are not optimised for specific tasks.

Fine-tuning bridges this gap. It takes a pre-trained model and adapts it to a domain, a behaviour, or a task by continuing the training process on a carefully curated dataset. The cost of fine-tuning is a tiny fraction of pre-training — hours or days instead of weeks or months — but the improvement in task-specific performance can be dramatic.

This guide covers every major dimension of LLM fine-tuning. We start with transfer learning foundations, walk through supervised fine-tuning, parameter-efficient methods like LoRA and QLoRA, preference optimisation techniques like RLHF and DPO, data preparation, hyperparameter selection, GPU requirements, evaluation strategies, and the decision framework for choosing between fine-tuning, RAG, and prompting. By the end you should have a complete mental model of when and how to fine-tune in production.

Transfer Learning Foundations

Fine-tuning is an application of transfer learning: knowledge acquired while solving one problem is transferred to a related problem. In the context of LLMs, the source problem is language modelling at internet scale, and the target problem is your specific domain or task.

Pre-training produces a model that has learned general patterns of language: syntax, semantics, factual knowledge, reasoning structure, and stylistic conventions. These patterns are captured in the model's weights — billions of floating-point numbers that represent the accumulated statistics of the training data. Fine-tuning adjusts these weights to emphasise the patterns relevant to your task while preserving the general knowledge that the model acquired during pre-training.

Figure: Transfer learning conceptual diagram. A model pre-trained on broad data is fine-tuned on a narrow domain or task. The pre-trained weights provide a strong initialisation, reducing the data and compute required for the target task. (Wikimedia Commons, CC BY-SA 4.0)

The key insight is that the pre-trained weights provide a far better starting point than random initialisation. Training a transformer from scratch on domain data would require orders of magnitude more data and compute. Fine-tuning exploits the fact that language structure transfers across domains — legal writing shares syntax with general English, even if the vocabulary and argument patterns differ. The pre-trained model already knows how to form grammatical sentences; fine-tuning teaches it what to say.

Full Fine-Tuning vs Parameter-Efficient Fine-Tuning

There are two broad families of fine-tuning: full fine-tuning and parameter-efficient fine-tuning (PEFT). They differ in how many parameters are updated and in how the training process interacts with the base model.

Full Fine-Tuning

Full fine-tuning updates every parameter in the model. For a 7-billion-parameter model, this means computing gradients and applying updates to all 7B weights. This gives the maximum possible capacity to adapt to the target domain, but it comes with significant practical costs. You need enough GPU memory to store the full model, the gradients, the optimizer states, and the activations — typically 4-6 times the model size in memory. For a 70B model, that means 8-16 H100 GPUs just to hold the training state.

Full fine-tuning also produces a complete copy of the model for every variant. If you fine-tune for five different tasks, you must store five independent copies of the weights. At 140 GB per copy for a 70B model in bfloat16, storage and serving costs multiply quickly.

Parameter-Efficient Fine-Tuning (PEFT)

PEFT methods freeze most of the model's parameters and only train a small number of additional parameters — typically 0.1% to 2% of the total. This reduces GPU memory requirements dramatically and allows you to train multiple adapters for different tasks on top of a single base model, switching between them at inference time with negligible overhead.

The dominant PEFT method is LoRA (Low-Rank Adaptation), which we cover in depth below. Other methods include prefix tuning, prompt tuning, adapter layers, and IA3. In practice, LoRA and its variants account for the vast majority of production fine-tuning because they offer the best trade-off between adaptation quality, memory efficiency, and inference flexibility.

Supervised Fine-Tuning: Data Preparation

Supervised fine-tuning (SFT) is the most common fine-tuning paradigm. You collect input-output pairs that demonstrate the desired behaviour and train the model to predict the outputs given the inputs. For instruction-tuned models, these pairs are typically formatted as conversational turns with system prompts, user messages, and assistant responses.

Data Formatting

The format of your training data must match the chat template expected by the model. Most modern LLMs use a structured conversation format with special tokens marking each turn. Llama 3 uses a format with <|begin_of_text|>, <|start_header_id|>, and <|eot_id|> tokens. Qwen uses a simpler format with <|im_start|> markers. Mistral uses [INST] and [/INST] tags.

Hugging Face's transformers library provides the apply_chat_templatemethod on each tokenizer, which automatically formats conversations according to the model's expected format. Always use this method rather than hard-coding format strings, because the template is part of the model's official configuration and changes between versions.

Loss Masking

During SFT, you only want the model to learn from the assistant's response, not from the user's input or the system prompt. Loss masking sets the loss contribution of non-assistant tokens to zero, so the model is never penalised for failing to predict the user's message. The SFTTrainer from TRL handles this automatically when you pass formatted conversations.

SFT: Loss Functions and Training Dynamics

Supervised fine-tuning uses the standard autoregressive language modelling loss: cross-entropy loss on the next-token prediction task. For each position in the target sequence, the model predicts a probability distribution over the vocabulary, and the loss is the negative log-probability of the correct token.

L = -∑ log P(y_t | y_<t, x)

Where x is the input (system prompt + user message) and y is the target (assistant response). The sum runs over all token positions in the target that are not masked.

Training dynamics during SFT differ from pre-training in important ways. The dataset is typically 100 to 100,000 examples, compared to trillions of tokens for pre-training. This means the model can easily overfit if trained for too many epochs. Early stopping, learning rate scheduling, and weight decay are essential. We cover these in the hyperparameters section below.

Another key difference is that SFT datasets are usually instruction-following or dialogue data, not raw text. The model must learn to distinguish between different roles in the conversation and to produce responses that follow the instruction. This requires the model to internalise the conversational format, not just language patterns.

LoRA: Low-Rank Adaptation

LoRA, introduced by Hu et al. in 2021 (arXiv:2106.09685), is the most widely used parameter-efficient fine-tuning method. The core insight is that the weight updates during fine-tuning have low intrinsic rank — they can be represented as the product of two smaller matrices.

For a pre-trained weight matrix W ∈ &Ropf;^d×k, LoRA constrains the update ΔW such that:

W′ = W + BA, where B ∈ &Ropf;^d×r, A ∈ &Ropf;^r×k, and r << min(d, k)

The rank r is typically 8, 16, or 32. At r = 8 and d = k = 4096 (typical for a 7B model), the full update matrix would have 16.8 million parameters, but the two LoRA matrices together have only 65,536 parameters. That is a 256x reduction.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.linear = nn.Linear(in_features, out_features, bias=False)
        # Freeze the original weights
        for param in self.linear.parameters():
            param.requires_grad = False
        # Low-rank decomposition matrices
        self.lora_a = nn.Parameter(torch.randn(in_features, rank) * 0.01)
        self.lora_b = nn.Parameter(torch.zeros(rank, out_features))
        self.scaling = alpha / rank

    def forward(self, x):
        # Original forward pass (frozen) + LoRA adaptation
        return self.linear(x) + (x @ self.lora_a @ self.lora_b) * self.scaling

In practice, LoRA is applied to the query and value projection matrices in the attention layers (and optionally the key and output projections). The rank, the set of target modules, and the scaling factor alpha are the main hyperparameters. Higher ranks allow more adaptation capacity but increase memory and compute. Typical values are r = 16 for most tasks, r = 8 for simple format adaptation, and r = 32 for domain-intensive tasks like code or medical text.

LoRA adapters can be merged into the original weights for inference, eliminating any latency overhead. They can also be kept separate and swapped dynamically, enabling multi-task serving from a single base model with zero additional latency per task.

QLoRA: Quantized LoRA

QLoRA, introduced by Dettmers et al. in 2023 (arXiv:2305.14314), extends LoRA by quantising the base model to 4-bit precision while training the LoRA adapters in full precision (bfloat16). This reduces GPU memory requirements by approximately 4x compared to standard LoRA with a bfloat16 base model.

QLoRA achieves this through two key techniques. The first is NormalFloat4 (NF4), a data-type designed for normally distributed weights that provides better signal-to-noise ratio than standard uniform quantisation at 4 bits. The second is double quantisation, where the quantisation constants themselves are quantised, saving additional memory.

With QLoRA, a 70B parameter model can be fine-tuned on a single 48 GB GPU (like an A6000 or 2x RTX 6000 Ada). A 7B model fits on an RTX 3090 or 4090 with 24 GB of VRAM. This has democratised fine-tuning, making it accessible to individual developers and small teams without access to multi-GPU clusters.

The quality degradation from 4-bit quantisation during training is surprisingly small. Dettmers et al. showed that QLoRA with NF4 quantisation achieves within 0.5-1% of full fine-tuning performance on most benchmarks, provided the LoRA rank is large enough (r ≥ 16). For many production use cases, this gap is negligible compared to the cost savings.

Advanced PEFT: AdaLoRA, IA3, DoRA

While LoRA and QLoRA dominate production deployments, several other PEFT methods offer advantages in specific scenarios.

AdaLoRA

AdaLoRA (arXiv:2303.10512) dynamically allocates rank across weight matrices based on their importance. Instead of using the same rank for every layer, AdaLoRA learns which layers benefit from higher rank and assigns more parameters to them. Layers that are already well-aligned with the target task receive lower rank, while layers that need significant adaptation receive higher rank. This produces better performance at the same total parameter budget.

IA3

IA3 (Infused Adapter by Inhibiting and Amplifying Activations) (arXiv:2205.05638) is even more parameter-efficient than LoRA. Instead of learning low-rank matrices, IA3 learns element-wise scaling vectors for the key, value, and feed-forward activations. This adds only 0.01% of the model's parameters, compared to LoRA's 0.1-2%. IA3 works well for classification and regression tasks but underperforms LoRA on generative tasks like instruction following and chat.

DoRA

DoRA (Weight-Decomposed Low-Rank Adaptation) (arXiv:2402.09353) decomposes the weight update into magnitude and direction components. The direction component is learned via LoRA, while the magnitude is learned as a separate scaling vector. This decomposition aligns better with the optimisation landscape of neural network training, and DoRA consistently outperforms standard LoRA at the same rank on instruction-following and reasoning benchmarks.

Instruction Tuning and Chat Fine-Tuning

Instruction tuning is a specific form of SFT where the training data consists of instructions paired with high-quality responses. The goal is to teach the model to follow instructions reliably, even for tasks it was not explicitly trained on. This is the technique that transformed base language models into useful assistants.

The seminal work on instruction tuning is Google's FLAN (Fine-tuned LAnguage Net) paper (arXiv:2201.13161), which showed that fine-tuning on a diverse collection of instructional datasets — including translation, summarisation, reasoning, and question answering — produces a model that generalises to unseen tasks significantly better than the base model or a model fine-tuned on a single task.

Chat fine-tuning extends instruction tuning with multi-turn conversation data. The model must learn to maintain context across turns, handle follow-up questions, and produce coherent multi-turn dialogues. The widely used ShareGPT dataset (containing real conversations from ChatGPT) and the OpenAssistant Conversations dataset are common starting points, though their quality is uneven and they require significant filtering.

Modern chat models are typically fine-tuned in three stages: first on general instruction data (10k-100k examples), then on domain-specific data (1k-10k examples), and finally on preference data (discussed below). Each stage uses a lower learning rate and shorter training duration than the previous one.

RLHF: Reinforcement Learning from Human Feedback

SFT teaches the model to imitate human-written responses, but imitation does not ensure that the model can distinguish good responses from bad ones. Reinforcement Learning from Human Feedback (RLHF), popularised by OpenAI's InstructGPT paper (arXiv:2203.02155), addresses this by training a reward model that scores response quality and then using reinforcement learning to align the language model with the reward signal.

The RLHF pipeline has three stages:

• SFT: Supervised fine-tuning on high-quality demonstration data to establish baseline behaviour.
• Reward modelling: A separate model is trained to predict human preference judgments. Given a prompt and two responses (A and B), the reward model learns to assign higher scores to the response that human raters preferred.
• RL optimisation: The language model is optimised using Proximal Policy Optimization (PPO) to maximise the reward score while staying close to the SFT model via a KL divergence penalty. The KL penalty prevents the model from exploiting the reward model by producing superficially high-scoring but nonsensical outputs.

RLHF produces noticeably better alignment than SFT alone. Models trained with RLHF are more helpful, more honest, and less likely to produce harmful outputs. However, RLHF is operationally complex: it requires maintaining four models simultaneously (the policy, the reference model, the reward model, and the value model), careful hyperparameter tuning, and significant compute — typically 2-3x the cost of SFT for the same base model.

DPO: Direct Preference Optimization

Direct Preference Optimization (DPO), introduced by Rafailov et al. in 2023 (arXiv:2305.18290), simplifies RLHF by eliminating the need for a separate reward model. DPO reparameterises the RLHF objective so that the language model itself implicitly learns the reward function through a binary preference loss.

The DPO loss compares the log-probabilities of the chosen and rejected responses under the current policy relative to the reference model:

L_DPO = -&Eopf; [log σ(β log (π_θ(y_w | x) / π_ref(y_w | x)) - β log (π_θ(y_l | x) / π_ref(y_l | x)))]

Where y_w is the preferred response, y_lis the dispreferred response, β controls how far the policy can deviate from the reference model, and σ is the logistic sigmoid function. Intuitively, DPO increases the relative probability of preferred responses while decreasing the relative probability of dispreferred ones.

from datasets import load_dataset
from trl import DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    torch_dtype=torch.bfloat16
)
ref_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    torch_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct"
)

dataset = load_dataset("json", data_files="preference_data.jsonl")

dpo_trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=training_args,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
    beta=0.1
)

dpo_trainer.train()

DPO has largely replaced RLHF in open-source fine-tuning pipelines because it is simpler to implement, requires fewer GPUs, and produces comparable or better results. The DPOTrainer from Hugging Face TRL handles all the details — loss computation, reference model management, and batch construction. For most teams building preference-aligned models in 2026, DPO is the default starting point.

ORPO and Other Preference Optimization Methods

DPO's success has sparked a wave of variants, each addressing specific limitations of the original formulation.

ORPO

Odds Ratio Preference Optimization (ORPO) (arXiv:2403.07691) fuses SFT and preference optimisation into a single stage. Instead of first fine-tuning on demonstrations and then running DPO, ORPO adds a penalty term to the standard language modelling loss that discourages the model from generating dispreferred responses. This eliminates the two-stage pipeline entirely, saving training time and simplifying the codebase.

ORPO has been particularly successful for medium-sized models (1-7B parameters) where the reference model in DPO can constrain the policy too tightly. By removing the reference model dependency, ORPO allows the model to explore more freely during training, often leading to better alignment at the same training budget.

KTO and IPO

Kahneman-Tversky Optimization (KTO) (arXiv:2402.01306) works with unpaired preference data — you only need examples of good or bad outputs, not pairs. This is useful when collecting pairwise preferences is impractical. Identity Preference Optimization (IPO) (arXiv:2310.12036) modifies the DPO loss to be more robust to noise in the preference labels, which helps when human raters disagree.

Data Quality: Curation, Deduplication, Formatting

In fine-tuning, data quality matters more than data quantity. A well-curated dataset of 1,000 examples consistently outperforms a noisy dataset of 100,000 examples across every published benchmark. This is the single most important lesson for practitioners.

Curation Principles

• Diversity over volume: Your dataset should cover the full range of inputs the model will encounter in production. A legal fine-tuning dataset needs contracts, correspondence, opinion letters, pleadings, and client communications — not 10,000 variants of a contract clause.
• Correctness first: Every example in your dataset should be factually correct and structurally sound. A single incorrect example can teach the model a wrong pattern that takes hundreds of correct examples to unlearn.
• Representative distribution: The distribution of tasks, lengths, and difficulty levels in your training data should match what the model will see in production. If 80% of production queries are short, your dataset should not be 80% long.

Deduplication

Duplicate examples skew the training distribution and waste the model's limited capacity. Exact deduplication (removing identical strings) catches the obvious cases, but near-duplicate detection using embeddings and cosine similarity is more important. Two examples that differ by a single word or a renamed variable will be treated as distinct by exact matching but carry nearly identical learning signal. MinHash LSH or embedding-based clustering with a threshold of 0.85-0.95 cosine similarity catches most near-duplicates.

Decontamination

If your fine-tuning dataset overlaps with benchmark evaluation sets, your evaluation results will be inflated and misleading. N-gram overlap checks (13-gram or longer) between training data and eval data catch the most common contamination sources. Many teams run automated decontamination pipelines before every training run, treating contamination flags as blocking bugs.

Hyperparameters: Learning Rate, Batch Size, Epochs, Warmup

Fine-tuning hyperparameters differ significantly from pre-training hyperparameters because the dataset is small and the model is already close to a good solution. The wrong hyperparameters can cause catastrophic forgetting, overfitting, or failure to learn the target behaviour.

Learning Rate

The learning rate for fine-tuning is typically 5-10x lower than the pre-training learning rate. For full fine-tuning of a 7B model, a learning rate of 1e-5 to 5e-5 is common. For LoRA fine-tuning, use 2e-4 to 5e-4. The 10x difference is because LoRA initialises the A and B matrices to near-zero values, so the effective gradient scale is much smaller and a higher learning rate compensates.

A cosine decay schedule with 5-10% linear warmup is the standard configuration. The warmup phase allows the optimizer (typically AdamW with β1 = 0.9, β2 = 0.95) to accumulate stable gradient statistics before the learning rate reaches its peak. Without warmup, early updates can be large and noisy, destabilising the fine-tuning process.

Batch Size

Effective batch sizes of 32-128 are typical for fine-tuning. Larger batches provide more stable gradient estimates but reduce the number of updates per epoch, which can limit the model's ability to explore the loss landscape. Small batch sizes (4-8) work well with QLoRA where memory is constrained, combined with gradient accumulation to reach the effective batch size.

Epochs

Fine-tuning rarely benefits from more than 3-5 epochs on clean data, and often 1-2 epochs are sufficient. The key signal to monitor is the gap between training loss and evaluation loss. When evaluation loss plateaus while training loss continues to decrease, the model is overfitting. Early stopping based on evaluation loss is more reliable than fixed epoch counts.

Weight Decay

Weight decay of 0.01 to 0.1 helps prevent overfitting by penalising large weight magnitudes. LoRA adapters benefit from lower weight decay (0.01) than full fine-tuning (0.1) because the adapter parameters are already small in magnitude. LoRA weight decay is applied only to the LoRA parameters, not to the base model weights.

Compute Requirements and GPU Memory Estimation

GPU memory is the primary constraint in fine-tuning. Understanding how memory breaks down helps you choose the right method and hardware.

The total memory required for fine-tuning consists of four components:

• Model weights: 2 bytes per parameter in bfloat16 (or 0.5 bytes in 4-bit NF4).
• Gradients: Same size as model weights (2 bytes per parameter in bf16).
• Optimizer states: AdamW stores two momentum terms per parameter, totalling 8 bytes per parameter in bf16 (4 bytes per state in float32).
• Activations: Variable depending on sequence length and batch size. For a 7B model with sequence length 2048 and batch size 4, activations consume roughly 8-12 GB.

A practical reference: fine-tuning Llama 3.2 3B with QLoRA (4-bit base, LoRA adapters in bf16, rank 16) requires ~6 GB of VRAM with batch size 1 and sequence length 2048. The same model in full fine-tuning with bf16 requires ~24 GB — four times more. A 70B model with QLoRA fits on a single A100-80GB or 2x RTX 6000 Ada (48 GB each).

For a deeper treatment of GPU selection and compute planning, see our dedicated guide on parallel processing and GPU architecture.

Evaluation: Perplexity, Downstream Tasks, Human Eval

Evaluation during fine-tuning is more nuanced than during pre-training because the metric that matters is downstream task performance, not language modelling perplexity.

Perplexity

Perplexity measures how well the model predicts a held-out set of tokens. Lower perplexity is better. However, perplexity correlates weakly with task performance after fine-tuning. A model can have low perplexity by simply memorising the training distribution while failing to generalise to new instructions. Use perplexity as a diagnostic tool (sudden spikes indicate training instability) but not as the primary success metric.

Downstream Benchmarks

For general-purpose fine-tuning, standardised benchmarks like MMLU (knowledge), HellaSwag (commonsense reasoning), GSM8K (math), HumanEval (code), and MT-Bench (instruction following) provide reproducible comparisons. However, these benchmarks rarely reflect your specific use case. Building a custom evaluation set of 50-200 examples that represent your production distribution is more valuable than optimising for public benchmark scores.

Human Evaluation

Human evaluation remains the gold standard for assessing output quality, particularly for subjective dimensions like helpfulness, tone, and safety. The standard protocol is pairwise A/B comparison with at least 3 raters per pair, using a standardised rubric. Inter-rater agreement (Krippendorff's alpha or Cohen's kappa) should be tracked to ensure rating reliability. Automated LLM-as-judge evaluation using GPT-4o or Claude 3.5 correlates reasonably with human judgment for well-defined criteria and can supplement human evaluation at lower cost.

For a comprehensive overview of evaluation methodology, see our post on LLM evaluation in production.

When to Fine-Tune vs RAG vs Prompting

Fine-tuning, retrieval-augmented generation (RAG), and prompting are complementary tools, not competing ones. Each has a distinct role, and the best production systems use all three in layers.

Prompting is the right choice when the desired behaviour can be described in a few sentences and the model already has the relevant knowledge. It costs nothing (no training, no infrastructure) and is trivially easy to iterate. Prompting fails when the task is too complex to specify in a prompt, when the model consistently ignores instruction details, or when output format and content must be precisely controlled.

RAG is the right choice when the knowledge required for the task is dynamic or when answers must cite specific sources. RAG retrieves current documents at query time and injects them into the prompt. It handles knowledge updates instantly and provides traceability. We cover this distinction in depth in RAG vs Fine-Tuning: A Decision Framework.

Fine-tuning is the right choice when the model needs to internalise a consistent behaviour, style, or reasoning pattern that cannot be reliably prompted. It is also the right choice when latency or cost constraints make RAG impractical. Fine-tuning and RAG are often combined: fine-tune for behaviour and output structure, then add RAG for factual grounding.

The emerging best practice in production AI systems is a layered architecture: prompt engineering for rapid iteration, RAG for dynamic knowledge, and fine-tuning for consistent behaviour. The layers are applied in this order, with each layer solving the limitations of the previous one. Start with prompting. Add RAG when you need facts. Fine-tune when you need reliable behaviour.

Tools: Axolotl, Unsloth, Hugging Face TRL, Lit-GPT

The fine-tuning ecosystem has matured significantly since 2023. Several tools provide high-level abstractions that handle the complexity of mixed-precision training, gradient checkpointing, LoRA merging, and evaluation, letting practitioners focus on data and hyperparameters.

Hugging Face TRL

The Hugging Face Transformer Reinforcement Learning library (TRL docs) provides SFTTrainer, DPOTrainer, and PPOTrainer — the three main training loops for LLM fine-tuning. TRL integrates seamlessly with the transformers ecosystem, supporting LoRA via PEFT, 4-bit quantisation via bitsandbytes, and flash attention. It is the most widely used fine-tuning library in the open-source ecosystem.

from datasets import load_dataset
from trl import SFTTrainer
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments
)

model = AutoModelForCausalLM.from_pretrained(
    "NousResearch/Llama-3.2-3B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "NousResearch/Llama-3.2-3B-Instruct"
)
tokenizer.pad_token = tokenizer.eos_token

dataset = load_dataset("json", data_files="training_data.jsonl")

training_args = TrainingArguments(
    output_dir="./llama-finetuned",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=3,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
    report_to="wandb"
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    dataset_text_field="text",
    max_seq_length=2048
)

trainer.train()

The SFTTrainerautomatically handles packing sequences, masking non-assistant tokens, and formatting conversations according to the model's chat template. This reduces the typical fine-tuning script from ~200 lines to under 30.

Axolotl

Axolotl (GitHub) is a YAML-driven fine-tuning framework that wraps TRL and PEFT with sensible defaults and advanced features like multi-epoch curriculum learning, dataset mixture ratios, and Flash Attention 3 support. Its main advantage is reproducibility: a full training configuration fits in a single YAML file that can be version-controlled and shared. Axolotl is the standard choice for teams running experiments at scale.

Unsloth

Unsloth (GitHub / docs) optimises the core training kernels to achieve 2x faster training and 50-70% lower memory usage compared to standard TRL training with the same hyperparameters. It works by hand-optimising the forward and backward passes of Llama, Mistral, Qwen, and Gemma architectures. Unsloth integrates as a drop-in replacement for PEFT and TRL — you replace the model loading call with UnslothMistralForCausalLMand the rest of your script stays the same. For teams running many fine-tuning iterations, Unsloth's speedup directly translates to faster experimentation cycles.

Lit-GPT

Lit-GPT (now part of the Lightning AI ecosystem, GitHub) is a lower-level, fully transparent implementation of GPT-style model fine-tuning. Unlike TRL, Lit-GPT does not abstract away the training loop — you see every call, every gradient step, and every checkpoint operation. This makes it ideal for researchers and teams that need full control over the training process. It supports LoRA, QLoRA, and Adapter, as well as full fine-tuning with FSDP and distributed training across multiple nodes.

Common Failure Modes and How to Debug Them

Fine-tuning projects fail in predictable patterns. Recognising these patterns saves weeks of wasted compute and debugging time.

Catastrophic Forgetting

The model loses previously learned capabilities — general knowledge, instruction following, or safety alignment — after fine-tuning on domain data. This happens when the learning rate is too high, the training goes for too many epochs, or the dataset is too narrow. The fix is a lower learning rate, fewer epochs, or a mixture of general instruction data with your domain data (typically 10-30% general data).

Output Collapse

The model produces repetitive, short, or generic outputs — a phenomenon called output collapse or mode collapse. This occurs when the dataset lacks diversity in responses. If every training example ends with a variation of “Let me know if you need anything else,” the model will learn to append that phrase to every response. The fix is to manually review your dataset for repetitive patterns and increase response diversity.

Reward Hacking (RLHF/DPO)

The model learns to maximise the reward signal without actually improving output quality. For example, a reward model that prefers longer responses will produce an LM that writes verbose, repetitive answers with no additional substance. The fix is to carefully audit reward model predictions, use length-normalised rewards, and set the KL penalty β high enough to constrain the policy.

Production Deployment Considerations

Taking a fine-tuned model to production involves more than just running a training script and serving the weights. Several operational considerations determine whether the deployment succeeds.

Adapter Merging vs Dynamic Loading

LoRA adapters can be merged into the base weights for inference, producing a single set of weights that incur zero additional latency. Merged adapters are ideal when you serve one variant of the model. If you serve multiple variants (one per customer, per domain, or per task), dynamic adapter loading without merging avoids storing duplicate base weights. Inference frameworks like vLLM and TensorRT-LLM support adapter switching in under 1 millisecond, making dynamic loading the standard pattern for multi-tenant deployments.

A/B Testing and Gradual Rollout

Never deploy a fine-tuned model directly to 100% of traffic. Run it alongside the previous version with 5-10% of traffic, measure quality metrics (automated eval scores, human ratings, user feedback signals), and ramp up gradually. Many fine-tuning gains that look impressive on static evaluations disappear when measured against live user behaviour.

Monitoring and Regression Detection

Fine-tuned models can regress on dimensions not measured during evaluation. Deploy with automated regression tests that run against a fixed evaluation suite before every release. Track output length distribution, refusal rate, sentiment, and task-specific quality scores over time. A sudden drop in any dimension should trigger an automatic rollback.

Future Directions

The fine-tuning landscape continues to evolve rapidly. Several trends worth watching in 2026 and beyond:

• Multi-task fine-tuning: Instead of training one adapter per task, emerging methods train a single adapter that can handle multiple tasks by conditioning on task identifiers or routing through specialised sub-networks.
• Test-time fine-tuning: Methods like TTT (Test-Time Training) and hyper-network approaches that adapt model behaviour at inference time without a separate training phase, blurring the line between fine-tuning and prompting.
• Automatic data curation: Models themselves are increasingly used to generate, filter, and augment training data. The Llama 3 paper demonstrated that synthetic data generation from stronger models produces high-quality fine-tuning datasets, and this pipeline is becoming a standard tool.
• Mixture-of-experts fine-tuning: As MoE models become the default architecture (DeepSeek V3, Mixtral), fine-tuning methods must account for sparse activation patterns. New techniques like expert-specific LoRA adapters and routing fine-tuning are emerging.

Conclusion

Fine-tuning is the bridge between general-purpose language models and production-ready AI applications. It is not a replacement for good prompting or RAG — it is the third layer in a stack where each layer addresses the limitations of the previous one. Start with prompting. Add RAG for current facts. Fine-tune when you need consistent, reliable behaviour that cannot be achieved through context alone.

The tools have matured to the point where fine-tuning a 7B or 13B model is accessible to any team with a single consumer GPU. QLoRA makes it affordable. TRL and Axolotl make it reproducible. Unsloth makes it fast. The hard part is no longer the training code — it is the data, the evaluation, and the operational discipline to deploy and monitor in production.

If your team is evaluating whether fine-tuning is the right approach for your use case, contact us for a consultation. We have fine-tuned models across legal, medical, financial, and engineering domains and can help you assess the data requirements, cost, and expected ROI before you commit to a training run. For a deeper understanding of the underlying architecture, read our guide on Transformer architecture.