Fine-Tune Llama With LoRA on a Single Cloud GPU
A tutorial for fine-tuning a Llama model with LoRA on a single cloud GPU, covering memory math, hyperparameters, and merging adapters.
Full fine-tuning of a large language model can demand a cluster of high memory GPUs, which puts it out of reach for most individuals and small teams. Low rank adaptation, known as LoRA, changes the economics entirely. By training only a small set of added weights while freezing the original model, LoRA lets you adapt a Llama model on a single rented cloud GPU. This tutorial walks through the full process, from picking an instance to merging and serving your adapter.
Why LoRA fits a single GPU
A standard fine-tune updates every weight in the model, which means you must hold the weights, their gradients, and optimizer state in GPU memory at once. That triples or quadruples the memory footprint of the model. LoRA instead freezes the base weights and inserts small trainable matrices into chosen layers. You only compute gradients and optimizer state for those tiny matrices, so the heavy memory cost of full training disappears.
Pair LoRA with a quantized base model, an approach often called QLoRA, and the base weights themselves shrink in memory. The combination means a model that would never fit for full training can be adapted comfortably on a single mid range or high end GPU.
Pick the right cloud GPU
Your instance choice depends mostly on the base model size and your context length. Use these rough guidelines, then verify with a short test run before committing to a long job.
| Base model size | Approach | GPU memory guidance |
|---|---|---|
| Small (a few billion params) | LoRA or QLoRA | A single mid range GPU is usually enough |
| Mid (around 7 to 13 billion) | QLoRA recommended | One high memory consumer or data center GPU |
| Large (30 billion and up) | QLoRA with care | A single large data center GPU, watch context length |
Longer context and larger batch sizes both raise memory use sharply, so if you are near the edge, shorten sequences or lower the batch before reaching for a bigger instance. On DeployCue you can compare hourly rates across providers to find the cheapest instance that clears your memory requirement.
Prepare your dataset
The quality of a fine-tune is mostly the quality of the data. Format your examples to match how the model will be used, typically as instruction and response pairs wrapped in the chat template the base model expects. A few principles help.
- Use a consistent prompt template that matches the model's chat format.
- Prefer a few thousand clean, varied examples over a huge noisy pile.
- Hold out a small validation set so you can watch for overfitting.
- Remove duplicates and near duplicates that would bias the model.
Run the fine-tune
With data ready, the training loop itself is straightforward. The flow looks like this.
- Load the base model, optionally quantized to four bit, and freeze its weights.
- Attach LoRA adapters to the attention and projection layers.
- Set the rank, the scaling factor, and which layers get adapters.
- Train for a small number of epochs while logging training and validation loss.
- Save the adapter weights, which are small enough to store and share easily.
Key hyperparameters to tune are the LoRA rank, which controls how much capacity the adapter has, and the learning rate, which is typically higher than full fine-tuning because you are training so few parameters. Start with a modest rank and raise it only if the model underfits. Watch validation loss and stop when it stops improving, since extra epochs invite overfitting.
Merge, evaluate, and serve
After training you have a base model plus a small adapter. You can serve them together, loading the adapter on top of the base at inference time, which keeps the adapter swappable. Or you can merge the adapter into the base weights to produce a single standalone model, which simplifies deployment at the cost of flexibility.
- Keep the adapter separate when you want to swap behaviors or stack adapters.
- Merge into the base when you want one clean artifact to deploy and serve.
Evaluate before you ship. Run your validation prompts, compare outputs against the base model, and check that the fine-tune improved the target behavior without degrading general ability. A quick side by side review catches regressions that loss numbers alone miss.
Keep costs in check
Single GPU LoRA jobs are cheap compared to full training, but a few habits keep the bill low. Run a short trial of a few hundred steps to confirm everything works before launching the full job. Use spot or interruptible instances with checkpointing if your provider offers them, since a fine-tune that checkpoints can resume after an interruption. Shut the instance down the moment training finishes, because an idle GPU still bills by the hour.
Tune the LoRA hyperparameters that matter
A handful of LoRA settings drive most of the outcome, and understanding them prevents a lot of trial and error. The rank sets how much capacity the adapter has: a low rank trains fewer parameters and resists overfitting on small datasets, while a higher rank gives the model more room to learn complex behavior at the cost of more memory and overfitting risk. The scaling factor controls how strongly the adapter influences the base model and usually moves together with the rank.
You also choose which layers receive adapters. Attention projection layers are the common targets, and adding adapters to more layers increases capacity. Start conservative with a modest rank on the attention layers, then expand only if validation loss shows the model is underfitting. Because you train so few parameters, you can afford a higher learning rate than full fine-tuning, but watch for instability and back off if loss spikes.
Avoid catastrophic forgetting
A fine-tune that nails your target task but forgets how to do everything else is a common failure. LoRA's frozen base helps here, since the original weights stay intact, but aggressive training on a narrow dataset can still skew behavior. Guard against it by keeping some general examples in your training mix and by stopping early rather than training to a memorized fit.
Always evaluate on prompts outside your fine-tuning domain as well as inside it. If the model has gotten markedly worse at general instructions, reduce the number of epochs, lower the rank, or diversify the data. The goal is a model that gained your target skill without losing its broad competence, and only out of domain evaluation reveals whether you struck that balance.
Conclusion
LoRA brings Llama fine-tuning within reach of a single rented cloud GPU. Freeze the base, train small adapters, lean on quantization to fit memory, and feed the model clean instruction data. With a sensible instance, a short trial run, and disciplined shutdown, you can ship a custom Llama variant for a fraction of the cost and complexity of full fine-tuning.