Fine-tuning small models — when to do it and what it actually involves

Fine-tuning a language model has acquired an aura of being either elite research work or pointless busywork, depending on which corner of the internet you read. The truth is more useful: it is a specific tool with specific uses, and most AI engineers will reach for it occasionally rather than constantly.

Here is when to fine-tune, when to absolutely not, and what the work actually looks like when you do.

What fine-tuning is

You take an existing pre-trained model — usually an open-source one like Llama 3, Mistral, or Qwen — and continue training it on a smaller dataset that is specific to your task. The result is a model that does that task better than the base model, often at lower cost and lower latency, because you can deploy a smaller model.

You are not training from scratch. That is research work that takes millions of dollars of compute. Fine-tuning a 7B-parameter model on a few thousand examples can be done in hours on a single rented GPU.

When to fine-tune

Four situations where it is the right move:

• The base model cannot do the task well, even with strong prompting. Try the prompt route first. If after a week of iteration the model still gets the task wrong, fine-tuning becomes worth considering.
• You need consistent structured output. Models can be trained to reliably produce specific JSON shapes, which prompts alone struggle to guarantee.
• Cost or latency demands a smaller model. A fine-tuned 7B model can match a GPT-4-class model at one-fiftieth the cost for one specific task.
• The data is sensitive enough that you need to self-host. Healthcare records, financial data, anything regulated. Fine-tuning lets you run a quality model on your own infrastructure.

When not to fine-tune

Five situations where fine-tuning is the wrong instinct:

• You have not tried prompt engineering. Most "we need to fine-tune" problems disappear with a better prompt.
• Your dataset is small (under 200 examples). Below that threshold the fine-tune will not generalise; you are better off using the data in few-shot prompts.
• Your task requires knowing facts. Fine-tuning teaches behaviour, not knowledge. Use RAG for facts.
• You need the system to handle a wide variety of tasks. Fine-tuning narrows a model down; you lose general capability in exchange for specialisation.
• You do not have a way to evaluate the result. Without an eval you cannot tell if the fine-tune helped, hurt, or did nothing.

What the work involves

Step by step:

• Define the task precisely. "Classify emails as spam, marketing, or important" is fine-tunable. "Write good emails" is not.
• Build the dataset. Pairs of input and the output you want. Quality dwarfs quantity — 500 carefully labelled examples beat 5,000 noisy ones, every time.
• Pick the base model. Llama 3 8B is a strong default; Mistral 7B is a fine alternative; Qwen 7B is competitive and often cheaper to host.
• Pick a fine-tuning technique. LoRA and QLoRA are the standard — they fine-tune a small fraction of the model's parameters, which makes the training cheap and the resulting adapter file small.
• Run the training. Most engineers use Axolotl or Unsloth on a rented GPU (Lambda, RunPod, Vast.ai) — typically a few hours of compute.
• Evaluate against the base model and your target. If the fine-tune is not better than the base on your eval, do not deploy it. Try again with better data.
• Serve it. Tools like vLLM, Ollama, or Text Generation Inference let you serve a fine-tuned model behind an API endpoint compatible with the OpenAI client libraries.

The dataset reality

Almost every fine-tuning failure comes down to the dataset. Specifically:

• Inconsistent labelling — two examples of the same input pattern get different outputs
• Unrealistic inputs — the training data does not match what real users send
• Output drift — the desired output style varies across examples
• Missing edge cases — the model has not seen the inputs that trip it up in production

Plan to spend three days on the dataset for every one day on training. The ratio is uncomfortable until you have lived it.

How much it actually costs

In 2026, a typical first fine-tune:

• GPU rental: $20–$80 for a 7B model on a single A100
• Embedding the dataset and producing training pairs (if you used a larger model to label data): $10–$50
• Evaluation runs: another $5–$20 of model calls

Total: under $200 to test whether fine-tuning helps your specific use case. That is cheap enough that "we will not know unless we try" is a fine answer for non-critical projects.

What this path leads to

Engineers who can fine-tune small models well are paid well, partly because the skill set is harder and partly because the economic value is concrete — every successful fine-tune saves a company real money on inference costs. The work also bleeds into adjacent skills (data curation, evaluation, model serving) that compound over a career.

It is not the right starting path for most beginners. It is a strong second specialisation for AI engineers who have shipped a few RAG or prompt-engineering features and want to go deeper.

Related: what AI Engineering actually is, end-to-end