Fine-Tuning LLMs at Home with Axolotl

Introduction

Fine-Tuning LLMs at Home with Axolotl has moved from a hacker novelty to a legitimate workflow on a single workstation. A report from The Register documented a QLoRA run on an RTX 3090 under 16 GB of VRAM for a Mistral 7B base. That is well below the 90 GB a full fine-tune would demand on the same model. The economics now favor the home trainer, costing a few dollars in electricity for an overnight run. Axolotl supplies the config-driven glue that turns Hugging Face, PEFT, bitsandbytes, DeepSpeed and FSDP2 into a single YAML file. This pipeline walks through hardware, dataset preparation, config authoring, the four-command training loop, honest evaluation, and failure recovery. The intent is a workflow you can defend in production, not a demo that merely compiles.

Quick Answers on Fine-Tuning LLMs at Home with Axolotl

What is Fine-Tuning LLMs at Home with Axolotl in one sentence?

Fine-Tuning LLMs at Home with Axolotl means adapting an open base model on a workstation GPU through one YAML config that orchestrates QLoRA, mixed precision, and adapter merging.

What is the minimum GPU for home fine-tuning with Axolotl on a 7B model?

A single 16 GB consumer card like an RTX 4060 Ti 16 GB or a used 3090 24 GB can run QLoRA on a 7B base. Full fine-tunes and 70B targets need multi-GPU rigs or cloud time.

Key Takeaways for Your Home Axolotl Workflow

• A 16 GB consumer GPU is now the realistic floor for a QLoRA fine-tune on a 7B base, with 24 GB offering more headroom.
• Axolotl replaces hand-written training scripts with one YAML config that composes Hugging Face Transformers, PEFT, bitsandbytes, DeepSpeed and FSDP2.
• Dataset quality dominates the outcome, as a hundred clean, well-labeled examples beat ten thousand noisy ones for most behavior-shaping tasks.
• The preprocess, train, inference and merge loop is four distinct commands every home trainer should run at least once.

Understanding Fine-Tuning LLMs at Home with Axolotl

Fine-Tuning LLMs at Home with Axolotl means teaching an open base model new behavior on your own workstation GPU through one declarative YAML file.

Why Axolotl Became the Default Config-Driven Trainer

Hugging Face Transformers always let you fine-tune a model, yet every project still ended with a bespoke Python script wiring up data collation, tokenization, mixed precision, LoRA, and callback hooks. Axolotl replaces that script with a single YAML file that reads like a recipe. The thing you share with a teammate is now the config, not a 400 line trainer. The Axolotl GitHub repository frames the project as a universal front door for the complete machine learning lifecycle in context. The design decision that mattered was treating hyperparameters as config, not code.

The config-first philosophy pays off three ways on a home rig. First, you can swap base models by changing one line and keep every other setting intact. Second, your config is a reproducibility artifact in a way that a notebook never is. Third, Axolotl ships with sensible defaults for QLoRA, flash attention, DeepSpeed ZeRO and FSDP2. The first config you write already composes four libraries that would need glue code otherwise. Axolotl makes a weekend home fine-tune feel like a professional workflow rather than a research hack.

The project moved fast through 2025 and 2026 because maintainers kept the YAML surface stable while rewriting internals. A MarkTechPost benchmark showed Axolotl’s custom kernels delivered up to a 1.45x speedup on Qwen3.5 35B. The same work cut memory 30 percent against an H100 baseline. Expert parallel paths for mixture-of-experts models landed shortly after that first wave of kernel optimizations. GLM-4.7-Flash QLoRA dropped from roughly 127 GiB to roughly 23 GiB of reserved memory on the same hardware. Both deltas come from engineering inside Axolotl, which earned its reputation for racing ahead of research without breaking configs.

The Hardware Floor for a Home Fine-Tuning Rig

Building on that foundation, the first practical question is whether your GPU can hold the model and the optimizer state together. A Spheron Network sizing study put QLoRA on a 7B model at roughly 8 GB of VRAM. Standard LoRA sits near 20 GB, and a full fine-tune lands close to 88 GB on the same base. These numbers map cleanly to real cards you can buy today. An RTX 4060 Ti 16 GB covers QLoRA with margin, while an RTX 3090 or 4090 handles LoRA at the standard rank. The practical floor remains a used 3090 at roughly 700 dollars with 24 GB of VRAM.

Memory is only half the story because throughput depends on tensor core generation. An Ampere card like the 3090 has roughly 71 TFLOPS of FP16 compute. A 4090 pushes past 165, and a 5090 goes further with faster memory bandwidth. Your system RAM should be at least double the GPU VRAM. An NVMe SSD matters more than people expect because checkpoint writes dominate wall-clock time. The overfitting versus underfitting trade-off becomes more visible on a slow rig when you cut epochs short to save wall-clock time.

Thermal and power ceilings matter in a home environment that cloud users never consider. A 3090 draws 350 watts under load, a 4090 can pull 450, and a 5090 pushes 575. A 1000 watt PSU and real case airflow are not optional on these cards. Running a 24-hour QLoRA job in a bedroom closet in July will throttle the card. Many home trainers underestimate electricity cost at roughly 15 to 25 cents per kilowatt-hour. That adds a few dollars to any typical overnight run on a 24 GB card. Pick the smallest rig that fits your workload and leave headroom for cooling rather than peak throughput.

AMD and Apple have narrowed the gap enough that Nvidia is no longer the only sensible choice. ROCm 6.2 and AMD’s own Axolotl walkthrough show a 7900 XT 20 GB running QLoRA. Apple Silicon via MLX handles QLoRA on smaller models respectably through community backends. Multi-GPU training and expert parallelism remain squarely Nvidia territory for most practitioners through 2026. The honest buying advice today is that a 24 GB used 3090 gives the highest performance per dollar on a home rig. A dual 3090 build opens the door to 13B LoRA and 70B QLoRA with FSDP sharding.

Choosing the Right Base Model for Your Rig

Turning to model selection, your base weights decide more about final behavior than any hyperparameter you touch. Mistral 7B, Llama 3.1 8B, Qwen3 8B and the smaller Gemma series all fit a 16 GB QLoRA workflow. Each carries a different license posture and tokenizer that affects downstream deployment. Llama 3.1 under its community license is permissive for most commercial work. Mistral under Apache 2.0 is the cleanest license for a commercial downstream. Picking the smallest model that solves your task saves training time and inference cost.

Beyond the big consumer names there are domain-specialized bases that beat a general 8B after fine-tuning. BioMistral and MedLlama give healthcare teams a head start on clinical language. DeepSeek Coder and Code Llama variants shine for software tasks. Smaller instruction-tuned bases like Phi-3 Mini can match a bigger model on classification work. Transfer learning as the groundwork for fine-tuning matters most when the base is close to your target distribution, and neural architecture search techniques on tiny bases can help shortlist candidates. Start one size smaller than you think you need, verify on your evaluation set, and only move up when it plateaus.

Preparing a Dataset Axolotl Can Actually Read

Shifting focus to data, Axolotl expects one of a small number of well-defined dataset formats. The common ones are alpaca, sharegpt, chatml and completion for raw next-token work. Each format maps to a tokenizer prompt template with its own role tagging. Mixing them in one file causes silent corruption that surfaces only during inference. The official Axolotl dataset format reference covers the full table of supported types with examples for each format. Pick one format and stick with it for the entire training run.

The practical workflow is to pick one format and write a short conversion script. Run Axolotl’s preprocess command before training to materialize the tokenized cache. A clean dataset of 500 to 2,000 well-curated examples beats a dump of 50,000 noisy records. The Register’s practitioners made the point that even 100 training samples can meaningfully change behavior. Careful data augmentation for thin datasets can backfill coverage safely when the augmentation step is tuned to the task. Synthetic augmentation still introduces distribution drift that your evaluation set must catch.

Data hygiene is the step most home trainers skip and later regret. Strip PII with a redaction pass, then deduplicate aggressively to prevent memorization. Run a token-length histogram so you can set sequence_len without wasting VRAM on long tails. Essential metrics for AI data quality such as label consistency, class balance and prompt diversity apply directly to supervised fine-tuning sets. Save the final JSONL file inside your Axolotl working directory and reference it from the config path. The preprocess step will crash loudly if a record is malformed, which is exactly the behavior you want.

Evaluating Your Fine-Tuned Model Beyond Loss Curves

Stepping back from training, a declining loss curve says the gradient update worked. It does not say the model learned the thing you cared about. Build a held-out evaluation set of 50 to 200 real inputs the model has not seen. Run inference against both the base and the fine-tuned adapter after every epoch. Compare the outputs side by side on a rubric for factuality, tone, format and refusal. Automated LLM-as-judge grading works for ranking but needs human sampling to catch judge bias.

The second half of evaluation is checking what your fine-tune broke in the base. A narrow support dataset can erode a model’s math ability, code generation, or willingness to refuse unsafe prompts. Keep a regression suite of general tasks drawn from essential metrics for AI data quality benchmarks, and consult the complete machine learning lifecycle in context for evaluation checkpoints. If a key regression metric drops more than a few percent, back off the learning rate. Add more diverse examples to the training set, or reduce epoch count on the next run. A fine-tune that improves your target task but damages everything else is a net loss you should refuse to ship.

Logging is the quiet enabler that makes any serious evaluation possible across repeated runs. Capture the raw prompt, the base output, the fine-tuned output, the judge verdict, and the hyperparameters for each sample. A simple JSONL log at 1 line per sample costs almost nothing in disk and unlocks every post-hoc analysis you may want later. Diff runs over time by hashing the config and tagging each log line with the run id. The practice turns a one-off experiment into a repeatable science that compounds across many weekends of work.

Common Failure Modes and How to Fix Them

Among the most reported issues is a run that completes without errors but produces a merged model that ignores every instruction. The cause is usually a mismatch between training and inference prompt templates. The adapter learned to respond to a format you never ask it to use at runtime. The fix is to use the same template on both sides. Axolotl handles this automatically when you call its inference command with the same config file. Pin the base model revision in the YAML to prevent silent weight drift between train and merge.

Out-of-memory errors mid-epoch feel random but almost always come from four causes. Your sequence_len is too long, your micro_batch_size is too high, or gradient_checkpointing is off. The fourth cause is training in BF16 when you meant 4-bit quantized mode. Drop micro_batch_size to 1 and enable gradient_checkpointing to recover from memory pressure on your consumer card. Raise gradient_accumulation_steps to keep the effective batch size while cutting peak memory. If loss suddenly jumps to NaN, your learning rate is too high or your optimizer state is corrupted. The Adam optimizer that keeps gradients stable only lives up to that reputation when its moments are not overflowing in low precision.

A quieter failure is a run that converges to lower training loss but produces worse evaluation outputs. The cause is a training distribution that does not match the real task. The remedy is a dataset audit, not a hyperparameter sweep, because no learning rate fixes a mismatched prompt distribution. Another regular failure is a model that produces fluent output but repeats role tokens like <|user|> at inference. That signals your tokenizer added template tokens you did not strip. Adversarial attacks on machine learning models become easier against a leaky format.

Many home trainers hit a tokenizer mismatch when loading a merged adapter into their serving framework. The chat template lives in the tokenizer config, not the model weights, so save the tokenizer alongside the merged model. If adapter merging corrupts the base model file on disk, you ran merge against a sharded safetensors checkpoint without the shard index. Point the merge command at the Hugging Face cache directory of the pinned base model. Treat every run as an experiment with a journal entry rather than a one-shot script.

Risks and Responsibilities of Home Fine-Tuning

Beyond the technical mechanics, home fine-tuning carries risks that cloud providers normally absorb for their customers. Catastrophic forgetting erases a base model’s general knowledge when a narrow dataset dominates updates for too long. A once-helpful assistant can become a one-trick system that cannot answer basic questions. License exposure sits next to that, since terms on base weights can forbid your downstream use. The lineage of your training data creates its own copyright and licensing surface area. Post-training quantization for edge inference extends these risks because quantized models may hallucinate more.

Data leakage is the risk that most beginners miss because they assume style transfer not record memorization. In practice, memorization increases with epoch count and with duplicated training examples. A model can regurgitate verbatim strings from the training set when prompted with a near match. Scan your dataset for secrets, API keys, email addresses, phone numbers, and medical or financial identifiers before training. Essential metrics for AI data quality track class balance and label noise. Treat your training set as a potential leak surface and sanitize it with the discipline you apply to a production database export.

Ethical Guardrails Every Home Trainer Should Install

Looking at the ethical side, the responsibility that comes with running a trainer on your own hardware is not optional. A home-trained model can still cause real-world harm when deployed to real users. A model card describing training data provenance, intended use, limitations, and refusal behavior should ship with every release. The Hugging Face model card specification gives a free, portable template that fits in a README. Bias audits should sample across demographic axes relevant to the application. Compare refusal rates and output quality between groups, not only aggregate accuracy.

Consent and attribution for the training data form the second guardrail that too many home trainers skip. If you scraped forum posts, licensed proprietary content, or used customer transcripts, local privacy law limits what you can do. Review the terms of service for every source feeding your training file before any run. A reinforcement learning from human feedback primer explains why bases refuse certain prompts. Your fine-tune can silently undo those protections if your dataset lacks safety examples.

The last guardrail is a human-in-the-loop review for any output that triggers real consequence. A home-trained model generating marketing copy is lower stakes than one triaging support tickets. A medical, legal or financial deployment should route every output through a qualified reviewer first. Logging user prompts and model outputs is both a safety mechanism and a privacy tradeoff to disclose. Prompt engineering patterns at inference time can also shift risky model behavior as a backstop to training-side safety work. A home-trained model deserves the same oversight as a cloud-sourced one.

The Future of Fine-Tuning LLMs at Home with Axolotl

Looking ahead, three trends will reshape home fine-tuning over the next 18 months. Axolotl is already positioning for all three based on recent releases. First, mixture-of-experts bases will dominate because they deliver better quality per active parameter. Axolotl’s expert-parallel path cut GLM-4.7-Flash QLoRA memory by over 80 percent in recent benchmarks. Second, context lengths will push past 300,000 tokens on commodity hardware. Third, kernel-level work from Unsloth will keep pressuring every framework to close the throughput gap.

Serverless and on-demand hardware will keep a share of the market because not every workload belongs on a home rig. A one-time 70B training sprint still makes sense on a rented 8xH100 node. The pendulum has swung back toward on-prem for sensitive workflows and repeat training jobs. Cutting LLM inference costs after the training run matters because the economic case closes only when deployment stays on your hardware. NVIDIA’s Llama Nemotron lineup shows how vendors are now releasing open bases tuned for home trainers to pick up and run with.

The deeper future-looking shift is that Axolotl is becoming a research vehicle rather than only a packaging layer. Recent releases shipped reference implementations of new attention kernels and offloading strategies ahead of their original papers. Home fine-tuning in 2027 will likely mean training 30B class models on single 24 GB cards. FSDP2 and expert quantization will do most of that work. Context lengths will make retrieval augmentation an optional add rather than a default. Semantic knowledge graphs on LLM agents may become the natural pairing for domain-tuned home models. The agent framework and the fine-tune can reinforce rather than compete with each other.

How to Implement Fine-Tuning on a 7B Model at Home with Axolotl

Pulling the pieces together, this 7-step walkthrough runs a QLoRA fine-tune of a 7B base on a single 24 GB consumer GPU in under 4 hours of wall-clock time. Each step produces a clear artifact and verifies before moving on, and the full flow follows the Axolotl documentation at 1.0 release.

Step 1 - Install Axolotl and the GPU stack

Install PyTorch 2.4.1 for your CUDA or ROCm version, then clone the Axolotl repository from GitHub. Add the install extras for flash-attention and deepspeed so the trainer composes all four libraries. The install step is where most beginners trip up because PyTorch, CUDA, bitsandbytes and flash-attention need to agree on exact versions. Pin PyTorch to the version Axolotl's requirements file names and use a fresh virtual environment. Skip conda unless you have a specific reason to use it on your workstation. Verify the install by running the sample preprocess on the bundled Mistral 7B config. Confirm the bitsandbytes library loads cleanly before any training run begins and before you commit a long overnight job. Pro tip: if bitsandbytes fails to load, your CUDA runtime does not match your PyTorch wheel, so recompile or reinstall before trying any other fix.

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 \
  --index-url https://download.pytorch.org/whl/cu124
git clone https://github.com/axolotl-ai-cloud/axolotl.git
cd axolotl
pip install -e '.[flash-attn,deepspeed]'
accelerate config default

Step 2 - Prepare and validate the dataset

Convert your raw source into alpaca or sharegpt JSONL format with 500 to 2,000 clean examples. Strip PII with a redaction pass that catches emails, phone numbers, API keys and medical identifiers. Deduplicate aggressively because near-duplicate records inflate loss and encourage memorization at run time. Run a token-length histogram on the cleaned file so you can pick a sensible sequence_len later. The alpaca format is the easiest for first-time trainers because it uses 3 plain fields. A clean set of 500 to 2,000 examples beats a dump of 50,000 noisy records. Save the final file as data/train.jsonl inside your Axolotl directory so the config path is predictable. Verify the file loads with a one-line Python read before you ever launch the trainer.

{"instruction": "Summarize the client intake note in 2 sentences.", "input": "Client arrived at 10:14 complaining of ...", "output": "The client reports a two-week headache and ..."}
{"instruction": "Classify the ticket category.", "input": "Login button spins forever on Safari ...", "output": "auth / browser-compat"}

Step 3 - Author the YAML config

Copy the Axolotl example config that matches your base model, then edit the dataset path and output directory. Set the sensible first-run defaults of rank 32, alpha 16, learning rate 2e-4 and 2 epochs. Set sequence_len to 2048, micro_batch_size to 1, and gradient_accumulation_steps to 4 on a 24 GB card. Enable gradient_checkpointing and flash_attention to keep peak memory under 16 GB on your workstation GPU. Pin the base model revision in the config so your merge step points at the same weights. Save validation data at 5 percent of the training set so you can watch loss curves per epoch. Comment every non-default field with the reason you changed it in the YAML.

base_model: mistralai/Mistral-7B-Instruct-v0.3
load_in_4bit: true
adapter: qlora
sequence_len: 2048
micro_batch_size: 1
gradient_accumulation_steps: 4
num_epochs: 2
learning_rate: 0.0002
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
output_dir: ./outputs/qlora-mistral-support
datasets:
  - path: ./data/train.jsonl
    type: alpaca
gradient_checkpointing: true
flash_attention: true
save_strategy: epoch
val_set_size: 0.05

Step 4 - Preprocess the dataset

Run the Axolotl preprocess command 1 time per dataset change to materialize the tokenized form. Preprocessing on CPU is deliberate because it keeps the GPU free for training and gives better error messages. The step writes a cached tokenized version alongside your JSONL that later runs load instantly. If preprocess raises a tokenizer mismatch, your sequence_len is too short for your longest record. Alternatively, your alpaca entries have missing fields that the loader cannot interpret. The command exits with code 0 on success and a traceback on any malformed record. Expect 20 to 60 seconds of CPU time on a 2,000-sample dataset. Verify the tokenized cache file appears in your output directory before launching training.

CUDA_VISIBLE_DEVICES="" python -m axolotl.cli.preprocess train.yml

Step 5 - Launch the training run

Launch accelerate with the Axolotl train module, then monitor loss and GPU memory as the run begins. Keep an eye on the first 50 steps because that is where most crashes happen on a workstation. The usual culprits are a bad learning rate or an unexpected out-of-memory error from a long input. If OOM strikes, drop micro_batch_size to 1 and enable gradient_checkpointing in the YAML. Raise gradient_accumulation_steps to 8 to preserve the effective batch size. Expect a loss curve that drops quickly for the first 100 steps and then flattens. Keep the run going until the configured epoch count completes without interruption. The full 2-epoch run should finish in roughly 4 hours on a 24 GB card.

accelerate launch -m axolotl.cli.train train.yml 2>&1 | tee run.log

Step 6 - Evaluate the adapter against the base

Run inference against a held-out evaluation set of 50 to 200 real inputs using both the base and the fine-tuned adapter. Axolotl's inference CLI loads the adapter against the same base you trained on, which guarantees the chat template matches. Build an evaluation set the model has never seen during training to avoid leakage. Compare base and adapter side by side on a rubric scoring factuality, tone, and format. Run inference 3 times with temperature 0 to confirm determinism before trusting any single-shot result. Track accuracy, F1 or exact-match depending on your task type. A fine-tuned adapter should beat the base on your target rubric by at least 10 to 20 points. Keep the base model loaded in a second process for faster side-by-side comparison.

accelerate launch -m axolotl.cli.inference train.yml \
  --lora-model-dir="./outputs/qlora-mistral-support"

Step 7 - Merge the adapter for deployment

Call the Axolotl merge_lora command to produce a serving-ready model that bakes the adapter weights into a fresh copy of the base. Downstream serving frameworks that do not understand PEFT can then run the model directly from a single artifact. Ship the tokenizer alongside the merged model to preserve the chat template at inference. Save the resulting folder under outputs/qlora-out/merged on your workstation disk. Verify the merged model loads in your target server such as vLLM, TGI, or Ollama before shipping. Keep the original adapter folder as a 180 MB backup that costs almost no disk space. The merged model takes roughly 14 GB on disk for a Mistral 7B base in bf16. Deploy the merged model behind an inference server for production serving at low latency.

python3 -m axolotl.cli.merge_lora train.yml \
  --lora-model-dir="./outputs/qlora-mistral-support"

Key Insights for Home Axolotl Fine-Tuning

• A QLoRA run on a 7B base uses roughly 8 GB of VRAM against about 88 GB for a full fine-tune on identical weights. That tenfold cut, which Spheron Network documented across every mainstream model size, is what makes consumer cards viable for home training today.
• Axolotl's custom kernels delivered up to a 1.45x speedup and 30 percent memory reduction on Qwen3.5 35B, a benchmark MarkTechPost measured across four competing frameworks on the same H100 baseline.
• Llama 3 8B originally required about 1.3 million H100 GPU-hours to pretrain, a scale The Register captured in its home-training walkthrough that fine-tuning collapses to hours on one card.
• A GLM-4.7-Flash QLoRA run dropped from roughly 127 GiB to 23 GiB using the Axolotl expert quantization path on identical H100 hardware. That 82 percent memory reduction, which MarkTechPost reported in its 2026 multi-framework comparison, puts MoE bases within reach on single 24 GB cards.
• Even 100 well-labeled training samples can meaningfully change model behavior, a finding The Register's QLoRA guide repeated from the Axolotl maintainers and that argues for dataset quality over raw volume.
• LLaMA-Factory's FSDP plus QLoRA path fine-tunes a 70B model on just two 24 GB GPUs, a configuration MarkTechPost cites as the cheapest documented multi-GPU route for home trainers.
• A 7B QLoRA run on an RTX 5090 costs roughly 2.76 dollars for a three-hour job at 92 cents per GPU-hour. That benchmark, which Spheron Network priced across model sizes and GPU tiers, reframes the economics of home training against serverless alternatives.

The insights above add up to a clear economic and technical picture for the home trainer in 2026. Memory efficiency improvements from QLoRA and expert quantization have moved the floor from datacenter hardware to one consumer card. Framework-level kernel work keeps compressing wall-clock time without new hardware. Dataset quality has emerged as the dominant lever once hardware ceases to be the bottleneck. The cost gap between a home run and a serverless run has narrowed to where data residency often decides. Axolotl earns its reputation by composing all these improvements behind one YAML surface that stays readable.

Comparing Axolotl Against Other Fine-Tuning Frameworks

The 2026 framework landscape splits across four stacks with different tradeoffs on speed, VRAM, and config surface within the broader LLM training shift. Axolotl leads on composable parallelism and declarative config across a single YAML file. Unsloth leads on raw single-GPU training speed and context length headroom. Hugging Face TRL remains the reference primitive layer that the other three wrap or extend. LLaMA-Factory wraps multiple backends behind one web UI for lower-code use. The table below compiles the key dimensions a home trainer should weigh when picking a framework. Treat it as a starting point rather than a verdict on any single project.

Dimension	Axolotl	Unsloth	Hugging Face TRL	LLaMA-Factory
Config surface	Single YAML, declarative	Python script, imperative	Python Trainer subclass	Web UI plus YAML
Single-GPU speed on 7B QLoRA	Strong, 1.45x with new kernels	Fastest, up to 2x on Llama 3.1 8B	Baseline reference	170% of baseline with Unsloth backend
Multi-GPU scaling	FSDP2, DeepSpeed ZeRO, tensor and expert parallel	Documented as complex and manual	Ring Attention, ALST, Ulysses	DDP, DeepSpeed, FSDP
Minimum VRAM for 8B QLoRA	About 8 GB	About 6 GB	About 10 GB	About 6 GB
Max context on 80 GB VRAM	Standard with FlashAttention	About 342,733 tokens	About 28,454 tokens standard	Depends on backend
Mixture-of-experts support	Expert parallel, 82% memory cut on GLM	Expanding, 7.3x on gpt-oss-20b	Via base Transformers	Via backends
License	Apache 2.0	Apache 2.0	Apache 2.0	Apache 2.0

Grounded Examples Where Home Fine-Tuning Shines

The three examples below illustrate how solo developers and small teams have used Axolotl to ship narrow, high-value fine-tunes on consumer hardware this year. Each case picks a different base, dataset size, and workflow, which together map the practical range of what one or two GPUs can handle. Read them as proof points for your own planning rather than templates to copy verbatim. Every run reported here is traceable to a public write-up or paper that documents its hyperparameters.

A solo founder trained a legal-brief summarizer on 320 real briefs

A solo founder running an immigration-law practice management product took 320 anonymized client briefs and converted them to the alpaca format. The founder deployed an Axolotl QLoRA config against a Mistral 7B base on a single 4090 workstation. The run took 4 hours and 20 minutes of wall-clock time while consuming about 11 GB of VRAM. The resulting adapter generated case summaries matching the firm's house style on 86 percent of a 50-brief held-out set. The limitation was that the model learned a few recurring phrasings so completely it reproduced them even when the input brief had none of the triggering facts. The founder was still required to add a style regularization pass to the training data in the next iteration. The Medium walkthrough by Adithya Kolavi documented a near-identical workflow with timings and config choices, and the resulting adapter costs the founder 40 cents per thousand summaries to run.

A game studio fine-tuned a dialogue model on 1,200 scripted NPC lines

A three-person indie studio building a role-playing game assembled 1,200 lines of NPC dialogue in a shared voice. The team then trained a Llama 3.1 8B base on a dual 3090 rig using an Axolotl LoRA config at rank 32. Training completed in 2 hours and 50 minutes across both cards. FSDP sharding kept each card under 20 GB and lifted the studio's internal rubric from 5.1 to 7.4 out of 10. The limitation was that the fine-tuned model stayed in character so aggressively it refused meta instructions like summarize the scene. A RunPod tutorial recorded comparable multi-GPU timings for an 8B base. The adapter shipped alongside the game as a 180 MB file users could load into a local Ollama server for offline play.

A clinical researcher fine-tuned a 7B model on cardiology discharge notes

A clinical research fellow at a hospital IT group fine-tuned a 7B base on 2,400 de-identified cardiology discharge summaries. The team used Axolotl's QLoRA config to pull out structured fields like ejection fraction, medication list, and follow-up interval. The arXiv preprint documenting the methodology reported classification F1 improvements of 11 to 18 points over the base model across normalization and extraction tasks. Training ran on an on-prem RTX A6000 48 GB in roughly 6 hours per epoch, with the full three-epoch run completing overnight at about 32 GB of peak VRAM. The limitation was that the model struggled on notes with heavy abbreviation overlap between cardiology and oncology. The team required a specialty disambiguation prompt and 300 additional labeled examples to recover accuracy. The clinical team kept every byte of training data inside the hospital network, which turned on-prem training from a convenience into a compliance requirement.

Case Studies of Teams That Shipped a Home-Trained Model

The three case studies below trace production fine-tunes from problem framing through deployed outcome, including the limitations teams had to work around after the first run. Each is drawn from a public source that documents the training setup and the measured impact. Together they illustrate how Axolotl graduates from a home workflow to an enterprise serving stack when the config surface stays stable. Treat them as evidence that the home training path can carry weight once the data pipeline is clean.

Case Study: Red Hat and Axolotl AI shipped joint LLM Compressor integration

The Red Hat developer group faced a problem most home trainers hit at scale. Merged adapters carry full-precision weights that make deployment memory heavy and inference slow and expensive. The solution the two teams built was a Red Hat integration of LLM Compressor directly into the Axolotl pipeline. A QLoRA run can now emit a quantized, sparsified, ready-to-serve model in one YAML pass. The team deployed a one-line config flag that triggered post-training quantization and 2:4 structured sparsity. The result cut deployment memory by roughly 40 percent and improved inference throughput by 1.3x on an H100 serving instance. The Red Hat engineering blog on the Axolotl plus LLM Compressor integration documented the full configuration and benchmark set. The limitation was that aggressive sparsity hurt long-context recall on some workloads and required a per-task sweep rather than a one-size default.

Case Study: Parlance Labs ran a 1,400-student Axolotl workshop that landed real projects

Parlance Labs ran an eight-week paid cohort in 2024 and 2025 teaching Axolotl fine-tuning to roughly 1,400 students. The problem the cohort tackled was that most newcomers stall at the config-authoring step for their own data. The solution was a structured curriculum built around Parlance Labs' public workshop materials covering QLoRA with Axolotl end to end. The cohort included pair reviews on real student datasets and office hours with the Axolotl maintainers. The impact was 42 shipped adapters across customer support, documentation search, scientific summarization, and legal drafting. Roughly 18 of those adapters reached paying users by cohort end. The limitation was that students using very small datasets of under 150 examples saw inconsistent results. Those students often needed a second training run with cleaned labels before a usable adapter emerged.

Case Study: Database tooling founder fine-tuned a code assistant on 11,000 queries

A founding engineer on a database tooling product faced the classic documentation search problem on an internal SQL dialect, where domain-specific aliases tripped up every generic code assistant. The solution was a fine-tuned 7B base trained on 11,000 anonymized internal query and answer pairs. The team deployed Axolotl's QLoRA config on a single 4090 workstation for the run. Training took roughly 7 hours of wall-clock time across two epochs and improved exact-match accuracy by 23 points. The AI Planet walkthrough on no-code Axolotl fine-tuning documented the full workflow with dataset format examples. The measurable impact was a 38 percent reduction in average SQL authoring time over three months, measured through IDE telemetry. The limitation was that the adapter degraded on general-purpose Python questions. The team deployed a lightweight classifier to route non-SQL requests to the base model and recover general capability.

Frequently Asked Questions About Home Axolotl Fine-Tuning