Post-training Quantization for Edge AI

Introduction

Post-training quantization for edge AI has become the quiet engine behind the recent wave of on-device intelligence. It shrinks large neural networks so they run on phones, sensors, and cheap boards without a data center behind them. Compressing a model from 16-bit floats to 4-bit integers can cut its size by four to sixteen times. That reduction maps directly to less memory, lower power draw, and faster local response on constrained hardware. Small language models now deliver roughly 80 to 90 percent of large-model capability while running entirely offline, a point made in a systematic evaluation of on-device LLMs. Teams reach for this technique because retraining a model from scratch is expensive, slow, and often unnecessary. Quantization instead reuses an already trained network and rounds its numbers down to a smaller format. This guide explains how the method works, what accuracy it costs, and how to ship it in production.

Quick Answers on Post-training Quantization for Edge AI

What is post-training quantization for edge AI?

It is a compression method that converts a trained model’s weights to lower-precision integers, like INT8 or INT4, so it runs on small edge devices without retraining.

How much accuracy does it lose?

Well-tuned INT8 quantization usually loses under one percent, while INT4 can hold above 98 percent accuracy when methods like GPTQ or AWQ protect sensitive weights.

Does it require a GPU or retraining?

No. It needs only a small calibration dataset and a few minutes of compute, which is why edge teams ship models to consumer hardware quickly.

Key Takeaways

• Post-training quantization compresses trained models to integer formats without retraining, making on-device deployment fast and affordable.
• INT8 typically costs under one percent accuracy, while aggressive INT4 needs smart methods to stay usable.
• Calibration data quality and activation outliers decide whether a quantized model succeeds or collapses.
• Runtimes like GGUF, ONNX Runtime, and TensorRT turn a quantized file into real speed on edge hardware.

Understanding Post-training Quantization for Edge AI

Post-training quantization for edge AI converts a trained model’s floating-point weights into low-precision integers, such as INT8 or INT4, after training finishes. It shrinks model size and power use so networks run on phones, sensors, and small boards without cloud access or retraining.

What Is Post-training Quantization for Edge AI?

Post-training quantization is the practice of converting a fully trained model's high-precision numbers into compact integers after training ends. A model normally stores its weights as 16-bit or 32-bit floating-point values, which are precise but heavy. Quantization maps those values onto a smaller grid, most often 8-bit or 4-bit integers, using a scale factor and a zero point. The result is a file that occupies far less memory and moves fewer bytes during inference. Because the conversion happens after training, it needs no labeled dataset and no gradient updates. That distinction separates it from quantization-aware training, which bakes low precision into the learning loop itself.

The core idea rests on a simple observation about neural networks. Most weights cluster in a narrow numeric range, so representing them with full floating-point precision wastes bits. A quantizer measures the range of a tensor, divides it into a fixed number of steps, and rounds each value to the nearest step. Understanding this process is easier once you grasp the basics of neural networks and how weights flow through layers. The rounding introduces small errors, and the art of quantization is keeping those errors from compounding. Good methods spend their effort protecting the few weights that matter most.

The goal is not just a smaller file, but a model that still answers correctly on a device with no cloud connection. Edge deployment changes the constraints that matter, because a phone or sensor has strict limits on memory, battery, and heat. A quantized model addresses all three at once by shrinking the data the processor must fetch and compute. This is why post-training quantization sits at the center of the on-device movement. It converts research-grade models into something a battery-powered chip can actually run. The next sections trace how that conversion happens and where it can go wrong.

Why Quantization Became Essential for On-Device Intelligence

The demand for local models grew faster than the hardware could keep up, and quantization closed the gap. Running a modern model in the cloud is simple but carries real costs in latency, bandwidth, and recurring fees. Serving a seven-billion-parameter small model can cost ten to thirty times less than a large seventy-billion-parameter system, a spread highlighted in industry coverage of edge SLMs in telecom workloads. Those economics push teams to move inference onto devices they already own. A quantized model makes that move feasible because it fits inside the tight memory budget of a handset or an industrial gateway. The privacy benefit is just as strong, since data never leaves the device.

Market forecasts strongly reinforce this same industry trend toward smaller, cheaper, and more compressed models. Analysts at Gartner expect organizations to use task-specific small models three times more often than general-purpose large models by 2027. That prediction only works if those small models can run cheaply on real hardware, and quantization is the mechanism that makes it so. Techniques like reduce LLM inference costs often start with quantization before touching anything else. The pattern mirrors an earlier era when cloud teams learned to optimize compute spend line by line. Edge AI is now having that same reckoning, and post-training quantization is the first tool most teams reach for.

How Post-training Quantization Actually Works

Building on that foundation, the mechanics of quantization come down to mapping a continuous range onto a discrete grid. Every weight tensor has a minimum and maximum value that define its range. A quantizer picks a scale factor equal to that range divided by the number of available integer steps. For INT8, there are 256 steps, and for INT4, only 16. Each floating-point weight is then divided by the scale, rounded to the nearest integer, and stored. At inference time, the integer is multiplied back by the scale to approximate the original value.

The choice between symmetric and asymmetric quantization shapes the result. Symmetric quantization centers the grid on zero, which simplifies the math and suits weights that spread evenly around zero. Asymmetric quantization adds a zero point offset, which helps when values skew in one direction, as activations often do. Granularity matters too, because a single scale for an entire tensor is coarse. Per-channel quantization assigns a separate scale to each output channel, which preserves far more detail. This is conceptually similar to how batch normalization speeds up neural networks by rescaling values layer by layer.

Weight-only quantization and full integer quantization solve different problems and carry different costs. Weight-only methods compress the stored parameters but still compute in higher precision, which protects accuracy at the cost of some speed. Full integer quantization converts both weights and activations, which unlocks fast integer math on the chip. The tradeoff is sensitivity, because activations swing more widely than weights and resist compression. Many production pipelines settle on weight-only INT4 for language models and full INT8 for vision models. The correct choice depends on the hardware target and the accuracy floor the application can tolerate.

Calibration ties the whole process together and deserves its own attention. A quantizer cannot pick good scale factors without seeing representative data flow through the network. It runs a few hundred sample inputs, records the range of activations at each layer, and sets the clipping thresholds accordingly. Skipping this step forces the quantizer to guess, which usually produces a broken model. The later section on calibration explains why this small dataset carries so much weight. Getting it right is often the difference between a model that ships and one that fails review.

Integer Formats: INT8, INT4, and the Push to Lower Bits

Shifting focus to the formats themselves, the choice of bit width is the single biggest lever in quantization. INT8 has long been the safe default because it balances compression and accuracy well. It halves the size of an FP16 model and runs on nearly every modern accelerator with native integer support. Moving to INT4 doubles the savings again, turning a large model into something that fits on a phone. That aggressive step is where the risk begins, because sixteen integer levels leave little room for nuance. The field has pushed toward three-bit and even two-bit formats, but returns diminish sharply below four bits.

Newer floating-point formats now compete with plain integers at the low end. FP8 keeps a tiny exponent, which preserves dynamic range better than INT8 in some workloads and stays essentially lossless. Emerging FP4 and microscaling formats extend that idea to four bits, pairing small blocks of values with shared scales. These formats matter because hardware vendors are building native support for them into new chips. A model quantized to a format the silicon understands runs faster than one that needs software emulation. The Llama Nemotron models for agentic AI reflect this move toward hardware-aligned low precision.

The rule of thumb is to use the highest bit width your device budget allows, then drop lower only with a method that protects accuracy. A model that fits comfortably at INT8 rarely benefits from the extra risk of INT4. When memory is the hard constraint, INT4 with a smart algorithm becomes the practical target. Below four bits, most teams accept a visible quality drop or turn to quantization-aware training instead. The decision is rarely about theory and almost always about the specific chip and application. Knowing your hardware's native formats first saves a great deal of trial and error later.

GPTQ, AWQ, and SmoothQuant Compared

Turning to the algorithms that make low-bit quantization viable, three names dominate current practice. GPTQ quantizes each layer while compensating for the error introduced in earlier weights, using second-order information to guide the rounding. This approach delivers near-lossless INT4 compression via GPTQ on many language models. It is precise but computationally heavier, since it processes weights in a careful sequence. The payoff is a four-bit model that behaves almost like its full-precision parent. For teams that need maximum quality at INT4, GPTQ remains a strong default.

AWQ takes a different route by protecting the weights that activations touch most. It identifies the small fraction of salient weight channels and scales them to preserve their information before quantizing. The method is faster than GPTQ and produces a format that runs efficiently on GPUs, with reports that AWQ delivers over 3x speedup on edge GPUs. That combination of speed and hardware friendliness makes it popular for real deployments. SmoothQuant tackles the activation problem head-on by shifting difficulty from activations onto weights. It rescales both so that neither carries extreme outliers, which lets a model quantize cleanly to eight bits across the board.

No single method wins everywhere, so the right pick depends on your bit target, hardware, and tolerance for compute time. GPTQ suits weight-only INT4 workloads where preserving output quality is clearly the paramount concern. AWQ fits GPU deployments that want speed without a large accuracy sacrifice. SmoothQuant shines when you need full INT8 with both weights and activations quantized. Many teams even combine ideas, using SmoothQuant preprocessing before a GPTQ or AWQ pass. Learning the strengths of each avoids the trap of forcing one tool onto every problem. The comparison table later in this guide lays these tradeoffs out side by side.

Calibration Data and Why It Decides Your Accuracy

Beyond the algorithms, calibration data quietly determines whether quantization succeeds. The quantizer must learn the realistic range of activations, and it learns that range from a small set of sample inputs. If those samples do not resemble production traffic, the scale factors will be wrong. A model calibrated on formal text may clip badly when it meets casual chat, and the reverse is equally true. A few hundred well-chosen examples usually suffice, but they must reflect the true distribution of use. This is one place where a little domain knowledge outperforms raw volume of data.

The mechanics of calibration are simple, yet the consequences are large. During a calibration pass, the tool records the minimum and maximum activation at each layer and often computes percentiles to reject rare extremes. Clipping at a sensible percentile prevents a single freak value from stretching the entire grid. Teams that treat calibration as an afterthought tend to ship models that behave strangely on edge cases. Those who curate their calibration set carefully see accuracy hold much closer to the original. The lesson mirrors broader work on fine-tuning LLMs at home, where data quality beats data quantity almost every time.

Handling Activation Outliers Without Retraining

Building on the calibration problem, activation outliers are the single biggest threat to clean quantization. Large language models tend to produce a handful of activation values that are far larger than the rest. These outliers stretch the numeric range so much that ordinary values lose precision when squeezed onto a small grid. A naive INT8 pass on such a model can crater its accuracy despite the format being generous. This is why methods that ignore outliers often fail on transformers while succeeding on simpler vision networks. The distribution of activations, not just weights, drives the outcome.

Several practical strategies tame these activation outliers without ever touching the original model training loop. SmoothQuant migrates the scale of extreme activations into the weights, flattening the activation distribution so it quantizes cleanly. Mixed-precision approaches keep the outlier channels in higher precision while compressing everything else to low bits. Per-channel and per-group scaling give the quantizer finer control over which values get protected. These techniques matter because they let teams keep the speed of integer math while avoiding its worst failure mode. Related defensive thinking appears in work on a breakthrough that challenges deep learning norms.

Diagnosing an outlier problem early saves hours of confused debugging after a model ships broken. A simple activation histogram often reveals the culprit, showing a long tail that a single scale cannot represent. Once identified, the fix is usually a smarter scaling scheme rather than a return to full precision. Teams that skip this diagnostic tend to blame the format when the real issue is distribution. Understanding outliers turns quantization from a gamble into a predictable engineering step. The next section quantifies how much accuracy these choices actually preserve.

Measuring the Accuracy Cost of Low-Bit Quantization

Stepping back from mechanics, the practical question is always how much accuracy quantization actually costs. The answer is reassuring at eight bits and more nuanced below. Well-implemented INT8 weight and activation quantization typically loses only one to three percent, and FP8 is essentially lossless. A recent study found that Llama 3.1 8B retained over 99 percent of its accuracy at eight bits, a result detailed in a systematic evaluation of on-device LLMs. Even at four bits, the same model held 98.9 percent accuracy on coding tasks with a good method. These numbers explain why teams trust quantization for production rather than treating it as a lab curiosity.

The picture changes with task type and bit width, so context matters. High-precision reasoning and math tasks show larger drops than casual generation or classification. Below four bits, accuracy tends to fall off a cliff unless a specialized method intervenes. The compression rewards are still real, since moving from FP16 to INT4 cuts model size by roughly 75 percent. Hardware studies also report 57 to 61 percent reductions in area and power, a finding backed by energy-efficiency benchmarks across 28 quantized models. Those savings are why phone makers and IoT vendors accept a small quality tradeoff.

Measuring accuracy correctly requires more than a single benchmark number. A model can score well on multiple-choice questions yet degrade on long-form generation, so evaluation must match the real use case. Teams should test on their own data, not just public leaderboards, because domain shift changes the result. Perplexity is a useful early signal, but downstream task accuracy is the metric that matters to users. Tracking both before and after quantization reveals whether the compression stayed within budget. This discipline separates a confident deployment from a hopeful one.

The honest summary is that INT8 is nearly free, INT4 is affordable with care, and anything lower demands real engineering. Most edge projects land on INT8 for safety or INT4 with GPTQ or AWQ for tight memory. The right target depends on the device, the task, and the accuracy floor the product can tolerate. Chasing the smallest possible model without measuring quality is a common and costly mistake. A structured evaluation loop catches regressions before users ever see them. The following sections turn from theory to the tools that make these gains real on hardware.

Runtimes and Formats: GGUF, ONNX, and TensorRT

Turning to execution, a quantized model is only useful when a runtime can run it fast. GGUF has become the standard format for llama.cpp deployments on CPUs and Apple Silicon, packaging quantized weights with metadata in one file. ONNX Runtime offers broad hardware coverage and pairs well with execution providers that target specific accelerators. TensorRT and TensorRT-LLM squeeze maximum speed from NVIDIA hardware, including newer FP4 paths on edge boards. The gap between these runtimes is large, as shown in the 2026 edge LLM runtime stack comparison. Picking the wrong runtime here can leave significant performance and battery life sitting on the table.

The format and runtime should be chosen together with the target chip, not bolted on at the end. A GGUF model with a four-bit quantization scheme runs well on a laptop CPU but may not exploit a phone's neural engine. ONNX Runtime with a hardware execution provider often wins on mixed devices that lack a dedicated GPU. TensorRT-LLM with speculative decoding can add a 2.5 times uplift on capable boards. Matching the runtime to the silicon is where theoretical savings become real latency wins. The deployment walkthrough below shows this pairing in concrete steps.

Hardware Realities: NPUs, Mobile Chips, and Single-Board Computers

Beyond software, the hardware target defines what quantization can achieve. Modern phones now ship neural processing units built specifically for low-precision integer math. The iPhone 17 Pro integrates neural accelerators into its GPU cores, reaching roughly 75 TOPS and running eight-billion-parameter models at over 20 tokens per second. That capability turns a quantized model from a slideshow into a responsive assistant. Android flagships and Qualcomm platforms offer similar NPU acceleration for INT8 and INT4 workloads. The chip decides which formats run natively and which fall back to slow emulation.

Single-board computers occupy the other end of the spectrum and prove how far quantization reaches. By compressing a model to eight or four bits and running it with an optimized runtime, interactive latency is achievable on devices costing under a hundred dollars. A Raspberry Pi class board can host a small quantized model for offline tasks with no cloud dependency. Industrial gateways use the same trick to add intelligence to factory floors and remote sites. This democratization connects naturally to the wider collaboration between AI and IoT. Cheap silicon combined with careful quantization equals real, useful intelligence running at the network edge.

The practical lesson is to profile on the actual target device rather than trusting a desktop benchmark. A model that flies on a workstation GPU may crawl on a battery-powered board with a different memory system. Thermal throttling, memory bandwidth, and NPU format support all shift the real numbers. Teams that test early on the deployment hardware avoid painful surprises near launch. The relationship between chip and format is the hidden variable in most edge projects. Knowing it upfront turns guesswork into a repeatable engineering plan.

Risks and Failure Modes of Aggressive Quantization

Turning to the downside, aggressive quantization introduces failure modes that teams must respect. The most common is silent accuracy loss, where a model still runs but answers slightly worse in ways that evade casual testing. Below four bits, that degradation can become severe, especially on reasoning and numeric tasks. Outlier-driven collapse is another risk, where a single unhandled activation range wrecks an entire layer. These failures rarely throw an error, which makes them harder to catch than a crash. Careful and repeated evaluation remains the only truly reliable defense against these quiet, silent failures.

Security and robustness concerns also deserve attention as models move on-device. A quantized model shipped to a user's device can be inspected, copied, or tampered with more easily than a cloud endpoint. Low precision can also change a model's behavior on adversarial or out-of-distribution inputs in unpredictable ways. Bias that was mild in the full model can occasionally sharpen after compression, so fairness testing should repeat post-quantization. These issues echo broader questions raised in Anthropic's edge AI safety approach. Compression is not neutral, and it can shift a model's failure surface.

The safe path treats quantization as a change that must be validated, not a free optimization to apply blindly. Every quantized build should pass the same accuracy, safety, and bias checks as its full-precision parent. Regression tests catch the silent drops that manual review misses. Keeping the original model available allows quick rollback when a quantized build misbehaves. Teams that build this discipline into their pipeline ship with confidence rather than crossed fingers. The next section widens the lens to the ethical stakes of on-device models.

Ethics, Privacy, and the Trust Case for On-Device Models

Widening the view, quantization carries an ethical dimension that goes beyond engineering. Running a model on-device keeps sensitive data local, which strengthens privacy in a way cloud inference cannot match. Health, finance, and personal messaging all benefit when data never leaves the user's hardware. This local-first posture connects to ongoing debates about AI in phones and data privacy. Quantization is what makes that privacy-preserving architecture affordable at scale. The trust case for edge AI rests heavily on this foundation.

The same technology that protects privacy can also obscure accountability if teams are careless. A model that runs on millions of devices is harder to monitor, patch, or audit than a central endpoint. Compression can hide subtle behavior changes that would be visible in a supervised cloud system. Responsible deployment pairs on-device inference with clear update paths and honest disclosure of limits. Federated approaches, discussed in work on secure federated learning for IoT, offer one way to balance privacy and oversight. Trust is ultimately earned through transparency and honest oversight, not through local processing alone.

The Future of Post-training Quantization for Edge AI

Looking ahead, post-training quantization for edge AI is moving toward smarter, more automatic precision. Mixed-precision methods now learn per-layer bit-width assignments rather than applying one format everywhere. Reinforcement-learning schedulers can search for the accuracy-efficiency sweet spot across a whole network. This trend means future pipelines will decide which layers deserve more bits without human tuning. The result should be higher quality at lower average precision than today's uniform schemes. Automation is steadily turning quantization from a manual craft into dependable, repeatable production infrastructure.

New numeric formats and hardware are arriving together, which changes the calculus. FP4 and microscaling formats pack four-bit values into blocks with shared scales, preserving more range than plain INT4. Chip vendors are baking native support for these formats into edge accelerators and mobile NPUs. As silicon and formats co-evolve, the accuracy penalty for very low precision keeps shrinking. The connection to broader model efficiency work, including neural architecture search, will deepen. Models will increasingly be designed and quantized as one joint problem.

The endpoint is a world where capable models run everywhere, and quantization is the invisible layer that makes it possible. On-device agents, offline assistants, and private copilots all depend on aggressive compression staying accurate. As agentic systems spread, described in coverage of agentic AI for smarter workflows, edge quantization becomes their enabling substrate. The field is maturing from a manual optimization into a standard build step. Teams that master it now will ship faster and cheaper than those who wait. The chart below visualizes how bit width trades size against accuracy.

Practical Guidance for Teams Adopting Edge Quantization

Pulling the threads together, adopting post-training quantization for edge AI is a sequence of pragmatic choices. Start with the highest bit width your device budget allows, then compress lower only if memory forces it. Invest in a representative calibration set, because that small dataset drives most of your accuracy outcome. Match the format and runtime to the target chip before writing any deployment code. These habits prevent the most common and expensive mistakes teams make.

Treat every quantized build as a first-class artifact that earns the same testing and monitoring as its full-precision source. Measure accuracy on your own data, watch for silent regressions, and keep a rollback path ready. Begin with a small pilot on real hardware rather than a full rollout on assumptions. The teams that succeed treat quantization as an ongoing engineering practice, not a one-time trick. Learning from published deployments, like guidance on install an LLM on macOS, shortens the path. Done well, edge quantization turns ambitious models into shippable products.

How to Implement Post-training Quantization on an Edge Device

Step 1 - Choose the model and bit width

Begin by selecting a base model that already fits your task and a target bit width. A small instruction-tuned model with 1 to 8 billion parameters is a sensible starting point for edge work. Check the memory available on the target device and work backward to the largest model that fits. INT8 is the safe first choice, while INT4 roughly doubles the savings when memory is very tight. Confirm which integer and floating-point formats your target chip accelerates natively before you commit to anything. A format the silicon does not support will fall back to slow software emulation instead. This single early decision quietly shapes every other step in the workflow that follows below.

Step 2 - Prepare a calibration dataset

Assemble a few hundred sample inputs that closely mirror the traffic your model will see in production. Around 300 representative examples is a reasonable target for most language and vision workloads today. The calibration set teaches the quantizer the realistic range of activations at every single layer. Draw examples from your own logs or domain data rather than a generic public corpus. Aim for variety across the styles, lengths, and topics that your real users actually send. A well-curated set of 300 samples beats a large, mismatched collection almost every single time. Save the samples as plain text that your quantization tool can stream during the calibration pass.

Step 3 - Run post-training quantization

Run a proven quantization method rather than a naive rounding pass, especially for four-bit targets. GPTQ and AWQ both protect the weights that matter most and preserve accuracy at INT4 levels. An AutoAWQ-style workflow reads your calibration data and writes a compact quantized model in one pass. The process usually finishes in under 30 minutes on a single GPU and needs no gradient updates. Watch the reported quantization error closely, since a large value signals an outlier or calibration problem. Keep the original 16-bit weights on disk so you can always compare outputs and roll back cleanly.

Step 4 - Convert to an edge-friendly format

Convert the quantized model into the exact format your runtime expects on the target device. GGUF is the natural choice for llama.cpp on CPU and Apple Silicon, while ONNX suits mixed hardware. The llama.cpp toolchain converts and packages the weights into a single portable file for you. Pick a 4-bit K-quant preset that balances size and quality well for your specific target chip. Verify that the output file loads cleanly on a workstation before moving it onto the device. A conversion error caught here is far cheaper than one discovered later out in the field.

Step 5 - Validate accuracy on the target

Test the quantized model on the actual device against your own evaluation set, not a public benchmark. Compare its answers to the full-precision model on the specific tasks your product depends on. Track both perplexity as an early signal and downstream task accuracy as the real deciding measure. Watch latency and memory under realistic load, including thermal throttling across 10 or more minutes. Run the same fairness and safety checks that you applied to the original full-precision model. Only promote the build once it clearly clears the accuracy floor your product can tolerate. This validation gate is what keeps a silent accuracy regression from ever reaching your real users.

Step 6 - Deploy and monitor on the device

Ship the validated model with a runtime tuned for your hardware and wire up basic monitoring. Load the GGUF or ONNX file into llama.cpp, ONNX Runtime, or TensorRT depending on the chip. Log latency, memory, and any user-visible errors so you can spot drift within the first 30 days. Keep an update channel ready so you can push a corrected build without a full app release. Maintain the original model and the 300-sample calibration set so future re-quantization stays fast and cheap. Treat the deployed model as a living artifact that needs the same care as any production service.

Key Insights on Edge Quantization Performance

• Llama 3.1 8B held over 99 percent accuracy at eight bits, and a systematic evaluation of on-device LLMs logged 98.9 percent on coding tasks at four bits. That margin shows INT4 stays viable for real production once a capable method protects the weights.
• Compressing FP16 to INT4 removes 75 to 93.75 percent of stored size, and energy benchmarks across 28 quantized models logged 57 to 61 percent lower area and power. Those paired savings in size and power are exactly what make offline, private inference practical on constrained edge hardware today.
• Activation-aware weight quantization yields a format that reaches over 3x speedup on edge GPUs, which proves that method choice affects latency as much as raw model size does.
• A four-bit GGUF conversion shrank Llama 3.2 3B by 68.66 percent in a mobile quantization study, which let the model run fully offline on a standard Android phone.
• The iPhone 17 Pro reaches roughly 75 TOPS and runs eight-billion-parameter models above 20 tokens per second. That throughput, noted in analysis of small models on edge devices, comes from dedicated on-chip neural accelerators.
• Speculative decoding alone added a 2.5 times throughput uplift on a Jetson Thor board, a result that the 2026 edge LLM runtime stack comparison measured directly. It clearly shows that careful runtime tuning compounds the very efficiency gains that quantization already delivers on its own.
• Well-tuned INT8 loses under one percent while FP8 stays essentially lossless, an outcome that NVIDIA's quantization guidance ties to careful calibration. Careful outlier handling makes the entire difference between a clean quantized result and a badly broken, unusable one.

Taken together, these numbers describe a technique that has crossed from research into dependable engineering. Eight-bit quantization is now nearly free, and four-bit is affordable when a smart algorithm protects the sensitive weights. The savings in size and power are what let capable models run on phones and cheap boards at all. Runtime and hardware choices multiply those gains, turning a compressed file into real interactive speed. The consistent thread is that measurement and method choice, not raw compression alone, decide the outcome. Teams that respect that pattern ship quantized models that users never notice were compressed.

Dimension	INT8 baseline	GPTQ	AWQ	SmoothQuant
Typical bit target	8-bit	4-bit	4-bit	8-bit
Core technique	Range mapping	Second-order error compensation	Activation-aware scaling	Activation-to-weight migration
What it quantizes	Weights and activations	Weights only	Weights only	Weights and activations
Accuracy at low bit	Under 1% loss	Near-lossless INT4	High, GPU-friendly	Clean W8A8
Speed on edge GPU	Good	Good	Over 3x uplift	Good
Cost to quantize	Very low	Higher	Low	Low
Best use case	Safe default	Max quality at INT4	Fast GPU deployment	Full INT8 with outliers

Real-World Examples of Quantized Edge Deployments

Llama 3.2 3B on an Android Phone via GGUF

Researchers deployed Llama 3.2 3B to a consumer Android phone by applying four-bit post-training quantization and converting the result to GGUF. The compression cut the model size by 68.66 percent, which let it fit in the phone's memory, as shown in a mobile quantization study on Llama 3.2 3B. They ran the quantized file through llama.cpp inside a Termux environment using the Ollama framework. The model produced usable responses fully offline, with no cloud call and no data leaving the device. The main limitation was inference speed, since a CPU-only phone cannot match a dedicated accelerator on longer prompts. The work still proved that a capable model can live entirely on commodity hardware.

Low-Power Local Inference on Consumer Hardware

An engineer ran a quantized model on consumer-grade edge hardware using ONNX Runtime with a TensorRT execution provider. The setup returned 20-token responses in roughly 0.4 seconds while drawing a maximum of about five watts, detailed in a guide to deploying quantized LLMs on edge hardware. CPU usage held near 30 percent and GPU near 55 percent, leaving headroom for other tasks. That efficiency turned a small board into a responsive local assistant without any cloud dependency. The limitation was that the result depended on a compatible GPU execution provider, which not every cheap device offers. It still remains a strong, repeatable template for privacy-preserving local inference on modest hardware.

Llama 3.3 70B on a Jetson Thor Edge Board

A benchmarking team ran Llama 3.3 70B on a single Jetson Thor board by combining aggressive quantization with an optimized runtime. Moving from a GGUF four-bit build to TensorRT-LLM with NVFP4 and EAGLE-3 speculative decoding changed throughput dramatically, per the 2026 edge LLM runtime stack comparison. Speculative decoding alone delivered a 2.5 times uplift on the same hardware and model. The result showed that a seventy-billion-parameter model can run at the edge when quantization and runtime work together. The limitation was cost, since a Jetson Thor is far pricier than a hobbyist board. For industrial edge use, though, the performance justified the hardware investment.

Lessons From Production Quantization Projects

Case Study: NVIDIA's Post-Training Quantization Workflow

NVIDIA faced the problem that large language models are expensive to serve and too heavy for constrained hardware. The company built a post-training quantization workflow that applies INT4, INT8, and FP8 formats with calibration and outlier handling. Using second-order methods like GPTQ, the workflow reached near-lossless INT4 compression on many models, documented in NVIDIA's guide to post-training quantization. Well-implemented INT8 held accuracy loss under one percent, and FP8 proved essentially lossless. The measurable impact was smaller, faster models that fit hardware the full-precision versions could not. The limitation the guide is candid about is that plain four-bit quantization degrades noticeably unless advanced methods intervene. That honesty makes the workflow a realistic template rather than a marketing claim.

Case Study: Benchmarking 28 Quantized Models for Energy

A research group faced the problem that no one had rigorously measured whether quantization actually saves energy on edge devices. To solve it, they built and benchmarked 28 quantized models from the Ollama library across five standardized datasets, including GSM8K and HumanEval. The study, published in an ACM analysis of sustainable LLM inference, measured energy, latency, and output accuracy at multiple bit widths. It found substantial memory and bandwidth savings alongside 57 to 61 percent reductions in area and power for aggressive quantization. The measurable impact was a clear efficiency map that teams can use to pick a bit width for their energy budget. The limitation the authors stress is that lower-bit quantization degrades accuracy on high-precision tasks like math. The result gives practitioners evidence rather than intuition for the size-versus-quality tradeoff.

Case Study: Activation-Aware Weight Quantization on Edge GPUs

Teams deploying four-bit models on GPUs hit the problem that naive low-bit quantization was both slow and inaccurate. The solution was activation-aware weight quantization, which identifies the salient weight channels that activations touch most and scales them before compression. This approach produced a hardware-friendly four-bit format that ran efficiently on GPUs, described in a breakdown of LLM quantization methods. The measurable impact was over a 3 times speedup on edge GPUs, a large throughput increase that kept accuracy close to full precision. That combination made it a popular default for real GPU deployments rather than a research demo. The limitation is that the method quantizes weights only, so activations still run in higher precision and cap the savings. For many edge GPU targets, that tradeoff is exactly the right balance.

Common Questions About Post-training Quantization for Edge AI