Adam Optimizer

Introduction

The Adam optimizer is the engine behind most deep learning models trained over the past decade. It decides how a neural network adjusts its weights after every batch of training data. Since 2014, the original Adam paper has gathered more than 100,000 citations across machine learning. That reach is not an accident, because the method works across vision, language, and reinforcement learning. It blends two older ideas, momentum and adaptive learning rates, into one compact update rule. This guide explains what the Adam optimizer does and how its underlying math actually works. You will also see how to use it in code and where plain gradient descent still wins.

Quick Answers on the Adam Optimizer

What is the Adam optimizer in simple terms?

The Adam optimizer is a training algorithm that adapts each weight’s learning rate using running averages of recent gradients and their squares.

What does Adam stand for?

Adam stands for Adaptive Moment Estimation. It tracks the first moment, the mean, and the second moment, the uncentered variance, of the gradients.

What are the default settings for the Adam optimizer?

The Adam optimizer defaults to a learning rate of 0.001, beta 1 of 0.9, beta 2 of 0.999, and epsilon of 1e-8.

Key Takeaways

• The Adam optimizer combines momentum and per-parameter adaptive learning rates into a single, easy to use update rule.
• Default values of beta 1 at 0.9, beta 2 at 0.999, and epsilon at 1e-8 work well for most models.
• The AdamW variant fixes how weight decay is applied and now trains nearly every large transformer model.
• Adam converges fast, yet tuned SGD with momentum can still generalize better on some vision benchmarks.

What Is the Adam Optimizer?

The Adam optimizer is a gradient-based method that gives every parameter its own adaptive learning rate. It estimates the first and second moments of the gradients, then uses them to scale each update.

From Gradient Descent to Adaptive Moment Estimation

Every neural network learns by gradient descent, a method that nudges weights toward lower loss. Plain gradient descent applies one fixed learning rate to every parameter in the model. That single rate is a blunt tool, because different weights often need very different step sizes. Early work added momentum, which remembers past gradients to push through flat regions and noise. The Adam extends this lineage by giving each parameter a rate that adapts as training proceeds. It builds directly on momentum and on a method called RMSprop, both explained further below.

To see the gap Adam fills, picture a loss surface shaped like a long, narrow valley. Plain gradient descent bounces across the steep walls while crawling slowly along the gentle floor. A rate large enough to move forward also causes wild oscillation across the valley. A rate small enough to stay stable makes the journey painfully slow. The Adam reads the recent history of each direction and rescales the steps automatically. Frequent, noisy directions get damped, while rare but informative directions keep their influence.

This adaptive behavior is why the Adam spread so quickly through deep learning practice. It needs little manual tuning to reach a good result on most architectures. A beginner can train a working model without hand designing a learning rate schedule. The same defaults that train a small classifier also train a large network reasonably well. You can compare this convenience against the broader family of common machine learning algorithms on our site. That low barrier to entry shaped a generation of models and tutorials.

The Math Behind the Adam Optimizer

The math of the Adam looks intimidating, yet it rests on a few simple averages. At each step, the algorithm computes the gradient of the loss for the current batch. It then updates a running average of that gradient, called the first moment estimate. It also updates a running average of the squared gradient, called the second moment estimate. These two moving averages capture the recent direction and the recent scale of the gradients. Beta 1 controls how fast the first average forgets, and beta 2 controls the second average.

Both averages start at zero, which biases them toward zero during the first steps. The Adam corrects this with a bias-correction term that grows the estimates early on. Without that fix, the first updates would be far too small to make progress. The corrected first moment acts like a smoothed direction of travel for each weight. The corrected second moment acts like a per-weight estimate of recent gradient size. Dividing the first by the square root of the second gives a normalized step. This normalization is what gives every parameter its own effective learning rate.

The final update multiplies that normalized step by the global learning rate alpha. A small constant called epsilon sits in the denominator to prevent division by zero. The interactive explorer above lets you watch these pieces move as the step number changes. Early in training, bias correction inflates the raw averages toward their true values. Later, the correction fades and the step settles toward the plain ratio of the moments. You can review the precise equations in the original Adam paper by Kingma and Ba. The notation there matches the names used throughout this guide.

It helps to connect this update to the loss function the network is minimizing. The gradient comes from differentiating that loss with respect to every trainable weight. Choices like cross entropy loss for classification shape the gradients Adam then consumes. A noisy or poorly scaled loss makes the second moment estimate swing widely. The Adam absorbs some of that noise through its exponential averaging. That smoothing is a feature, but it can also hide real signal when betas are set too high. Understanding the loss and the optimizer together leads to far better training outcomes.

The Hyperparameters of the Adam Optimizer

Building on that update rule, the Adam exposes four settings you can tune. The learning rate alpha sets the overall size of each step, with 0.001 as a common default. Beta 1, usually 0.9, controls the memory of the first moment, the running gradient mean. Beta 2, usually 0.999, controls the memory of the second moment, the running squared gradient. Epsilon, often 1e-8, is a tiny constant that keeps the denominator safely away from zero. These defaults come straight from the framework documentation and rarely need large changes. The torch.optim.Adam documentation lists every one of these four settings very clearly.

The betas deserve special attention because they set the optimizer’s memory horizon. A beta 1 of 0.9 means the first moment averages roughly the last ten gradients. A beta 2 of 0.999 means the second moment averages roughly the last thousand squared gradients. Higher betas produce smoother but slower reacting estimates of direction and scale. Lower betas react faster but let more noise into each update. For most problems the standard values strike a sensible balance between speed and stability. You only need to adjust them when training behaves strangely or diverges.

The learning rate remains the single most important knob in practice. Too high a rate makes the loss explode, while too low a rate stalls progress. Many teams pair the Adam with a warmup schedule that raises alpha gradually. Others decay the rate over time to fine tune the final weights. Epsilon usually stays untouched, though some large models raise it for numerical stability. A good default workflow keeps betas fixed and tunes only the learning rate first. That discipline saves hours of confused experimentation across the whole project.

How Adam Compares to SGD, Momentum, and RMSprop

Turning to the wider field, the Adam is one of several gradient methods. Stochastic gradient descent, or SGD, uses a single learning rate with no adaptation at all. SGD with momentum adds a velocity term that smooths the path toward the minimum. RMSprop introduces per-parameter scaling by dividing through a running average of squared gradients. Adam essentially fuses momentum and RMSprop, taking the direction smoothing of one and the scaling of the other. That combination is why it often converges faster than any single predecessor on its own. The trade is more state to store and a few more moving parts.

The practical difference shows up most clearly in how much tuning each method needs. SGD frequently needs a carefully designed learning rate schedule to reach top accuracy. The Adam usually reaches a strong result with its defaults and little fuss. On large vision benchmarks, though, well tuned SGD with momentum can still generalize better. That tension between convenience and final accuracy runs through the rest of this guide. Knowing the difference between machine learning and deep learning helps frame these choices. The right optimizer depends on the model, the data, and the time available.

Why Adam Became the Default Optimizer in Deep Learning

Beyond the math, the rise of the Adam is a story about convenience. When it appeared in 2014, training deep networks was fragile and tuning heavy work. Adam offered strong results out of the box on a wide range of models. That reliability turned it into the default choice in tutorials, courses, and research code alike. Frameworks shipped it with sensible defaults, so newcomers reached for it first. Each successful project added to its reputation and pulled in the next wave of users. Network effects then locked the Adam into the field’s shared habits.

The timing also matched the explosion of large datasets and deeper architectures. Models with millions of parameters made manual rate tuning impractical for most teams. The Adam scaled to these models without demanding a custom schedule per layer. It handled the sparse, noisy gradients that come from techniques like dropout and data augmentation. Pairing it with tricks such as batch normalization for faster training made deep models trainable. The result was a virtuous cycle of bigger models and easier optimization. That self reinforcing cycle defined much of the entire deep learning boom.

Popularity, of course, is not the same as being optimal for every task. The Adam became a safe default precisely because it rarely fails badly. Safe defaults matter when a team has limited time and many experiments to run. Yet researchers kept finding cases where other methods edged it out on accuracy. Those findings did not dethrone Adam, but they did refine when people reach for it. The honest view treats it as an excellent starting point rather than a final answer. Later sections in this guide show exactly where that nuance becomes important.

Using the Adam Optimizer in PyTorch and Keras

In practice, calling the Adam takes only a line or two of code. In PyTorch you create an instance of torch.optim.Adam and pass it the model parameters. You supply the parameters through model.parameters() and set a learning rate through the lr argument. The two decay rates arrive together in the betas tuple, defaulting to 0.9 and 0.999. From there the training loop calls three methods in order: zero the gradients, run backward, then step. The step method is where the Adam applies its bias-corrected update to every weight. That short ritual repeats once for every batch in your dataset.

Keras keeps the pattern just as compact for higher level workflows. You pass the string adam to the compile method, or build an Adam object for more control. The object exposes the same learning rate, beta, and epsilon values under friendly names. The Keras Adam documentation shows the exact argument names and defaults. Once compiled, the fit method handles the gradient steps without further ceremony. This is why so many beginners meet the Adam on their very first model. The same call works for a tiny classifier or a deep convolutional network.

A few habits make these calls far more reliable in real projects. Pair the optimizer with the right loss for your task, then watch the training curve closely. Our guide to PyTorch loss functions for machine learning covers sensible choices. Log the loss every few steps so you can spot divergence early. Start with the default learning rate and change it only after seeing the curve. Save checkpoints so a bad run never costs you a full training cycle. These small disciplines turn a one line call into a dependable workflow.

Adam Across Neural Network Architectures

Among the many model families, the Adam adapts to almost all of them. Convolutional networks for images train smoothly with its default settings in most cases. Recurrent networks, which once suffered from unstable gradients, became far easier to train with Adam. Transformers, the architecture behind modern language models, depend on an Adam variant for stable training. Even graph networks and diffusion models reach for the same family of update rules. The activations feeding these layers, such as the ReLU activation function used widely, shape the gradients. Adam then handles whatever scale and noise those gradients carry.

Architecture does change which settings work best, even when the optimizer stays the same. Transformers usually lower beta 2 toward 0.98 and add a learning rate warmup phase. Very deep vision models sometimes prefer SGD for the last points of accuracy. Small networks built on standard layers, like those in how neural networks work, often train fine with the plain Adam defaults. The lesson is that the Adam is a flexible base, not a fixed recipe. You adjust a knob or two to match the architecture and the data. That adaptability is exactly why it spans so many different model types.

AdamW and the Weight Decay Fix

Despite the strengths of the Adam, one detail caused years of subtle problems. Weight decay is a regularizer that gently shrinks weights to reduce overfitting. The classic Adam implementation folded weight decay into the gradient as an L2 penalty. With adaptive scaling, that penalty got distorted differently for each parameter. The result was weaker regularization and worse generalization than practitioners expected. Researchers Ilya Loshchilov and Frank Hutter diagnosed this mismatch and proposed a clean fix. They separated weight decay from the adaptive gradient step entirely.

Their method, AdamW, applies weight decay directly to the weights after the adaptive update. This decoupling restores the intended regularization for every parameter regardless of its gradient history. Their paper, Decoupled Weight Decay Regularization, reports up to a fifteen percent relative improvement. That gain appeared in test error across several image recognition benchmarks. AdamW also made the best weight decay value more independent of the learning rate. That independence simplified tuning, since the two knobs stopped fighting each other. The fix was small in code yet large in measurable effect.

AdamW now trains nearly every large transformer, including the models behind modern chat systems. Frameworks expose it as a separate class, so switching costs only a single line. The defaults match plain Adam, with weight decay set as its own clear argument. For any model that uses regularization, AdamW is the safer modern choice. Teams training language models reach for it almost without thinking now. The plain Adam remains common where weight decay plays no role. Knowing which one to pick is part of using the family well.

The AdamW story carries a useful lesson about defaults and hidden assumptions. A widely used tool can ship a subtle flaw that millions of runs quietly inherit. It took careful analysis, not raw scale, to surface the weight decay problem. Once named, the fix spread fast because the improvement was easy to measure. This pattern repeats across optimization research, where small corrections compound into real gains. It also shows why reading the source papers still rewards serious practitioners. The best defaults are the ones you understand well enough to question.

Other Adam Variants Worth Knowing

Beyond AdamW, the Adam has inspired a whole family of refinements. AMSGrad keeps the maximum of past second moments to fix a convergence gap, discussed later. Nadam folds Nesterov momentum into Adam for a slightly sharper sense of direction. AdaMax replaces the second moment with an infinity norm for certain stability benefits. RAdam adds a rectification term that tames the unstable variance of early training steps. Each variant targets a specific weakness rather than replacing the core method. You can think of them as careful patches on a very successful foundation.

Newer optimizers like Lion and Sophia push beyond the Adam family entirely. Lion uses only the sign of a momentum term, which cuts memory use sharply. Some teams report competitive results with these methods on very large models. Others find the gains shrink once the Adam baseline is tuned carefully. The field still treats Adam and AdamW as the trusted reference point. New methods must beat that baseline clearly to earn wide adoption. For most readers, mastering the Adam first remains the practical priority.

Tuning the Adam Optimizer for Better Results

For teams chasing the last points of accuracy, tuning the Adam pays off. Start by sweeping the learning rate across a few orders of magnitude. A common grid tries values near 0.0001, 0.001, and 0.003 to bracket the best region. The learning rate matters far more than the betas in almost every experiment. Plot the loss for each setting and pick the rate with the smoothest, fastest descent. Only after fixing the rate should you consider touching the decay values. This ordered search keeps the experiment count manageable and the results interpretable.

Schedules add another layer of control on top of the base learning rate. A warmup phase raises the rate slowly so early steps do not destabilize training. A decay phase then lowers the rate to settle the weights near a good minimum. Cosine and linear decay schedules both pair well with the Adam. For transformer models, a warmup phase is almost mandatory rather than optional. Combine schedules with techniques like cross validation to reduce overfitting for honest results. The schedule and the optimizer should be tuned as one system.

Good tuning also means knowing when to stop adjusting the optimizer at all. Many accuracy problems trace back to data quality, not the optimizer settings. A clean dataset with the right loss often beats heroic optimizer tuning. Watch for diminishing returns once the loss curve looks smooth and stable. At that point, effort is better spent on the model or the data pipeline. The Adam rewards a little tuning but punishes obsessive fiddling. Balance is the clear mark of an experienced machine learning practitioner here.

Watching a Single Adam Update Unfold

From there, it helps to trace a single Adam update from start to finish. Imagine one weight whose recent gradients have been small, steady, and mostly positive. The first moment average for that weight grows into a modest positive value over time. The second moment average stays small, because the squared gradients themselves remain small. Dividing the first moment by the root of the second yields a healthy, confident step. That weight moves steadily, since its gradient history stays consistent and low in noise. The same logic runs in parallel for every weight in the entire network. This parallel, per weight reasoning is what makes the method feel almost automatic.

Now picture a second weight whose gradients swing wildly between large positive and negative values. Its first moment average nearly cancels out, landing somewhere close to zero. Its second moment average grows large, because the squared gradients are consistently big. Dividing a near zero first moment by a large root produces a very tiny step. The method therefore moves this noisy weight cautiously, which is exactly the desired behavior. This per weight caution is the practical heart of adaptive moment estimation. A guide to the sigmoid function in neural networks shows where such gradients arise. Saturated activations, in particular, can produce exactly this kind of unstable gradient signal.

Bias correction adds one more wrinkle worth picturing at the very start of training. Both averages begin at zero, so early estimates lean too small without a fix. The correction divides each average by one minus the relevant beta raised to the step. Early on, that divisor sits well below one, which inflates the estimates appropriately. As steps accumulate, the divisor approaches one and the correction quietly fades away. This is why the first few updates still make meaningful progress rather than stalling. Tracing these cases by hand builds real intuition for what the algorithm actually does. The interactive explorer earlier in this guide animates exactly these moving pieces.

This mental model also explains why noisy data does not derail training as often as expected. The squared gradient average acts like a built in volume control for each direction. Loud, erratic directions get turned down, while quiet, reliable directions keep their voice. Techniques such as the softmax function in neural networks feed gradients into this same machinery. The optimizer does not know or care which layer produced a given gradient. It simply rescales each one by its own recent history of magnitudes. That uniform treatment is both the strength and the blind spot of the approach. It treats every parameter fairly, yet it ignores the structure linking them together.

Common Mistakes When Using Adam

Choosing among settings is easy to get wrong, so a few mistakes recur often. The most frequent error is leaving the learning rate at a value that is far too high. A high rate with the Adam can make the loss diverge within a few steps. The second common mistake is confusing plain Adam with AdamW when weight decay matters. Adding L2 penalty to plain Adam does not regularize the way most people expect. Teams also forget to zero the gradients each step, which corrupts every update. Each of these slips produces confusing curves that waste hours of debugging.

Another trap is trusting the default learning rate on a brand new architecture. Defaults are a starting point, not a guarantee across every possible model. A model with unusual scale or custom layers may need a smaller rate. People also misread a flat loss as convergence when it is really a stalled run. Comparing against a baseline like linear regression in machine learning grounds expectations. A simple baseline, such as classification and regression trees, reveals whether the network learns anything at all. That sanity check catches more problems than any optimizer tweak.

Mismatched epsilon values cause a subtler class of numerical problems. On models with very small gradients, the default epsilon can dominate the update. Some large models raise epsilon to keep the second moment ratio well behaved. Mixed precision training makes this issue more visible because of reduced numerical range. Watching for stalled or exploding losses usually surfaces an epsilon problem quickly. The Adam is forgiving, yet these edge cases still bite unprepared teams. Documenting your chosen settings prevents the same mistake from returning later.

Risks and Limitations of the Adam Optimizer

Despite the convenience, the Adam carries real limitations worth taking seriously. The first is a convergence gap that surfaced in careful theoretical analysis. Researchers showed a simple convex problem where Adam fails to reach the optimal solution. The cause is the exponential moving average, which can forget large but rare informative gradients. Their paper, On the Convergence of Adam and Beyond, proposed the AMSGrad fix. AMSGrad keeps the maximum of past second moments to restore a convergence guarantee. The flaw is narrow, yet it punctured the assumption that Adam always converges.

The second limitation concerns generalization rather than pure convergence speed. Adaptive methods can reach low training loss while landing at solutions that test worse. A well known study made this gap concrete across several learning problems. The paper, The Marginal Value of Adaptive Gradient Methods, compared Adam against SGD. On one separable problem, adaptive methods reached test error near fifty percent while SGD hit zero. That extreme case is illustrative rather than typical of every real dataset. Still, it explains why some vision teams prefer tuned SGD for final accuracy.

Memory cost is a third practical limitation of the Adam. It stores a first and second moment for every single parameter in the model. That roughly triples the optimizer state compared with plain SGD, which keeps one value. For a model with billions of weights, this overhead becomes a serious budget item. The chart below visualizes that memory cost against the generalization gains from AdamW. Teams training the largest models weigh this overhead against the speed Adam provides. The trade is usually worth it, but it is never free.

None of these limits make the Adam a poor choice for most work. They simply mark the boundaries where a thoughtful practitioner reaches for alternatives. AMSGrad addresses the convergence corner case when theory demands a guarantee. Tuned SGD answers the generalization gap on certain vision benchmarks. Sharding and offloading reduce the memory cost on very large models. Knowing these limits turns Adam from a magic button into an understood tool. That understanding is exactly what separates careful work from cargo cult tuning.

The Ethics and Reproducibility of Optimizer Choices

Looking beyond accuracy, optimizer choices carry quiet ethical and scientific weight. Reproducibility suffers when papers omit the exact learning rate, betas, and schedule used. Two teams using the Adam with different settings can reach very different conclusions. Reporting the full optimizer configuration is a basic requirement for honest, repeatable research. Random seeds, batch sizes, and warmup steps all shape the final reported numbers. When these details vanish, comparing methods fairly becomes nearly impossible. Good practice treats the optimizer setup as part of the experimental record.

There is also an energy and access dimension to these decisions. Faster convergence with the Adam can mean fewer training runs and lower energy use. The extra memory it needs, though, can push work onto larger and costlier hardware. That cost can widen the gap between well funded labs and smaller teams. Sharing tuned configurations openly helps level that uneven playing field. Reproducible defaults let newcomers stand on prior work rather than guesswork. Responsible reporting is a small habit with a broad and lasting payoff.

The Future of the Adam Optimizer

Looking ahead, the Adam is unlikely to disappear from deep learning soon. Its defaults are baked into countless tutorials, libraries, and production training pipelines. New optimizers keep appearing, yet they still measure themselves against the Adam baseline. Memory efficient variants are the most active frontier, since model sizes keep climbing fast. Methods that cut optimizer state without losing speed attract intense research interest. Sign based updates and low rank moment estimates are two promising directions. The core idea of adaptive moments looks set to endure for years.

Hardware trends will shape which optimizer refinements actually matter in practice. As accelerators grow, the memory cost of moment estimates becomes a sharper constraint. That pressure rewards methods that match Adam’s quality with a smaller footprint. Better theory may also narrow the gap between fast convergence and strong generalization. Research into the niche of specialized optimizers, like the Coati optimization algorithm, continues. Most of these stay specialized while Adam holds the general purpose role. The likely future is refinement of Adam rather than wholesale replacement.

For practitioners, the safe long term bet is to master the Adam family deeply. Understand the moments, the bias correction, and the weight decay fix in AdamW. Keep an eye on new methods, but adopt them only when the evidence is clear. The chart that follows summarizes the defaults and trade-offs in one compact view. Treat it as a quick reference whenever you set up a new training run. The Adam rewards understanding far more than it rewards blind faith. That stance will serve you across whatever optimizers come next.

How to Implement the Adam Optimizer Step by Step

Moving on to hands-on work, this section walks through a clean Adam training setup. The steps below assume a standard supervised model in a framework like PyTorch. Follow them in order to wire the Adam into a reliable training loop.

Step 1 – Prepare the model and data

Begin by defining your model and loading the data into batches. The model holds the trainable weights that the optimizer will adjust during training. Split the data into training and validation sets, often around an 80 to 20 ratio. Pick a loss function that matches the task, such as cross entropy for classification. Confirm that a single batch flows through the model without any shape errors at all. Pro tip: verify one forward pass on a tiny batch before launching a long run. A clean data pipeline prevents most of the mysterious training failures seen later on. Spending 10 minutes on these checks can save you hours of confused debugging.

Step 2 – Create the Adam optimizer

Instantiate the optimizer by passing the model parameters and a learning rate. The model parameters arrive through the parameters method, and the rate sets the step size. Keep the default betas of 0.9 and 0.999 unless you have a reason to change them. Leave epsilon at its default of 1e-8 for most standard models. Decide now whether you need plain Adam or the AdamW variant with weight decay. Choose AdamW whenever your model relies on weight decay for regularization. Recording these choices keeps your experiment reproducible from the very start.

Step 3 – Build the training loop

Write a loop that iterates over the training batches for each of your epochs. Inside the loop, zero the stored gradients before computing anything new at all. Run the forward pass to get predictions, then compute the loss against the targets. Call backward on the loss so the framework fills in all of the gradients. Then call the step method so the optimizer updates every one of the weights. Always zero the gradients first, or stale values will corrupt the next update. This four part rhythm repeats once for every batch across all of your epochs. A single misplaced call among these 4 steps can silently break the whole run.

Step 4 – Add a learning rate schedule

Layer a schedule on top of the base rate to improve final results. Add a short warmup phase that raises the rate over the first 500 steps or so. Follow that warmup with a gentle decay, using a cosine or a linear curve. Step the schedule once per batch or per epoch, matching its intended design. For transformer models, treat warmup as a requirement rather than an optional extra. Pro tip: log the current learning rate so you can confirm the schedule works. A well shaped schedule often matters as much as the optimizer itself does. Even a simple 2 phase warmup and decay can noticeably stabilize early training.

Step 5 – Monitor and checkpoint

Track the training and validation loss as the run progresses over time. Print or log the loss every 50 steps or so, so divergence shows up early. Save a checkpoint of the model and optimizer state at regular intervals. The saved optimizer state preserves the moment estimates for a clean resume later. Watch the validation curve to catch overfitting before it wastes more compute time. Save the optimizer state, not just the weights, so that training resumes correctly. Reliable monitoring turns a fragile run into a dependable, repeatable process. Keeping the last 3 checkpoints is usually enough to recover from a bad run.

Step 6 – Tune and iterate

Once a baseline trains cleanly, sweep the learning rate to improve it further. Try 3 or 4 values spanning an order of magnitude around the default rate. Compare the validation loss curves and keep the rate with the best result. Adjust betas or epsilon only if training still looks unstable afterward in testing. Consider switching to AdamW if regularization seems too weak during your testing. Change one setting at a time so each result stays easy to interpret. Disciplined iteration converts a working loop into a strong final model over time. A handful of careful runs usually beats dozens of random, unguided experiments.

Key Insights

• With over 100,000 citations recorded on the arXiv record for Adam, it ranks among the most cited works in modern machine learning.
• The defaults of beta 1 at 0.9, beta 2 at 0.999, and epsilon at 1e-8 are listed in the torch.optim.Adam documentation and rarely need changing.
• By decoupling weight decay, AdamW improved test error by up to 15 percent relative, a gain the Decoupled Weight Decay Regularization paper measured on image tasks.
• The 175 billion parameter GPT-3 model, as the GPT-3 research paper documents, lowered beta 2 to 0.95 for stable training at scale.
• Adam stores both a first and second moment per weight, and the Keras Adam documentation shows this roughly triples optimizer memory versus SGD.
• A convex counterexample in On the Convergence of Adam and Beyond shows Adam can fail to converge, which motivated the AMSGrad fix.
• On a separable task, the study The Marginal Value of Adaptive Gradient Methods found adaptive methods near 50 percent test error while SGD reached zero.

Taken together, these findings paint a balanced picture of a remarkable tool. The Adam earns its popularity through fast, reliable convergence with very little tuning. Its defaults, drawn from the source papers, work across an enormous range of models. The AdamW fix and the AMSGrad fix show how careful analysis keeps improving the method. The generalization gap and the memory cost mark the real edges of its strength. Used with that awareness, Adam remains the sensible default for most deep learning work.

Dimension	SGD	Momentum	RMSprop	Adam	AdamW
Adaptive learning rate	No	No	Yes	Yes	Yes
Momentum term	No	Yes	No	Yes	Yes
Per-parameter scaling	No	No	Yes	Yes	Yes
Optimizer memory cost	Low	Medium	Medium	High	High
Typical tuning effort	High	Medium	Medium	Low	Low
Generalization tendency	Strong	Strong	Mixed	Mixed	Improved
Weight decay handling	Clean	Clean	Distorted	Distorted	Decoupled
Best-fit use case	Tuned vision	Vision	Recurrent	General default	Transformers

Adam in Practice

Adam in the Original 2014 Benchmarks

In the original 2014 study, Kingma and Ba trained logistic regression and small neural networks. They ran the Adam against AdaGrad, RMSprop, and plain SGD on MNIST and CIFAR-10. Adam reached low training loss in fewer iterations, a measurable reduction in training cost. The advantage held across both the convex models and the deeper networks they tested. One clear limitation is that these benchmarks were tiny next to the models trained today. The competing methods still required careful learning rate tuning to compete at all. The full protocol appears in the original Adam paper by Kingma and Ba.

GPT-3 Trained With Adam at Scale

When OpenAI trained the 175 billion parameter GPT-3 model, it relied on the Adam. The team set beta 1 to 0.9 and lowered beta 2 to 0.95 for stability at scale. They paired Adam with gradient clipping and a warmup schedule across the enormous training run. The measurable outcome was a steady increase in capability across many language benchmarks. One limitation was the immense compute and memory the run required, costing millions of dollars. The optimizer state alone added billions of extra values to track during training. These settings are documented in the GPT-3 research paper from OpenAI.

BERT Pretraining With Adam and Warmup

Google’s BERT language model was pretrained using the Adam with weight decay. The team trained it with a learning rate near 1e-4, plus warmup and a decay schedule. They ran the optimization across billions of words from books and online text. The measurable outcome was a large increase in scores across the GLUE benchmark suite. One limitation was the heavy compute, since pretraining still required days on many accelerators. The recipe leaned on Adam style updates rather than plain gradient descent throughout. The configuration is detailed in the BERT research paper from Google.

Lessons From Real Adam Optimizer Deployments

Case Study: AMSGrad and the Convergence Fix

The convergence behavior of Adam was more than a theoretical curiosity for researchers. Reddi and colleagues faced a clear challenge, since Adam could fail to converge on a simple convex task. Their 2018 analysis built an online problem where the exponential average discarded large, rare, informative gradients. The solution, AMSGrad, kept the running maximum of the second moment to preserve that information. This change delivered a clear reduction in worst case regret, closing a gap plain Adam left open. One limitation is that AMSGrad often showed little improvement on real networks, so its benefit stayed contested. Many teams still adopted it only when a strict guarantee mattered for their work. The argument, laid out in On the Convergence of Adam and Beyond, still shapes this debate today.

Case Study: Tuned SGD Versus Adam in Vision

A second deployment question is whether adaptive methods truly generalize well in production. Wilson and colleagues faced the problem that Adam reached low training loss but tested noticeably worse. They needed to know if the convenience of Adam came at a hidden accuracy cost. They adopted tuned SGD with momentum as the baseline and compared it across several supervised problems. On one separable task, adaptive methods reached test error near 50 percent while SGD hit zero. The practical impact was a strong case for tuned SGD on certain vision benchmarks. One limitation is that the harshest example is illustrative, and modern language models still rely on Adam. The full comparison, detailed in The Marginal Value of Adaptive Gradient Methods, still informs vision practice.

Case Study: AdamW and the Weight Decay Fix

Transformer teams faced a stubborn problem, since weight decay in plain Adam regularized far too weakly. The adaptive scaling distorted the L2 penalty differently for every parameter in the network. Loshchilov and Hutter developed AdamW, which decoupled weight decay from the adaptive gradient step. Teams then adopted AdamW across large language and vision models almost universally. The measurable impact reached up to a 15 percent relative reduction in test error on image tasks. That gain made AdamW the default for training modern transformer systems everywhere. One limitation is that the extra moment estimates still required roughly triple the optimizer memory of SGD. The decoupling fix, introduced in Decoupled Weight Decay Regularization, now underpins transformer training.

Common Questions About the Adam Optimizer