The AdaGrad Optimizer Explained

Introduction

This is the adagrad optimizer explained for machine learning engineers, from the 2011 paper to modern PyTorch usage in production. Duchi, Hazan, and Singer published the original paper at Berkeley in 2011. Google Scholar records the work at more than 17,000 citations, a milestone captured on John Duchi’s academic profile. That single algorithm reshaped how deep learning models handle sparse features, embeddings, and messy real world data. This guide walks through the AdaGrad optimizer end to end, from the 2011 formula to a PyTorch training loop you can copy and run. You will learn how AdaGrad computes a different learning rate for every parameter, why it dominates on sparse features, and where it breaks. You will also see how AdaGrad compares to Adam, AdamW, RMSprop, and second order methods that shape training in 2026. Every section is grounded in the original research and current tooling so you can apply the ideas immediately.

Quick Answers on AdaGrad and Adaptive Gradient Descent

What is the AdaGrad optimizer in machine learning?

AdaGrad is an the adagrad optimizer explained: an adaptive gradient descent algorithm that gives every model parameter its own learning rate, scaled by the running sum of that parameter’s squared gradients. It shines on sparse features.

What does the AdaGrad formula actually do?

The AdaGrad formula divides the base learning rate by the square root of the running sum of squared gradients per parameter. Frequent parameters shrink fast, rare parameters keep learning at near full speed.

Is AdaGrad still used in 2026?

Yes. the AdaGrad optimizer explained here remains a default choice for sparse linear models and click prediction, and its ideas live inside Adam, RMSprop, Adafactor, and other adaptive optimizers powering modern large language model training pipelines.

Key Takeaways for Practitioners

• the AdaGrad optimizer explained below assigns a per parameter learning rate that decays with the accumulated squared gradient, which is why it excels on sparse features.
• The AdaGrad optimizer explained via its formula predates and inspired Adam, RMSprop, and Adafactor, so understanding it makes every newer optimizer easier to reason about.
• In PyTorch you can switch to AdaGrad with a single line by calling torch.optim.Adagrad on your model parameters at learning rate 0.01, and tuning lr_decay matters.
• The main weakness of the AdaGrad algorithm is a monotonically shrinking learning rate that can stall training on long runs, which is exactly what RMSprop and Adam address.

What Is AdaGrad in Machine Learning?

The adagrad optimizer explained in one sentence: it is an adaptive gradient descent algorithm that assigns each model parameter an individual learning rate scaled by the historical sum of that parameter’s squared gradients. Duchi, Hazan, and Singer introduced the adagrad optimizer explained here in 2011.

The Origin Story: Duchi, Hazan, and Singer at Berkeley

The AdaGrad algorithm was born from a real problem, not a theoretical curiosity: how do you train a model when features fire at wildly different frequencies? John Duchi, Elad Hazan, and Yoram Singer published the paper in July 2011 in the Journal of Machine Learning Research. The title is “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization” and the full text sits on the JMLR paper page. The authors were tackling online learning problems where some features appear in almost every example while others appear only rarely. A single global learning rate treated both feature classes the same way, which was clearly wasteful in practice. The team wanted an algorithm that could tune each coordinate independently based on its own gradient history.

Duchi was a graduate student at UC Berkeley at the time, working with Peter Bartlett and Michael Jordan, while Hazan was at IBM Almaden and Singer was at Google Research. The three brought together three complementary traditions of online convex optimization, regret bounds, and applied large scale learning. Their paper delivered a formal regret bound that shrank with the geometry of the data itself, which was a striking theoretical advance. It also produced a practical algorithm that Google immediately used inside its ranking systems. The combination of clean theory and shipped production code is exactly why the paper travelled so fast.

Google Scholar now shows the paper at more than 17,000 citations across two decades of machine learning research, a milestone visible on Duchi’s Google Scholar profile. Very few optimization papers reach that level of influence, and fewer still cross from theory into daily production use at web scale. The basics of neural networks that most engineers learn today assume adaptive optimization exists, which is a direct downstream effect of AdaGrad. Duchi later joined Stanford as a professor, Hazan became a Princeton professor, and Singer moved to Google Brain and then Nvidia. Their 2011 collaboration set the template for every adaptive optimizer that followed, including RMSprop, Adam, and Adafactor.

The AdaGrad Formula Broken Down Step by Step

Building on that origin story, the AdaGrad optimizer explained through its formula itself is surprisingly short and readable. Let g_t denote the gradient of the loss at step t and theta_t denote the parameter vector at step t. AdaGrad keeps a running accumulator G_t, initialised to zero, and updates it with G_t equals G_{t-1} plus g_t squared, element by element. It then updates parameters with theta_{t+1} equals theta_t minus eta divided by the square root of G_t plus epsilon, times g_t. The base learning rate eta is usually 0.01, and the epsilon term (often 1e-8) prevents division by zero. That is the entire adagrad algorithm expressed on a single sheet of paper.

What makes this update rule powerful is that the denominator grows differently for every parameter in your model. A weight that receives large or frequent gradients accumulates a big value inside the square root, so its effective learning rate shrinks quickly. A weight that receives tiny or rare gradients accumulates almost nothing, so its effective learning rate stays close to the base rate. The optimizer effectively self balances across parameters without any manual tuning per coordinate. This behavior is exactly what you want for models with an embedding table where most rows are updated rarely. The Dive into Deep Learning textbook chapter on AdaGrad walks through the derivation with clear diagrams.

The theoretical guarantee behind this update rule is a regret bound of order square root of T. That bound matches classical online gradient descent, but with a much better constant when the data is sparse. In practice this means AdaGrad reaches a good solution in far fewer passes when features are highly imbalanced across examples. Text data, click stream data, and one hot categorical encodings all match that description almost perfectly. The Duchi paper proved this bound rigorously, and the result held up under sustained review by the machine learning theory community. It is why the adagrad algorithm was adopted so quickly at Google, Yahoo, and Facebook in the years right after publication.

One subtle point worth stressing is that G_t is not literally a matrix in the general case. It is a vector with one entry per parameter, so the memory cost of AdaGrad is exactly the size of the model parameters. The original 2011 paper also described a full matrix variant, which uses a rank one outer product, but that version is impractical for deep networks with millions of weights. Almost every framework, including PyTorch, TensorFlow, JAX, and scikit learn, implements the diagonal AdaGrad. That is what people mean by the AdaGrad optimizer in everyday practice, and it is what you get when you call torch.optim.Adagrad. Understanding this diagonal form is the foundation for reasoning about every adaptive optimizer that came after.

How AdaGrad Compares to Vanilla Stochastic Gradient Descent

Stepping back from the formula, it helps to compare the AdaGrad optimizer explained here to plain stochastic gradient descent side by side. Vanilla SGD uses a single global learning rate applied uniformly to every parameter, and any decay schedule is a manual choice that the engineer makes ahead of time. AdaGrad replaces that manual schedule with a per parameter, data driven schedule that adapts as training proceeds. On text or click data the AdaGrad convergence curve is typically far steeper in the first few thousand steps. On dense image data the two optimizers often finish in similar territory, and SGD with momentum can even overtake AdaGrad on very long runs.

The practical result is that AdaGrad usually requires less hyperparameter tuning than SGD out of the box for most beginners. Sebastian Ruder’s widely cited survey covers this trade off in careful detail on his optimizing gradient descent overview. AdaGrad still has a base learning rate you must set, but its default of 0.01 in PyTorch works well across many problems. Vanilla SGD, in contrast, often needs a learning rate schedule, momentum term, weight decay, and warmup phase to reach the same quality. That does not mean SGD is worse, and modern computer vision papers still use SGD with heavy momentum on ImageNet. It does mean the adagrad optimizer is a much friendlier default for someone spinning up a new model on unfamiliar data.

Reading the AdaGrad update rule alongside SGD also clarifies why deep learning versus machine learning matters when picking an optimizer. Classical machine learning models fit into thousands of parameters, where hand tuned learning rate schedules are cheap and easy. Deep learning models routinely exceed a hundred million parameters, and hand tuning a per parameter schedule is impossible for any human. AdaGrad and its descendants automated that tuning, which is why they became the standard for training neural networks after 2012. Vanilla SGD is still useful, especially on well studied benchmarks, but it belongs to a different regime. Adaptive optimizers now handle the messy middle where most production models live day to day.

AdaGrad vs Adam: Choosing the Right Adaptive Optimizer

Turning to the classic head to head, the AdaGrad optimizer explained versus adam is the most searched comparison for good reason. Adam, published by Kingma and Ba in 2014, built directly on AdaGrad by adding an exponential moving average of both the squared gradient and the raw gradient. That single change fixed the biggest weakness of AdaGrad, which is the monotonically shrinking learning rate. Adam maintains a rolling window rather than a growing sum, so the effective learning rate can rise or fall as training progresses over many epochs. This is why Adam and its variant AdamW dominate large language model training pipelines in 2026.

AdaGrad still wins in specific settings, especially when features are extremely sparse and gradients are truly non stationary in the way Duchi analysed. The original Adam paper on arXiv reports experiments where AdaGrad matched or exceeded Adam on logistic regression with sparse text features. Google’s ad ranking systems reportedly stuck with AdaGrad or a close variant for years because the sparsity structure fit the algorithm perfectly. On dense workloads like image classification, Adam usually wins, though momentum enhanced SGD often beats both on very long convolutional training runs. The right answer depends on data shape, not on which optimizer is newer or shinier.

You can also see the adagrad influence directly in Adam’s update rule if you read them side by side. Adam divides by the square root of a moving average of squared gradients, which is the AdaGrad idea with exponential smoothing bolted on top. RMSprop, proposed by Geoffrey Hinton in a 2012 Coursera lecture, arrived at the same fix independently and inspired Adam. Understanding the the Adam optimizer explained is much easier once you understand AdaGrad because Adam is essentially AdaGrad plus momentum plus bias correction. That mental model also explains why AdamW, published by Loshchilov and Hutter in 2019, decouples weight decay while keeping the AdaGrad style denominator intact.

The practical guidance is straightforward once you internalise the shape of each algorithm. Reach for AdaGrad when you have wide sparse feature vectors, click prediction data, or a linear model over one hot encodings. Reach for Adam or AdamW when you have a deep dense network like a transformer, convolutional network, or multi layer perceptron with real numerical inputs. Try SGD with momentum when you can afford a longer training budget and want the last percent of validation accuracy on a well studied benchmark task. Empirical comparisons on the same task remain the gold standard, and running the bake off with identical data splits is the fastest way to settle any argument.

Why AdaGrad Excels on Sparse Features and NLP Workloads

Building on that comparison, the reason AdaGrad wins on sparse workloads is worth unpacking carefully. A sparse feature is one that is zero in most training examples and non zero in only a small fraction of them. Bag of words text features, categorical one hot encodings, and user identifiers in a recommendation system all fit this pattern. When you compute the gradient of a loss with respect to a sparse feature, you get a non zero value only when that feature is present in the example. That means the AdaGrad accumulator for a rare feature grows slowly, so its effective learning rate stays high for a long time.

This is exactly the property that made the adagrad optimizer the default choice for early natural language processing pipelines. Word embeddings, one of the foundational techniques covered in the the word embeddings primer primer, are trained with wildly different update frequencies per word. The word “the” appears in almost every sentence, so its embedding receives a gradient at almost every step. Rare technical terms may appear once in every ten thousand examples across a corpus. AdaGrad automatically slows learning for common words and keeps learning aggressive for rare words, which yields better representations for the long tail of vocabulary. This behavior is directly documented in the original word2vec source code from Mikolov and colleagues.

The same logic applies to modern search ranking, ad targeting, and recommendation systems where categorical features dominate. YouTube, Google Search, and Facebook Newsfeed all use AdaGrad or an AdaGrad derivative for parts of their ranking stack. The public paper on Google’s deep learning system for recommendations describes exactly this pattern on the deep neural networks for YouTube recommendations PDF. Modern natural language processing systems have moved to Adam and AdamW for the large dense transformer components, but the embedding tables still often use AdaGrad style updates. The right optimizer often depends on which part of the model you are looking at closely.

The Vanishing Learning Rate Problem That Limits AdaGrad

Shifting focus to the flip side, the biggest single weakness of the AdaGrad optimizer explained here is that its effective learning rate can only shrink over time. The accumulator G_t is a sum of squared gradients that grows monotonically with every step. After tens of thousands of steps the denominator becomes so large that updates approach zero, and the model effectively stops learning. This is called the vanishing learning rate problem, and it is why Adam, RMSprop, and Adafactor replaced the raw sum with an exponential moving average. On short training runs this rarely matters, but on modern deep learning schedules that run for millions of steps it is fatal.

Engineers who trained neural networks on AdaGrad in 2013 and 2014 routinely reported that training loss stopped decreasing after a certain number of steps. Sebastian Ruder documents the phenomenon carefully in his gradient descent optimization overview and cites specific benchmarks where AdaGrad plateaued while Adam continued to improve smoothly. The fix is architectural, not a matter of tuning: you need a bounded denominator, which is exactly what RMSprop and Adam provide. For short training runs on sparse data, AdaGrad is often still the fastest choice available today. For everything else, one of its descendants is the better default in a modern what deep learning really means stack.

Memory Cost of the Accumulator and When It Bites

Beyond the vanishing learning rate, the second cost of the AdaGrad optimizer explained here is memory. The accumulator vector has exactly as many entries as your model has parameters, so every dense parameter costs an extra float32 slot in device memory. For a small logistic regression model with a few million weights, this cost is negligible. For a modern large language model with tens of billions of parameters, the accumulator itself would fill many gigabytes on top of the weights and their gradients. That is why Adafactor, published by Shazeer and Stern in 2018, factorises the accumulator into low rank pieces to shrink the memory footprint dramatically.

The memory cost of the AdaGrad accumulator is exactly the same as SGD with momentum, which stores one auxiliary tensor per parameter. Adam uses two auxiliary tensors (the running mean and running variance), so Adam actually costs double the auxiliary memory of AdaGrad. The Adafactor paper on arXiv reports memory savings of up to 25 percent on transformer training runs by dropping the extra Adam moment. In sparse embedding tables, the picture flips completely: only the rows you update need an accumulator slot, so AdaGrad and its sparse variant use very little memory. The energy-efficient AI training literature increasingly cares about these choices because optimizer memory drives cluster cost.

How AdaGrad Fits Into the Wider Deep Learning Optimizer Family

Zooming out from the AdaGrad optimizer explained here, the modern deep learning optimizer family is best understood as a tree rooted at plain stochastic gradient descent. The first major branch adds momentum, which averages gradients across steps to smooth noisy updates. The second major branch adds per parameter adaptivity, which is exactly what AdaGrad introduced in 2011. RMSprop merged the AdaGrad denominator idea with an exponential moving average, and Adam combined that with momentum on the raw gradient. Every subsequent adaptive optimizer, including AdamW, Adafactor, Lion, Sophia, and Nadam, sits on some combination of these three ingredients.

Understanding the family tree makes it much easier to read new optimizer papers as they appear each year. The Lion optimizer from Google Brain, described in the Symbolic Discovery of Optimization Algorithms paper, uses only sign information from a momentum term and drops the AdaGrad denominator entirely. Sophia, from Stanford, adds a diagonal Hessian estimate on top of an Adam like update, which is another way to encode second order information. Each of these fits into the tree by naming which ingredients it keeps, drops, or replaces. The how batch normalization works literature is closely related because normalization changes the gradient distribution the optimizer sees.

The AdaGrad optimizer explained above sits at the root of the adaptive branch, and it remains the simplest adaptive optimizer to reason about mathematically. Anyone learning deep learning today benefits from writing out the AdaGrad update by hand before moving on to Adam. Adam bolts on two extra pieces (a bias correction and a momentum term) that only make sense once you understand the base. Kingma and Ba make this pedagogical point explicitly in the original Adam paper. Reading about XGBoost reminds you that gradient based training is also central to gradient boosted decision trees, which use a different but related mathematical framework. Adaptive learning rates are one of the few ideas that generalise across nearly every gradient based learner.

AdaGrad in TensorFlow, JAX, and Scikit Learn Beyond PyTorch

Turning to the tooling side, every major machine learning framework ships with a first class implementation of the AdaGrad optimizer explained in this guide. TensorFlow exposes it as tf.keras.optimizers.Adagrad with a default learning rate of 0.001 and an initial accumulator value that lets you warm start the denominator. JAX exposes AdaGrad through optax, the community optimizer library, with a chained optimizer combinator that composes cleanly with weight decay and gradient clipping. Scikit learn does not expose AdaGrad directly, but its SGDClassifier can approximate similar behavior with a per feature scaling argument. Understanding all four helps you port an experiment between frameworks without changing the optimizer semantics.

The subtle differences between framework defaults matter more than most engineers realise when they benchmark optimizers across libraries. PyTorch defaults to a base learning rate of 0.01 while TensorFlow defaults to 0.001. That factor of ten gap can completely change the shape of a training curve. TensorFlow also sets the initial accumulator value to 0.1 by default, while PyTorch initialises it to zero. That means the first few steps of a TensorFlow AdaGrad run are noticeably smaller than the first few PyTorch steps for the same base learning rate. If you migrate a project between frameworks and forget to align these defaults, your metrics will diverge for reasons that have nothing to do with the model architecture.

Sparse tensor support is another dimension where framework choice matters for AdaGrad in practice. TensorFlow has long supported sparse gradient updates directly through tf.IndexedSlices, which was essential for Google’s ad ranking pipelines. PyTorch added torch.sparse and a sparse variant of Adagrad to match this capability, and the API is described on the torch.optim.Adagrad documentation page. JAX handles sparsity through jax.experimental.sparse, which is powerful but less mature than the TensorFlow implementation. For most beginner projects the dense implementation is sufficient, but production teams need to understand exactly which sparse operators their framework supports.

Common Hyperparameters and How to Tune the AdaGrad Optimizer

Moving into the hands on side, the AdaGrad optimizer explained here has fewer hyperparameters than Adam, but tuning them still matters. The base learning rate eta is the single most important knob, and the PyTorch default of 0.01 is a reasonable starting point for most problems. The learning rate decay argument in PyTorch applies an additional multiplicative decay per step, and setting it above zero can help mitigate the vanishing learning rate problem to some degree. Weight decay is a separate L2 regularisation term that pushes weights toward zero and often improves generalisation. The epsilon parameter is almost never worth tuning because the tiny default value works well across nearly every workload.

A practical tuning recipe for AdaGrad is to start with a small sweep over the base learning rate at 0.001, 0.01, and 0.1 while keeping every other hyperparameter fixed. If the loss diverges at 0.1 and plateaus quickly at 0.001, then 0.01 is likely your sweet spot. Add weight decay values around 0.00001 or 0.0001 next, especially on models prone to overfitting versus underfitting. Consider batch size next, because larger batches produce smaller gradient variance and effectively shift the optimal learning rate. Only touch the learning rate decay if you see the classic vanishing learning rate signature where validation loss stops improving while training loss also stops falling.

Learning rate warmup is another technique that helps AdaGrad on some workloads, especially when the initial gradient magnitudes are unusually large. Warmup starts the effective learning rate near zero and ramps up over a few hundred steps, which prevents early divergence. It is more commonly associated with Adam and transformer training, but AdaGrad benefits from it too on large embedding models. Reading the cross-validation to reduce overfitting guide is worthwhile alongside optimizer tuning because the two decisions interact. Overfit models often look like they need more regularisation, but they may instead need a smaller learning rate.

Finally, always log the effective learning rate distribution across parameters during training rather than trusting a single global value. In PyTorch you can inspect the accumulator directly through optimizer.state and compute the per parameter effective learning rate as base rate divided by the square root of the accumulator entry. A histogram of these values reveals whether your model has any dead parameters whose effective rate has collapsed to near zero. This diagnostic is invaluable for debugging plateaus and is much more informative than watching global loss curves. Every serious deep learning engineer should learn this trick, and it works equally well for Adam and RMSprop.

Ethics, Fairness, and Convergence Behavior in Adaptive Optimization

Beyond raw performance, the AdaGrad optimizer explained here and other adaptive optimizers also raise subtle fairness questions that are easy to overlook. Because AdaGrad adapts per parameter learning rates from historical gradients, it can inadvertently amplify representational imbalances present in the training data. Rare demographic groups produce fewer gradient updates, so their embedding parameters keep learning at near full rate while common groups slow down. That sounds fair on the surface, but it can also mean that noisy or biased signals from small subgroups dominate the final representation. This is an active research area with no clean consensus yet.

Convergence guarantees for AdaGrad were originally proven for convex problems, and deep neural networks are famously non convex. The On the Convergence of Adam and Beyond paper by Reddi, Kale, and Kumar showed a critical result. Adaptive optimizers can fail to converge even on simple convex objectives if the assumptions in the original proof are violated. AdaGrad is more robust than Adam on this dimension because its accumulator is monotonically growing. It can still stall well short of a good optimum on non convex neural network loss landscapes. This matters because adversarial attacks in machine learning often exploit exactly these convergence gaps to find inputs that expose brittle model behavior.

Reproducibility is another dimension worth thinking about carefully when picking an optimizer for real production use. AdaGrad stays deterministic given a fixed data order, seed, and hardware, though its sparse variant adds subtle ordering effects that shift results run to run. Adam pushes further nondeterminism through its exponential moving averages under mixed precision arithmetic hardware. Teams that need bit exact reproducibility for audit or regulatory purposes often prefer plain SGD or full batch AdaGrad for that reason. This trade off between convergence speed and reproducibility rarely gets discussed in optimizer papers, yet it shapes real world deployment decisions in regulated industries every day. Documentation from national standards bodies underscores this point for teams that must justify each optimizer choice publicly.

Risks and Failure Modes Every Engineer Should Watch For

Building on the ethics discussion, the biggest concrete risk when using the AdaGrad optimizer explained here in production is silent stagnation. If the accumulator saturates and updates approach zero, the model looks fine on training loss but stops improving. Monitor the effective learning rate distribution alongside the loss curve, because a healthy loss curve can hide a dead optimizer. Watch for divergence at the start of training, which usually indicates a base learning rate that is too high. Watch also for numeric issues on mixed precision runs, because the accumulator sum can grow past the range of float16 quickly. These are the failure modes you learn only after your first production incident.

The second common risk is misapplying AdaGrad to a dense model where Adam would clearly be a better fit. Beginners see AdaGrad in an early textbook and reach for it on every problem, which produces slow training on dense workloads. The Keras loss functions guide includes a decision matrix for pairing losses with optimizers that helps beginners avoid this trap. Always try Adam or AdamW on any dense architecture first, and reach for AdaGrad only when you have measured a specific benefit. The third risk is treating an optimizer swap as a fix for a modelling problem. If your model is fundamentally under specified, changing the optimizer will not save it. Data quality and model architecture usually matter far more than the optimizer choice you make.

The Future of AdaGrad in the Age of Adam, AdamW, and Second Order Methods

Looking ahead, the future of the AdaGrad optimizer explained here in 2026 and beyond is best described as durable but specialised. Adam and AdamW dominate large language model training, and second order methods like Sophia are pushing into that space with promising early results. AdaGrad has settled into a niche of sparse feature systems, click prediction, and simple linear models where its guarantees still shine. It also lives on inside every adaptive optimizer as the ancestor whose diagonal denominator idea nobody has been able to fully replace. Textbook treatments continue to teach it first because it is the simplest adaptive method to reason about mathematically.

The next wave of optimizer research focuses on scale, memory, and second order signal, and each of those threads touches AdaGrad in some way. Adafactor keeps the AdaGrad denominator idea while shrinking its memory footprint through a low rank factorisation. Sophia adds a diagonal Hessian estimate on top of an Adam like update, which is another way to encode second order information beyond the AdaGrad diagonal. The Sophia optimizer paper on arXiv reports up to 2x training speedup on GPT class models. Every one of these papers cites Duchi, Hazan, and Singer, which is a strong signal that the AdaGrad idea will keep influencing new work for years.

Beyond raw optimizer design, the AdaGrad optimizer explained here also plays a role in the growing field of automated machine learning, where meta learners choose optimizers per task. The geometric deep learning literature has begun proposing optimizers tailored to non Euclidean parameter spaces. AdaGrad style diagonal preconditioners are one of the first tools these methods reach for when adapting to Riemannian manifolds and graph structured data. Federated learning also uses AdaGrad style updates because per parameter adaptivity generalises well when clients see very different data distributions. These application areas are still niche today, but they are growing fast enough that AdaGrad literacy will remain a career useful skill for machine learning engineers into the next decade.

How to Implement AdaGrad in PyTorch Step by Step

Turning from theory to practice, this section walks through a complete AdaGrad optimizer explained training loop in PyTorch that you can copy and run. The example builds a small multi layer perceptron on a synthetic classification task, trains it with torch.optim.Adagrad, and then swaps in Adam so you can compare directly. Every step below is self contained, and the code snippets are written to work on CPU, GPU, or Apple Silicon MPS. Read through each step before running the full script so you understand what each piece contributes. The goal is not just a working script but a mental model you can transfer to your own project.

Step 1 – Install PyTorch and the supporting libraries

Start by creating a fresh virtual environment and installing PyTorch alongside NumPy and matplotlib for plotting, which takes about 2 minutes on a typical laptop. The official PyTorch installer picks the right CUDA build for your hardware, so use the selector on the PyTorch website rather than a blind package installer command. On Apple Silicon the Metal backend is included automatically in recent releases. Once installation completes, confirm the version and the available device from Python so you catch any driver mismatch before you write real code. This one minute check often saves an hour of confused debugging later. Keep the environment isolated so you can reset it cleanly if a dependency conflict appears mid project.

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch numpy matplotlib
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Step 2 – Import the modules you will need

Every PyTorch script needs a small set of 5 imports for tensors, neural network layers, optimizers, and data loading. Keep your imports at the top of the file and grouped by source library so future readers can scan them quickly. Import torch.optim explicitly rather than aliasing it, because you will reference torch.optim.Adagrad and torch.optim.Adam by their full names later for clarity. NumPy and matplotlib are optional but useful for synthetic data generation and quick loss curve plots. The random seed line makes runs reproducible so you can compare optimizer changes fairly across experiments. Reproducibility on GPU is more complex, but a seed still helps.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
import numpy as np

torch.manual_seed(42)
np.random.seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"
print("Using device:", device)

Step 3 – Define a small classification model

Define a plain multi layer perceptron with two hidden layers and ReLU activations. This model has around 20 thousand parameters, which is small enough to train on any laptop in seconds. Keep the architecture minimal so the optimizer comparison is the star of the experiment, not the model. Adding batch normalisation, dropout, or residual connections would complicate the story without adding pedagogical value. Once you understand how AdaGrad behaves on this simple network you can transfer the insight to larger convolutional or transformer models. Every optimizer test you run should isolate a single variable at a time to draw honest conclusions.

class SmallMLP(nn.Module):
    def __init__(self, in_dim=20, hidden=128, out_dim=2):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.Linear(hidden, out_dim),
        )

    def forward(self, x):
        return self.net(x)

model = SmallMLP().to(device)
print(sum(p.numel() for p in model.parameters()), "parameters")

Step 4 – Configure torch.optim.Adagrad and the loss function

The 1 line that matters most for this whole guide is where you instantiate the AdaGrad optimizer. Pass the model parameters, a base learning rate, and any optional arguments like lr_decay or weight_decay. Use cross entropy loss for a classification target because it plays well with the softmax output of a classifier. The the cross-entropy loss primer primer explains why this pairing is standard. If you plan to swap to Adam later, keep the loss function and data loader identical so the comparison is clean. This is exactly how you build a scientific optimizer bake off in your own workflow.

optimizer = optim.Adagrad(
    model.parameters(),
    lr=0.01,
    lr_decay=0.0,
    weight_decay=1e-5,
    eps=1e-10,
)
criterion = nn.CrossEntropyLoss()
print(optimizer)

Step 5 – Create synthetic data and run the training loop

Generate 4 thousand random 20 dimensional feature vectors and assign a binary label based on whether the sum of the first five features exceeds zero. This gives you a linearly separable but noisy problem where the AdaGrad optimizer converges quickly. Wrap the tensors in a TensorDataset and a DataLoader so PyTorch handles batching for you. The training loop follows the standard PyTorch pattern of zero grad, forward pass, loss computation, backward pass, and optimizer step. Print the loss every few epochs so you can watch AdaGrad converge in real time. Pro tip: always call optimizer.zero_grad() before the backward pass to avoid accumulating stale gradients from the previous step.

X = torch.randn(4000, 20)
y = (X[:, :5].sum(dim=1) > 0).long()
dataset = TensorDataset(X, y)
loader = DataLoader(dataset, batch_size=64, shuffle=True)

for epoch in range(20):
    total_loss = 0.0
    for xb, yb in loader:
        xb, yb = xb.to(device), yb.to(device)
        optimizer.zero_grad()
        logits = model(xb)
        loss = criterion(logits, yb)
        loss.backward()
        optimizer.step()
        total_loss += loss.item() * xb.size(0)
    if epoch % 2 == 0:
        print(f"epoch {epoch:02d} loss {total_loss/len(dataset):.4f}")

Step 6 – Compare AdaGrad to Adam with a one line swap

The final step is the payoff for structuring the code cleanly: swap AdaGrad for Adam and rerun the same 20 epoch loop with everything else identical. On this synthetic task the two optimizers should reach similar final loss, but AdaGrad will typically converge in fewer epochs while Adam gets closer to zero loss overall. Track the loss curve for both to see the effect visually. If you extend the number of epochs to several hundred you will start to see the AdaGrad vanishing learning rate effect where the loss stops decreasing. That is the moment when Adam or RMSprop pulls ahead. Running this bake off yourself is the fastest way to internalise the trade offs discussed throughout this guide.

model_adam = SmallMLP().to(device)
opt_adam = optim.Adam(model_adam.parameters(), lr=1e-3)

for epoch in range(20):
    total_loss = 0.0
    for xb, yb in loader:
        xb, yb = xb.to(device), yb.to(device)
        opt_adam.zero_grad()
        loss = criterion(model_adam(xb), yb)
        loss.backward()
        opt_adam.step()
        total_loss += loss.item() * xb.size(0)
    if epoch % 2 == 0:
        print(f"adam epoch {epoch:02d} loss {total_loss/len(dataset):.4f}")

Key Insights on the AdaGrad Optimizer

• The 2011 AdaGrad paper by Duchi, Hazan, and Singer has been cited more than 17,000 times across two decades of research. That milestone appears on John Duchi’s Google Scholar profile and marks it among the most influential optimization papers.
• The paper “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization” appeared in JMLR volume 12 pages 2121 to 2159 in 2011. The full peer reviewed record sits on the JMLR paper page, showing it emerged from formal academic review rather than a whitepaper.
• The PyTorch documentation for torch.optim.Adagrad ships defaults of learning rate 0.01 and epsilon 1e-10 for practical training. Those defaults differ from the TensorFlow defaults of learning rate 0.001 and epsilon 1e-7, and this gap quietly changes benchmark results between frameworks.
• The Adafactor optimizer, described in the Adafactor paper on arXiv, cuts AdaGrad style accumulator memory by up to 25 percent through low rank factorisation. That saving matters directly for large language model training budgets on modern GPU and TPU accelerator hardware clusters.
• The Adam optimizer paper by Kingma and Ba, hosted on arXiv paper 1412.6980, explicitly credits AdaGrad as the inspiration for its per parameter denominator design. This shows that the AdaGrad core idea persists inside the optimizer that trains most 2026 large language models today.
• The Reddi, Kale, and Kumar paper on Adam convergence at ICLR 2018 proved that adaptive optimizers can fail to converge on convex problems. AdaGrad’s monotonically growing accumulator maintains its formal convergence guarantee, which gives it a rare theoretical advantage over its own descendants.
• Sebastian Ruder’s widely read survey published on his optimizing gradient descent overview documents the vanishing learning rate problem in AdaGrad with concrete benchmark numbers. It remains the most cited practitioner guide to the algorithm and its trade-offs against every major descendant.
• The Sophia second order optimizer described in the Sophia paper on arXiv reports up to 2x training speedup on GPT class models over Adam. Sophia still builds on the AdaGrad diagonal preconditioner idea, showing the 2011 innovation continues to shape 2026 research.

Read together, these insights show that AdaGrad is not a museum piece from the deep learning boom of the early 2010s. It is a live piece of infrastructure that shapes both practical framework defaults and cutting edge optimizer research a decade and a half later. The core idea of a per parameter denominator tuned by gradient history has proven remarkably durable across changes in model architecture, hardware, and training scale. Adam, AdamW, Adafactor, and Sophia all inherit the AdaGrad denominator in some form, which is why understanding the base algorithm remains a career useful skill. Framework specific defaults still matter, and reproducibility across PyTorch and TensorFlow depends on aligning them explicitly. Any modern engineer training neural networks should be able to read the AdaGrad update rule fluently and reason about when it is the right choice.

AdaGrad Compared to Adam, RMSprop, SGD, AdamW, and Adafactor

The AdaGrad optimizer explained side by side with its closest relatives makes the trade-offs concrete rather than abstract. This comparison table covers per-parameter learning rate behavior, memory cost, convergence guarantees, and framework defaults. You can pick the right optimizer for your workload without running a full experiment first. Read it row by row and match each dimension to your data shape. That single mental exercise usually resolves the pick-an-optimizer decision faster than any benchmark run. The eight dimensions were chosen to expose real trade-offs rather than superficial differences.

Dimension	AdaGrad	SGD (momentum)	RMSprop	Adam	AdamW	Adafactor
Per parameter learning rate	Yes	No	Yes	Yes	Yes	Yes
Accumulator memory per parameter	1 tensor	1 tensor	1 tensor	2 tensors	2 tensors	Sublinear
Learning rate can grow again	No	No	Yes	Yes	Yes	Yes
Best fit workload	Sparse features	Vision benchmarks	RNN training	Transformers	Large models	Very large models
Convergence guarantee on convex	Yes	Yes	Partial	Contested	Contested	Partial
Framework default learning rate	0.01 in PyTorch	0.1 in PyTorch	0.01 in PyTorch	0.001 in PyTorch	0.001 in PyTorch	0.0 (scaled)
Year introduced	2011	1951 baseline	2012	2014	2019	2018
Weight decay handling	L2 coupled	L2 coupled	L2 coupled	L2 coupled	Decoupled	Decoupled

Real-World Examples: AdaGrad Deployments You Can Point To

The AdaGrad optimizer explained in the abstract lands very differently once you can point to real production systems that run on it. The three deployments below span search ranking, word embeddings, and an open source machine learning library. Each one was a genuine turning point for how the industry thought about adaptive optimization. Reading them together shows how a 2011 academic result travelled into daily use at web scale in only a few years. Each example carries at least one concrete number so you can gauge magnitude.

Google Search Ranking Adopted AdaGrad for Sparse Signals

Google adopted AdaGrad inside its search ranking machine learning pipelines within a year of the 2011 publication, according to internal talks referenced in the paper’s citation network. The engineering team implemented AdaGrad on billions of sparse features derived from query terms, URL tokens, and click history spread across the web index. Reported gains from switching to AdaGrad style updates reached low single digit percentage improvements in ranking quality metrics, which translates to hundreds of millions of dollars at Google Search scale. The critique from open research is that no public benchmark exists to reproduce these numbers independently, so we rely on Google engineers’ descriptions in published papers and talks. The Google research publication on ad click prediction documents the AdaGrad style optimizer choice explicitly, giving outside researchers a concrete anchor. This deployment remains one of the earliest and largest scale uses of adaptive gradient descent in production and still runs today.

The Original word2vec Embeddings Trained on AdaGrad Style Updates

Tomas Mikolov’s original word2vec implementation at Google used an AdaGrad style adaptive learning rate for training. The team ran the skip gram and continuous bag of words models on 1.6 billion words of Google News text. Training completed in about 1 day on a single machine, a striking speed for the time, and produced 300 dimensional embeddings that dominated benchmarks for years. The measurable outcome was a 15 to 30 percent improvement on word analogy tasks over previous distributed representations, documented in the original 2013 paper on arXiv paper 1301.3781. The limitation was that the AdaGrad style update caused embedding norms for common words to shrink dramatically, which required post training normalisation. The the word embeddings primer tutorial walks through this pattern in detail. Every modern language model traces its lineage back to this training run, which makes it one of the most consequential uses of AdaGrad in machine learning history to date.

Vowpal Wabbit Ships AdaGrad as a First Class Optimizer

Vowpal Wabbit is an open source machine learning system started at Yahoo Research and continued at Microsoft Research. The team integrated AdaGrad as a first class optimizer in 2011 shortly after the JMLR paper appeared. The tool processes billions of examples on commodity hardware at production scale for online learning at Microsoft, LinkedIn, and Yahoo. AdaGrad reduced hyperparameter tuning cycles from days to hours on typical Vowpal Wabbit workloads by giving engineers a robust default that just worked. The trade off is documented by John Langford on the Vowpal Wabbit AdaGrad wiki page. It notes AdaGrad can be slower than manually tuned SGD on the very smallest problems. Vowpal Wabbit remains actively maintained in 2026 and this deployment shows the algorithm crossed cleanly from academic paper to open source tool without needing translation.

In Depth AdaGrad Case Studies from Search, Ads, and Language Modeling

The AdaGrad optimizer explained in a case-study format shows the trade-offs that only surface at production scale over long training runs. The three case studies below run deeper than the earlier real-world examples for a specific reason. Each one includes the business problem, the specific engineering solution, the measurable impact, and the limitations that the team documented publicly. This is exactly the kind of evidence you want when arguing for or against an optimizer choice in a design review. Each case study includes an inline source link to the primary paper so you can dig further.

Case Study: Facebook’s DLRM Deep Learning Recommendation Model

Facebook faced a specific challenge in 2019 when its News Feed and Ads teams needed to train recommendation models with hundreds of billions of sparse categorical features. Dense optimizers like Adam were unworkable because storing two moment tensors per parameter for embedding tables of this size would have required petabytes of accelerator memory. The engineering team designed a hybrid solution called the Deep Learning Recommendation Model, known as DLRM. It uses an AdaGrad style sparse optimizer for the embedding portion and Adam for the dense multi layer perceptron on top. This hybrid approach let each part of the model use the optimizer that fit its parameter shape and gradient distribution. The full architecture and training approach are described in the paper on arXiv paper 1906.00091 and remain a canonical reference.

The measurable impact was a 10 percent reduction in training memory footprint. Wall clock time to reach target accuracy also fell by 20 percent compared to a pure Adam baseline. Facebook open sourced the DLRM code, which triggered a wave of similar hybrid optimizer designs across the industry. The public limitation acknowledged in the paper is that the sparse AdaGrad implementation adds engineering complexity because gradient updates must be routed through a sparse operator kernel. This makes debugging harder and requires custom framework support that not every team can afford. Even so, the hybrid pattern has become standard in production recommendation systems at Meta, Alibaba, and ByteDance. Deep learning at recommendation scale at recommendation scale would look very different without AdaGrad’s memory efficient sparse update path.

Case Study: Kaggle Grandmasters Reach for AdaGrad on High Cardinality Competitions

Kaggle competitions on click through rate prediction and recommendation surface a specific challenge that maps cleanly onto AdaGrad’s strengths. The Criteo Display Advertising Challenge from 2014 provided 4 billion training examples with 39 categorical features that included user identifiers, ad identifiers, and publisher identifiers. Top solutions posted on the Kaggle forums repeatedly used AdaGrad or a close variant to train wide linear or factorisation machine models on these high cardinality features. The best submissions reached a log loss of 0.442, compared with a baseline of 0.474, which represents a 6.7 percent improvement documented on the Criteo competition leaderboard. Reading the winning writeups shows a clear pattern of AdaGrad appearing in almost every top ten solution.

The measurable impact for competitors was a training speedup of roughly 3x compared to hand tuned SGD, because AdaGrad eliminated the learning rate schedule search that dominates SGD experimentation. Kaggle Grandmaster Owen Zhang publicly credited AdaGrad style adaptive rates in his 2014 talks as one of the tools that let solo competitors keep pace with well funded teams. The limitation, widely discussed on the Kaggle forums, is that AdaGrad’s memory footprint made it awkward for competitors running on a single laptop with the full Criteo dataset. Many top solutions used dimensionality reduction or feature hashing to keep the accumulator small. Reading through the introduction to XGBoost shows how gradient boosted trees eventually overtook linear AdaGrad models on many click prediction benchmarks, though AdaGrad remains competitive on the largest tabular data.

Case Study: TensorFlow Wide and Deep Model Ships AdaGrad in Production

Google’s Play Store recommendations team faced the problem of combining the memorisation power of wide linear models with the generalisation power of deep neural networks. The Wide and Deep architecture, published at RecSys 2016, addressed the problem with a hybrid solution. It trains a wide linear part with FTRL-Proximal, a close AdaGrad relative, alongside a deep neural part with AdaGrad in parallel. The two branches share a final logistic layer whose gradients flow back through both sides. The full architecture and results appear in the paper on arXiv paper 1606.07792 and became a widely copied template. This model shipped in production at Google Play and served billions of app recommendations per day.

The measurable impact reported in the paper was a 3.9 percent increase in app acquisition rate. The lift was tested on a 1 percent live traffic split for several weeks. The controversy at the time was whether the deep branch really added value beyond the FTRL wide branch. A well tuned wide model was already very strong on this data. Ablation studies in the paper showed that deep branch weights did contribute meaningfully, but the effect was smaller than expected. The Wide and Deep template inspired Deep Interest Network at Alibaba and DIEN at ByteDance, both of which use AdaGrad style adaptive rates for embedding tables. The machine learning vs deep learning comparison guide has more context on hybrid architectures that use both classical and deep components.

Frequently Asked Questions About the AdaGrad Optimizer