How to Measure Developer Productivity Gains From AI Coding Tools

Introduction

Learning how to measure developer productivity gains from AI coding tools is urgent because 84 percent of developers use or plan to use AI tools in the latest large survey. Most engineering leaders already pay for licenses, yet few can say whether those licenses shorten delivery or simply move effort from writing code to reviewing it. The honest answer is that the evidence is mixed, and the mix depends on what a team chooses to count. Controlled experiments have reported speedups above 50 percent on small tasks and slowdowns of 19 percent on mature codebases. Activity counts such as suggestions accepted or pull requests opened rise quickly after rollout, but they say little about whether customers receive working software sooner. This guide explains a practical approach built on baselines, balanced metrics, and short feedback loops. By the end you will know which numbers to trust, which to treat with suspicion, and how to run a defensible evaluation in your own organization.

Quick Answers on Measuring Developer Productivity Gains From AI Coding Tools

What is the best way to measure developer productivity gains from AI coding tools?

Combine a pre-rollout baseline with delivery metrics, quality signals, and developer surveys, then compare AI-assisted work against a matched group. No single number, such as lines of code or acceptance rate, captures real productivity gains.

Which metrics matter most when evaluating AI coding assistants?

Track cycle time, review time, change failure rate, rework, and developer satisfaction together. The SPACE framework recommends at least three metrics from two or more dimensions so that no single measure can be gamed.

How long does it take to see measurable gains from AI coding tools?

Expect a usable signal about AI coding tools after four to eight weeks of steady use on a stable team, because early weeks include learning costs. Effects on quality and stability need longer windows to appear.

Key Takeaways for Measuring AI Coding Productivity

• Capture a baseline of cycle time, review time, and defect rates before any rollout, or every later number will lack a reference point.
• Treat suggestion counts, accepted lines, and raw pull request volume as activity signals rather than proof of productivity.
• Pair system telemetry with short developer surveys, because controlled studies show that people often misjudge how AI changes their speed.
• Watch review load, rework, and stability alongside speed, since faster code creation can shift the bottleneck downstream.

What Is AI Coding Productivity Measurement?

How to measure developer productivity gains from AI coding tools means comparing delivery speed, code quality, and developer experience before and after adoption, using baselines and matched groups to separate real improvement from activity inflated by suggestions.

Why Lines of Code and Acceptance Rates Mislead Teams

Building on that definition, the first lesson is that the easiest numbers to collect are usually the least informative. Vendors report suggestion acceptance rates because the figure is simple to log, and it climbs as developers grow comfortable with the tool. An accepted suggestion may still be rewritten ten minutes later, deleted during review, or replaced when requirements change. Counting lines of code adds a second distortion, because assistants generate verbose output that looks like progress on a dashboard. A metric that rewards producing more code will always reward the assistant, even when the team ships nothing faster. Engineers who feel measured on volume also change their behavior, which makes the metric less reliable the moment it becomes a target.

A better habit is to ask what each number is supposed to predict and then test that link. Acceptance rate is meant to predict saved typing time, but typing was never the main constraint on delivery. Requirements clarity, test feedback, code review queues, and deployment friction usually decide how quickly a change reaches users. Reading about how AI has changed software development makes the same point from a historical angle, since each productivity tool shifted effort rather than removing it. Measurement should therefore follow the whole path from idea to production instead of stopping at the editor.

Teams that adopt this wider view tend to retire several favorite statistics within the first quarter. They keep acceptance rate as a diagnostic for tool fit, but they stop reporting it as a result. They add measures that sit closer to customer value, such as lead time for changes and the share of work that reaches production without rework. They also record how long reviewers spend on AI-assisted changes compared with similar human-written changes. Those comparisons often reveal that the assistant has moved effort from authors to reviewers rather than reducing it.

Separating Output From Outcomes in Engineering Work

Moving on to a distinction that underlies every sound measurement plan, output is what the team produces and outcome is what changes for the business. Output includes commits, pull requests, and tickets closed, while outcomes include faster feature delivery, fewer incidents, and lower cost per change. AI coding tools clearly increase output in many settings, and that fact explains why early dashboards look so encouraging. Outcomes improve only when the extra output passes through review, testing, and release without creating new queues. The 2025 DORA research, summarized by InfoQ's coverage of the report, found that AI adoption now relates positively to delivery throughput while delivery instability continues to rise.

That pairing of gains and strain is the signature of a system that produces more than it can safely absorb. A team might merge far more pull requests, yet spend the saved time on longer reviews, flaky test reruns, and hotfixes. Looking only at output would count that team as a success, whereas an outcome view would flag growing instability. The practical response is to choose at least one outcome metric for every output metric you track. If pull request volume is on the dashboard, change failure rate and review duration should sit beside it.

Outcome thinking also protects against local optimization by individual developers. A person who finishes a task quickly with an assistant may leave behind code that colleagues struggle to extend. The cost appears later as slower onboarding of new modules and more questions in review threads. Capturing that cost requires metrics measured at the team or service level rather than the individual level. It also requires patience, because maintenance effects take months to surface.

Shifting Focus to Baselines: How to Prepare Before Any Rollout

Shifting focus to the weeks before a rollout, a credible measurement program starts with a baseline that nobody can dispute later. Without one, every post-adoption improvement can be explained away by seasonal workload, staffing changes, or an unrelated process fix. A baseline should cover at least eight weeks of data for cycle time, pull request size, review wait time, deployment frequency, and escaped defects. Gather the same measures for the teams that will adopt the tool and for comparable teams that will not adopt it yet. The comparison group is what turns a before-and-after story into evidence.

Baselines also need context notes, because raw numbers hide events that change them. Record major releases, reorganizations, holidays, incident weeks, and tooling migrations on the same timeline as the metrics. When the pilot begins, keep those notes current so that analysts can explain unusual spikes without guesswork. Teams working through AI coding assistants in startup product development often skip this step because their codebases and staffing change quickly. The cost of skipping is an evaluation that cannot be defended when finance asks for proof.

A third requirement is keeping every metric definition stable throughout the pilot. Decide how you will compute cycle time, which repositories count, and how bot-authored changes are treated, then freeze those rules for the pilot. Changing a definition halfway through makes trend lines meaningless, even when the change seems harmless. Write the definitions in a short document that engineering managers and data analysts both sign. That document becomes the reference whenever someone questions a number months later.

Finally, baseline the human side as well as the system side. Run a short survey asking developers how much time they spend waiting, context switching, and fixing avoidable defects. Ask how satisfied they are with tooling and how often they reach a state of focused flow. These answers give you a pre-adoption picture of friction that the assistant is supposed to reduce. After rollout the same questions show whether the tool addressed real pain or merely added a new interface.

Turning to the SPACE Framework for AI Coding Tools

Turning to the research literature, the SPACE framework published in ACM Queue offers the most useful vocabulary for balanced measurement. It describes five dimensions of developer productivity: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. The authors argue that no single dimension tells the full story and that metrics should pull against each other on purpose. They recommend choosing at least three metrics across a minimum of two dimensions, including one perceptual measure such as a survey. For AI coding tools this means pairing an activity metric like pull request volume with a performance metric like change failure rate and a satisfaction score. Keeping the set small prevents the dashboard from becoming a wall of numbers that nobody acts on.

Applying the framework to an assistant rollout is straightforward once the dimensions are mapped to data sources. Satisfaction comes from a quarterly survey, performance from incident and defect data, activity from version control, and collaboration from review latency. Efficiency and flow can be approximated through the time developers report being blocked or interrupted. Because each source is independent, a gain that appears in only one dimension deserves skepticism until another dimension confirms it. Review the dimensions together every quarter so that trade-offs are discussed openly rather than hidden.

Beyond Speed: Quality, Stability, and Rework Signals

Beyond speed, a complete evaluation asks whether the code produced with assistance stays healthy over time. GitClear's analysis of 211 million changed lines reported that copy-pasted code rose from 8.3 percent to 12.3 percent of changed lines between 2021 and 2024. The same analysis found that moved or refactored code fell from 25 percent of changed lines to under 10 percent. Those patterns suggest that assistants make it easy to add new code and comparatively harder to consolidate what already exists. Rising duplication is a leading indicator of future maintenance cost, even when current delivery metrics look healthy.

Teams can track comparable signals inside their own repositories without buying a new platform. Measure the share of changes that touch code written in the previous thirty days, because high churn on recent code signals rework. Track defect escape rate, the proportion of bugs found in production relative to those caught in review or testing. Watch reverts and hotfixes per hundred deployments, and compare AI-assisted pull requests with matched human-authored ones. Any gap between those groups is a more honest quality verdict than a vendor benchmark.

Reading these signals requires care because correlation is easy to misread. A rise in churn might reflect a healthy habit of rapid iteration rather than poor quality. The way to tell the difference is to look at downstream effects such as incident counts and the time reviewers spend per change. If churn rises while incidents and review time fall, the team is probably iterating well. If all three rise together, the assistant is likely generating code that needs repeated correction.

Reviewers deserve special attention in this analysis because they absorb the hidden cost of generated code. Ask them how often they find subtle logic errors, unnecessary abstractions, or tests that assert nothing meaningful. Their comments often explain a statistic better than any dashboard can. Capture a few representative examples each month and share them with the team as calibration material. This calibration keeps everyone aligned on what good AI-assisted work actually looks like.

Looking at DORA Metrics in an AI-Assisted Workflow

Looking at the delivery side of the picture, the four classic DORA measures remain the best compact view of system health. Deployment frequency and lead time for changes describe speed, while change failure rate and time to restore service describe stability. An assistant that truly helps should improve the speed pair without worsening the stability pair, and that is a demanding test. DORA's 2025 research on AI-assisted development describes AI as an amplifier of existing strengths and weaknesses, so strong teams gain more than struggling ones. Treat the DORA four as a guardrail: speed gains that cost stability are not productivity gains.

Agent-based tools introduce a further wrinkle, since autonomous runs consume budget and produce changes that nobody typed. Teams adopting them can borrow techniques from measuring AI agent performance, such as success rate per task and cost per completed task. Those measures belong on the same dashboard as the DORA four so that leaders see speed, stability, and spend together. Cost per completed task matters because autonomous runs can consume budget without producing a mergeable change. Review those figures monthly so that spending growth can be tied to measurable delivery gains.

Developer Experience Surveys and Perceived Productivity

From there, the conversation turns to what developers themselves report, which is valuable but must be interpreted carefully. Surveys capture friction that telemetry cannot see, such as frustration with context switching or confidence in the code they ship. The 2025 Stack Overflow survey found that 66 percent of respondents cited AI answers that are almost right but not quite as their top frustration. That finding explains why perceived time savings can coexist with extra debugging effort. Self-reported speedups should be treated as a hypothesis to verify against telemetry, never as the result itself.

Controlled research gives developers and managers a sharp reason for that caution. In the METR study of experienced open-source developers, participants expected AI to speed them up by 24 percent. Afterward they still believed it had helped by 20 percent, although measured task time had risen by 19 percent. Perception and measurement diverged by a wide margin, and the participants were skilled practitioners working on their own repositories. The lesson is not that surveys are useless but that they answer a different question. They tell you how work feels, which matters for retention and morale, rather than how long work takes.

Good survey design keeps the instrument short and the cadence steady. Five to eight questions every four to six weeks is enough to detect shifts without causing fatigue. Ask about specific recent tasks instead of general impressions, for example whether the last pull request took more or less time than usual and why. Include one open-text question about the most frustrating moment with the tool. Reviewing those comments in the retrospective often surfaces fixable problems, such as missing context in prompts or poor repository documentation.

Segment results by experience level and task type before drawing conclusions. Less experienced developers often report larger gains, while senior engineers working in familiar code report smaller ones or none. Task type matters as well, since boilerplate, tests, and documentation behave differently from architectural changes. Reading about measuring generative AI adoption in the workplace adds a useful adoption lens, because gains cannot appear in teams that rarely open the tool. Reporting by segment prevents an average from hiding real winners and real losers.

Putting Telemetry Into Practice Without Surveillance

In practice, telemetry should answer team-level questions rather than rank individuals. The most useful data comes from version control, the code review system, the continuous integration pipeline, and the incident tracker, all of which already exist. Join those sources by repository and week, and tag each change as AI-assisted when the tool exposes that signal or when a team convention marks it. Aggregate to the team or service level before anyone looks at a chart, because individual leaderboards destroy trust and invite gaming. Observability practices from AI observability and tracing for LLM applications translate well, since both disciplines rely on traceable events rather than anecdotes.

Privacy and data residency also shape which telemetry is possible for a given team. Some organizations run assistants locally to keep source code inside their network, which limits the vendor analytics available. The tradeoffs described in local AI coding stack challenges include weaker usage reporting, so teams may need to build their own event logging. A small internal service that records tool invocations, accepted completions, and session length is usually enough. Store only anonymous identifiers and delete raw events after the aggregation window closes.

Keep the pipeline simple enough that one analyst can maintain it. Start with a weekly export of pull request timestamps, review events, and build results, then compute medians instead of means to blunt the effect of outliers. Publish the dashboard to everyone whose work it describes, along with the definitions behind each metric. Transparency lets developers challenge odd numbers early and reduces the sense of being watched. It also produces better data, because people who trust the system tag their work accurately.

Weighing Cost, Licenses, and Return on Investment

Given the findings so far, the financial case must be built from net effects instead of gross promises. A license may cost a modest amount per seat each month, but the true cost includes onboarding time, review overhead, security review, and the premium models some teams reach for. Savings come from time returned to engineers, which only has value when that time is redirected toward work the business wants. Convert measured hours saved into dollars only after subtracting the extra review and rework time that the same study period recorded. The framework in measuring ROI on AI investments helps structure that arithmetic into benefits, costs, and confidence ranges.

Usage-based pricing for agents adds a second cost line that scales with activity rather than headcount. Tracking spend per completed task, as described in agent cost per task measurement, keeps those bills tied to outcomes. Report the result as a range with stated assumptions, since studies disagree about the size of the underlying effect. Finance leaders trust a range with clear assumptions more than a single optimistic figure. Revisit the assumptions each quarter as license prices, model costs, and usage patterns shift.

Where Measurement Falls Short: Risks and Blind Spots

Despite the best design, every measurement program carries blind spots that leaders should name openly. Small pilot teams produce noisy results, so a three-week experiment with eight developers can show almost any outcome. Novelty effects inflate early enthusiasm, and learning costs depress early speed, which means the first month rarely reflects steady state. Selection bias enters when volunteers adopt first, since enthusiastic engineers differ from the average. The most dangerous risk is a confident number built on a biased sample.

Tooling changes faster than most evaluation cycles can keep pace with. A study run on one model generation may not describe the next, and the METR authors themselves note that their early-2025 results reflect the tools of that period. Teams should schedule re-measurement whenever the underlying model, the editor integration, or the usage pattern changes materially. Pair every reported number with the date and tool version so readers can judge how current it is. Without that discipline, an old benchmark can quietly justify a new purchase.

Gaming is a quieter but equally serious risk to any measurement program. When bonuses or promotions depend on a metric, people optimize the metric, and AI tools make optimization cheap. Splitting pull requests inflates counts, padding tests inflates coverage, and verbose code inflates line totals. Mitigate by using several metrics at once, rotating the emphasis, and keeping measures out of individual performance reviews. The practices in when vibe culture breaks code illustrate how quickly informal habits can undermine quality when guardrails are weak.

Stepping Back to Ethics, Privacy, and Fairness in Developer Metrics

Stepping back from the mechanics, developer metrics raise ethical questions that deserve explicit policies. Workers have a legitimate interest in knowing what is collected, how it is used, and who can see it. Collecting keystroke-level or prompt-level data crosses a line in many jurisdictions and erodes trust even where it is legal. Measure the work system and the team, and keep individual-level data out of performance decisions. Publish a short data charter that lists the signals collected, the retention period, and the people with access.

Fairness matters here because assistants affect different people in different ways. Developers working in less common languages, or in domains with sparse training data, may see smaller benefits and could be unfairly judged against colleagues in better-supported stacks. Accessibility needs, language fluency, and prior experience also change how people use the tool. Building a culture where experiments are discussed openly, as outlined in enabling a better programming culture in teams, keeps measurement collaborative rather than punitive. Review the data charter with employee representatives at least once a year.

Looking Ahead to the Future of AI Coding Measurement

Looking ahead, measurement will have to adapt as assistants evolve from autocomplete into agents that plan, edit, and test across many files. Per-keystroke metrics lose meaning when a developer delegates an entire issue and returns to review a finished change. The unit of analysis shifts toward tasks and outcomes, such as the share of issues resolved without human rewriting and the cost per accepted change. Future dashboards will track delegation quality, meaning how well a team scopes work for agents and how reliably the results pass review. Teams that track delegation quality early will spot weak scoping habits long before they become expensive.

Standards for evidence are also rising across the whole industry. More vendors and independent labs now publish randomized trials, and buyers increasingly ask for matched-group comparisons rather than testimonials. Platform integrations such as those described in AI supercharges MCP developer workflow will make context sharing between tools richer, which in turn changes what telemetry is available. Teams that invest in clean baselines today will be able to re-run their evaluations each time the technology shifts. Clean baselines turn each new model release into a quick experiment instead of a debate.

Documentation quality will become a measurable productivity lever as well. Assistants that read live reference material, a trend covered in AI coding agents and live API docs, depend on accurate and current sources. Organizations that treat documentation as a product will likely see larger gains than those that do not. Documentation freshness is cheap to measure and easy to improve, which makes it a strong first investment. Expect future surveys to ask not only whether developers use AI but how well their environment supports it.

How to Measure Developer Productivity Gains From AI Coding Tools Step by Step

With that foundation in place, the following seven steps turn the concepts into a repeatable evaluation you can run within a single quarter. Each step produces an artifact, such as a baseline report, a metric plan, or a decision memo, so the work stays auditable. The sequence assumes a pilot of at least 2 teams and a comparison group of similar size. You can scale it down for a single team, but expect wider uncertainty in the results. Plan roughly 2 weeks for setup, 8 weeks for the pilot, and 2 weeks for analysis. Budget a small amount of analyst time each week rather than one large effort at the end.

Step 1 - Define the outcomes you want to improve

Start by writing down the two or three outcomes the business actually wants from the assistant, such as shorter lead time, fewer production incidents, or faster onboarding for new hires. Vague goals like becoming more productive cannot be measured, so attach a direction and magnitude to each, such as cutting median cycle time by 10 percent. Involve engineering managers, product leads, and finance in this conversation so that everyone agrees on what success means. List the decisions the results will inform, such as expanding licenses to 200 more seats or pausing the rollout. If no decision depends on the answer, the measurement effort is probably not worth running. Record the outcomes in a one-page charter that you will reference in the final review.

Step 2 - Capture a baseline for pilot and comparison teams

Export at least 8 weeks of pull request data for every team in the pilot and the comparison group. You need opened, first review, approved, and merged timestamps, plus lines changed and the author type. Compute medians for cycle time and review wait time, because means are easily distorted by a handful of stalled changes. The short script below shows the calculation from a CSV export, and you can adapt it to any source that provides the same timestamps. Keep the raw export under version control so that the baseline can be reproduced later. Share the resulting numbers with team leads before the pilot starts to catch data errors early.

import csv
from statistics import median
from datetime import datetime

FMT = "%Y-%m-%dT%H:%M:%S"

def hours_between(start, end):
    delta = datetime.strptime(end, FMT) - datetime.strptime(start, FMT)
    return delta.total_seconds() / 3600

with open("pull_requests.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))

cycle = [hours_between(r["opened_at"], r["merged_at"]) for r in rows if r["merged_at"]]
wait = [hours_between(r["opened_at"], r["first_review_at"]) for r in rows if r["first_review_at"]]

print("Merged pull requests:", len(cycle))
print("Median cycle time (hours):", round(median(cycle), 1))
print("Median wait for first review (hours):", round(median(wait), 1))

Add deployment and incident data to the same baseline report. Count deployments per week, the share that required a rollback or hotfix, and the median time to restore service after an incident. Include the number of escaped defects found in production over the same 8 weeks. If your organization records developer survey scores, include the most recent 2 cycles. A complete baseline typically fits on two pages and becomes the denominator for every claim you make afterward.

Step 3 - Select a balanced metric set

Choose between 5 and 8 metrics covering at least 3 SPACE dimensions, and record the owner, source, and review cadence for each one. A sensible default pairs cycle time and review wait for efficiency, change failure rate and escaped defects for performance, merged pull requests for activity, and a survey score. Assign each metric a direction of good, a guardrail value that must not worsen, and a note describing how it could be gamed. The configuration below shows one way to record the plan in a version-controlled file that analysts and managers can both read. Agree on the list before the pilot begins, and do not add new headline metrics halfway through. Extra metrics are fine as diagnostics, but only the pre-registered set decides the outcome.

measurement_plan:
  window_weeks: 8
  groups:
    pilot: [payments, search]
    comparison: [billing, catalog]
  metrics:
    efficiency: [median_cycle_time_hours, review_wait_hours]
    performance: [change_failure_rate, escaped_defects]
    activity: [merged_pull_requests]
    satisfaction: [survey_score]
  guardrails:
    change_failure_rate: no_increase
    escaped_defects: no_increase

Step 4 - Instrument the workflow with team-level telemetry

Connect your version control, review, build, and incident systems into a weekly job that writes one row per team per week. Tag AI-assisted changes through a label, a commit trailer, or tool-provided analytics, and document which method you chose. The command below pulls 8 weeks of merge history from a repository as a starting point for the dataset. Store only team-level aggregates in the dashboard, and keep raw events in a restricted location with a short retention period. Run the job for 2 weeks before the pilot begins to confirm that the numbers match what engineers see in their own tools. Fix any data quality issues now, because errors discovered after the pilot starts are expensive to correct.

git log --since="8 weeks ago" --merges --date=iso --pretty=format:"%h,%ad,%s" > merges.csv
wc -l merges.csv

Step 5 - Roll out in stages with a comparison group

Introduce the assistant to the pilot teams in waves rather than all at once, and keep the comparison teams unchanged for the full 8 weeks. Provide the same onboarding session, documentation, and prompt guidelines to every pilot team so that training does not become a hidden variable. Match teams on size, codebase age, and workload type as closely as you can, and write the matching rules down before assignments are made. Where possible, randomize which eligible teams start first, since random assignment protects against the bias that comes from letting enthusiasts self-select. Hold weekly office hours so developers can raise problems and the pilot lead can log them as qualitative evidence. Resist the urge to tweak the setup mid-pilot unless a serious problem appears.

Pro tip: freeze the tool version and model settings during the pilot, or record every change with a date, because a silent model upgrade can invalidate your comparison. Expect a learning dip in the first 2 weeks as developers adjust their habits. Treat that dip as normal and exclude the first fortnight from the headline comparison if you declared that rule in advance. Document any holidays or incidents that hit one group harder than the other. These notes will matter when you explain the results to skeptical colleagues.

Step 6 - Combine telemetry with surveys and quality review

Run a 6-question survey at the start, midpoint, and end of the pilot, asking about focus time, confidence in shipped code, and time lost to rework. Add a quality review in which 2 senior engineers sample 20 AI-assisted and 20 human-authored pull requests and score them against a shared rubric. Teams that already use automated grading of model output can reuse the approach in evaluating outputs with LLM as a judge to triage review comments at scale. Compare the scores between groups, and look for patterns such as unnecessary abstractions, weak tests, or missing edge cases. Keep the sample blind where possible so that reviewers do not know which changes involved the assistant. The goal is to confirm that speed has not been bought with a hidden quality cost.

Step 7 - Analyze the results and decide what to do next

After 8 weeks, compare pilot and comparison teams on the pre-registered metrics using medians and simple confidence ranges. Report the effect for each metric separately, and show the guardrail metrics beside the headline gains so readers see the trade-offs. If the pilot shows faster cycle times but longer reviews or more reverts, state that plainly instead of averaging the signals into one score. Separate results by experience level and task type, because gains often concentrate in specific groups. Write a decision memo of no more than 2 pages that recommends expanding, adjusting, or pausing the rollout. Schedule a re-measurement for 6 months later, since models and workflows will have changed.

Key Insights From the Latest AI Coding Productivity Research

• The 2025 Stack Overflow Developer Survey found that 84 percent of developers use or plan to use AI tools, so most evaluations now begin without a clean non-user control group.
• The same Stack Overflow results on trust show only 33 percent of developers trust AI output and 46 percent distrust it, so verification time belongs in every productivity calculation.
• METR's randomized trial of 16 experienced developers on 246 issues found tasks took 19 percent longer with AI, even though participants believed they had been 20 percent faster.
• Google's randomized study of 96 engineers estimated a 21 percent speedup, yet the reported confidence interval included no effect, so the gain was not statistically significant.
• Telemetry from 1,255 teams analyzed by Faros AI showed 98 percent more merged pull requests alongside 91 percent longer review times and no measurable company-level delivery gain.
• Field experiments at Microsoft, Accenture, and a Fortune 100 company covering 4,867 developers found a 26.08 percent increase in completed tasks, with larger gains for less experienced developers.
• GitClear's analysis of 211 million changed lines found copy-pasted code rising from 8.3 percent to 12.3 percent while moved code fell below 10 percent, a sign of growing maintenance debt.
• The SPACE framework advises tracking at least 3 metrics from two or more dimensions, including one perceptual measure, so a single inflated number cannot dominate the picture.

Taken together, these results describe a technology that helps in many settings but does not help automatically. Controlled trials range from large gains on constrained tasks to measurable slowdowns on complex, familiar codebases, while field telemetry shows output rising faster than delivery. Perception adds noise, since developers across several studies felt faster than the clocks confirmed. The common thread is that benefit depends on context, including task type, experience level, review capacity, and codebase maturity. A defensible measurement plan therefore combines baselines, balanced metrics, and surveys rather than trusting any single headline figure. Teams that build that plan early will adjust faster as the tools keep changing.

Dimension	Activity counts	Delivery metrics (DORA)	Quality and rework	Developer surveys	Controlled experiments
What it captures	Volume of suggestions, commits, and pull requests	Speed and stability of shipping to production	Defects, reverts, duplication, and churn	Satisfaction, confidence, and perceived flow	Causal effect of the tool on time or output
Typical data source	Editor and version control logs	Deployment and incident systems	Review tools, bug trackers, static analysis	Short recurring questionnaires	Randomized or matched group designs
Main strength	Easy to collect and available immediately	Ties directly to business delivery	Reveals hidden maintenance cost	Captures friction telemetry cannot see	Strongest evidence of cause and effect
Main weakness	Rewards volume rather than value	Slow to react and affected by many factors	Needs careful interpretation and context	Perception can diverge from measured time	Costly, small samples, and narrow tasks
Risk of gaming	High	Medium	Medium	Low	Low
Time to first signal	Days	Four to eight weeks	Eight to twelve weeks	Two to four weeks	Eight weeks or longer
Effort to maintain	Low	Medium	Medium to high	Low	High
Best use	Diagnosing tool adoption and fit	Guardrails for speed and stability	Checking that speed is not buying debt	Understanding experience and morale	Settling disputes about real effect size

Evidence in Practice: Three Experiments That Measured AI Coding Gains

Google's Enterprise Randomized Trial

Among the most carefully designed studies, Google researchers ran a randomized trial with 96 engineers, assigning half to use an internal AI assistant. The other half completed the same task without any assistant. Participants implemented a new logging service inside an existing change list of ten files and 474 lines, a task designed to take between 30 minutes and 2.5 hours. Engineers using the assistant finished in a mean of 96 minutes compared with 114 minutes for the control group, an estimated speedup of about 21 percent after adjusting for covariates. The researchers reported that the effect was not statistically significant at conventional thresholds, with a p-value of 0.086, so the true gain could be near zero or much larger. The study also has clear limits, because all participants were Google employees, the authors evaluated their own team's tools, and a single task cannot represent months of ordinary work. Readers can inspect the full design in the published arXiv paper on enterprise development speed before relying on its figures.

METR's Study of Experienced Open-Source Developers

METR ran a randomized controlled trial with 16 experienced open-source developers who worked on 246 real issues from repositories they already knew well. Each issue was randomly assigned to allow or disallow AI tools, and developers mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet. Tasks took 19 percent longer when AI was allowed, even though developers had forecast a 24 percent speedup before starting. After finishing, the same developers still believed AI had saved them about 20 percent of their time, which shows how far perception can drift from measurement. The sample is small, and the authors caution that results from early-2025 tools in mature repositories may not describe other settings. METR notes that it has since published updated data, so read the original METR study of early-2025 AI and experienced developers as one data point rather than a verdict.

A Controlled Experiment on Building an HTTP Server

A research team ran a controlled experiment in which recruited developers were asked to implement an HTTP server in JavaScript as quickly as possible. One group had access to the Copilot AI pair programmer, and the other group worked without it. The treatment group completed the task 55.8 percent faster than the control group, the largest speedup reported among the widely cited early studies. The task was a well-defined, self-contained exercise, so the result says little about debugging an unfamiliar legacy service or navigating a large monorepo. That narrow scope is the main limit, and later field studies on real work reported smaller gains of around 21 to 26 percent. The experiment is documented in the arXiv paper on the impact of AI on developer productivity for anyone who wants the method.

Lessons From Three Large Deployments and Field Studies

Case Study: Accenture's Enterprise Copilot Rollout

Rounding out the evidence with real organizations, Accenture and GitHub tackled a problem many leaders share: proving the value of an AI assistant beyond anecdotes. The team rolled out GitHub Copilot to developers and combined telemetry with surveys to capture both behavior and sentiment. Developers accepted about 30 percent of suggestions, and 88 percent of the characters Copilot generated were retained in the editor. At the pipeline level, pull requests increased by 8.69 percent, the merge rate for pull requests rose by 15 percent, and successful builds increased by 84 percent. Surveys showed that 90 percent of developers felt more fulfilled in their jobs and 95 percent enjoyed coding more.

These results come with important limits that readers should weigh. The report was co-authored with GitHub and Microsoft staff, and it does not publish sample sizes, confidence intervals, or the measurement period for each figure. Acceptance and retention measure how often code is kept, which is not the same as how quickly valuable features reach customers. The 84 percent rise in successful builds is striking, but without a stated baseline it is hard to know how much reflects better tests versus more builds. The case remains useful as a template for pairing telemetry and surveys, as shown in GitHub's research with Accenture for organizations planning a similar rollout.

Case Study: Three Company Field Experiments With 4,867 Developers

A recurring problem in the literature is that most early evidence came from short lab tasks rather than real jobs. They worked with Microsoft, Accenture, and an anonymous Fortune 100 company to run randomized field experiments, giving some developers access to an AI coding assistant and withholding it from others. The study covered 4,867 developers and measured outputs from their normal work over time. Developers who were given the assistant completed 26.08 percent more tasks than those without it. Less experienced developers had higher adoption rates and greater productivity gains, according to the abstract.

The design is stronger than most, but it still has limits. Completed tasks are an output measure, and the study does not by itself show whether quality or long-term maintainability changed. Gains concentrated among newer developers, so a team of senior engineers should not expect the same lift. Because the experiments were run at large firms with mature engineering practices, smaller organizations might see different results. Teams that implemented the assistant should read the full SSRN paper on three field experiments with software developers before quoting the headline figure.

Case Study: Faros AI's Telemetry Across 1,255 Teams

Faros AI set out to answer a problem that executives kept raising: if developers feel faster, why do company dashboards barely move? The company analyzed telemetry from more than 10,000 developers across 1,255 teams, comparing teams with high AI adoption against those with low adoption. Teams with high adoption completed 21 percent more tasks and merged 98 percent more pull requests. Their pull request review time grew by 91 percent, and bugs per developer rose by 9 percent. The report found no significant correlation between AI adoption and improvement at the company level.

A sensible solution is to treat AI as a system change rather than a tool purchase, adjusting review capacity, testing, and workflow alongside adoption. This case also has limits, because telemetry shows association rather than causation, and vendor reports can emphasize findings that support their own products. The data come from teams in the field rather than randomized groups, so unobserved differences between enthusiastic and cautious teams could explain part of the pattern. Even so, the pattern of rising output and rising review load matches what controlled studies suggest about shifting bottlenecks. The full findings appear in the AI Productivity Paradox report from Faros AI along with the underlying methodology.

Frequently Asked Questions on Measuring Developer Productivity Gains From AI Coding Tools