AI

The Misuse of AI

Human misuse elevates AI risks and dangers like bias, cyberattacks, and disinformation, highlighting need for ethics.
Editorial collage illustrating the misuse of AI through deepfake video calls, election disinformation screens, biased decision alerts, and a stylised fraud timeline.

Introduction

The misuse of AI has moved out of research papers and into the daily news cycle, and the volume keeps rising. A Google DeepMind team mapped more than two hundred real misuse cases and found that impersonation and opinion scaling account for about seventy-four percent of recorded incidents. That shift is new in scale, not in kind, because people have abused new tools for centuries. What is new is cost: a deepfake convincing enough to drain a corporate account used to need a studio and now needs a laptop. Attackers reach millions of targets with a prompt, defenders mostly still rely on human review, and the gap widens month to month. This guide maps the current terrain, names the harms, and shows what works for individuals, product teams, and regulators who want to narrow that gap. Readers who want a quick primer on where misuse begins can also review our survey of AI-driven misinformation and manipulation.

Quick Answers on the Misuse of AI

What is the misuse of AI, in one sentence?

The misuse of AI is the deliberate use of machine-learning systems to deceive, defraud, surveil, exclude, or harm people, in ways designers of the misuse of AI tools did not intend.

What are the most common misuse of ai examples right now?

Political deepfakes, voice-cloning fraud, non-consensual intimate imagery, phishing scaled by language models, and biased decision tools in hiring, lending, and policing are the five most reported misuse of AI categories.

Is the misuse of AI already illegal under current law?

Many misuse of AI cases violate existing fraud, defamation, or discrimination law. Prosecution remains slow. The EU AI Act and several US state statutes now target the technology, not only the downstream harm.

Key Takeaways on AI Misuse and Harm

  • Impersonation and opinion-shaping dominate recorded cases, together making up roughly three-quarters of mapped misuse of AI incidents.
  • Deepfake-enabled fraud has crossed into nine-figure losses in a single case, with the Arup video-call scam alone draining about twenty-five million dollars.
  • Child safety organisations recorded a sharp rise in AI-generated child sexual abuse material, with the Internet Watch Foundation flagging more than twenty thousand such images in a single month.
  • Policy is tightening unevenly, with the EU AI Act’s prohibited-practice provisions in force while most US rules sit at state level or inside sectoral guidance.

Understanding the Misuse of AI in Plain Terms

The misuse of AI is the intentional application of machine-learning systems to deceive, defraud, surveil, exclude, or harm people. It differs from an honest AI mistake, which is an error without intent. Misuse of AI points at the human choice.

An Interactive From AIplusInfo

Score your organisation’s misuse of AI exposure

Pick a sector, then rate your deepfake defence, employee training, and content-provenance adoption. The score blends real-world incident weights with your inputs to flag which misuse category is most likely to bite first.

Financial services

Changes category weighting

4 / 10

NoneFully drilled

5 / 10

Ad hocQuarterly

3 / 10

NoneFull C2PA

Composite exposure

63 / 100

Elevated: targeted controls recommended within 90 days.

Likely first-hit category

Deepfake fraud

Finance teams are the most-targeted workflow for voice-cloning and video-call fraud, per the Veriff 2026 industry tracker.

Weighting model: DeepMind misuse-category frequencies + Veriff 2026 sector-fraud deltas. See the full method in the article’s methodology notes.

The Main Categories of AI Misuse Today

Building on the definition above, researchers who track AI incidents group misuse into five rough categories. These include impersonation, opinion scaling, falsification of records, exploitation of model flaws, and discriminatory decision support. The DeepMind misuse map covered a two-year window across public reports. Impersonation sits at the top because it works at scale, costs very little, and transfers across text, audio, and video. Opinion scaling is close behind because the same models that generate persuasive essays also generate persuasive replies, reviews, and comment threads. Falsification turns up in fake academic papers, synthetic customer complaints, and forged identity documents.

Building on that framing, the practical lesson is that most AI misuse of ai is social engineering with a model attached, not an exotic hack of the model itself. Attackers rarely need to break a system when they can impersonate someone the system already trusts. That is why voice-cloned CEO fraud and spear-phishing have taken off faster than, say, model-extraction attacks. The research community does track technical attacks too, but incidents involving everyday deception far outnumber them. For a sector-by-sector tour of related harms, our overview of the data-exploitation side of AI misuse maps the same pattern across data brokers and ad targeting.

Beyond those five main categories, a sixth category of ai misuse is quietly emerging in the data. Autonomous agents chaining tools together can now take actions in the world, not only produce text. Agent misuse opens attack paths that look less like a chatbot going off-script and more like a junior employee acting with stolen credentials. Early reports include spam agents, automated harassment pipelines, and jailbroken agents used to drain crypto wallets. The share is small today, but the trend lines matter. The risks of autonomous AI in cybersecurity cover why detection becomes harder when the attacker is itself a model.

How Deepfakes Became the Signature Weapon of AI Misuse

Beyond the categories themselves, the single most common vector, deepfakes have become the signature weapon because they fit every one of the top misuse categories. A deepfake can impersonate a politician, scale a false opinion, falsify a bank’s video call, and bypass identity checks. The hardware cost of making a passable deepfake has dropped from studio-grade kit to a mid-range laptop, which is the single most important change in the economics of misuse. The democratisation of generation, more than any single breakthrough in model quality, is what put deepfake misuse of ai into the mainstream. That shift pulls the problem out of national-security circles and places it in the daily reality of HR teams, journalists, teachers, and parents.

Shifting focus to defence, detection has not kept pace with generation, and the asymmetry has widened each year since 2022. Audio deepfakes in particular remain hard to spot in real time, which is why phone-based fraud leads the loss columns. Image forensics tools work well in a lab setting and poorly on compressed social-media video. Watermarking standards exist on paper, but adoption is partial, and bad actors have no incentive to use them. For readers who want a working primer, our guide on how to spot a deepfake today walks through the visual, audio, and metadata cues that still catch most consumer-grade fakes.

Source: YouTube

Election Interference and AI-Driven Disinformation

Shifting from technology to politics, elections have become a stress test for every assumption about the misuse of AI. The 2024 cycle brought the first large-scale wave of AI-generated political content, with incidents in the United States, India, Slovakia, Argentina, Bangladesh, and across the European Parliament vote. The Partnership on AI’s post-election review catalogued dozens of synthetic political ads, cloned candidate voices, and manipulated videos. Not every synthetic clip changed a mind, but the aggregate effect was to lower the bar for plausible deniability. Any real recording could now be waved away as a possible deepfake, and any fake could be posted at the pace of outrage.

Building on that pattern, the Biden robocall case in New Hampshire gave regulators their first clean prosecution target. A political consultant commissioned an AI-cloned voice urging Democrats not to vote in the primary. The Federal Communications Commission moved to ban AI-generated voices in robocalls within weeks of the incident. The FCC’s declaratory ruling under the Telephone Consumer Protection Act treated the voice itself as the illegal element, not only the content. That ruling matters because it moved US enforcement from the harm layer to the capability layer, which is where EU regulators had already been working through the AI Act. State attorneys general have followed the FCC lead and brought civil cases that treat the cloning tool itself as the actionable product.

Beyond named cases, the broader worry is content pollution rather than any single viral clip. Research teams tracking synthetic political content on major platforms report that it rarely goes mega-viral, but it fills the long tail of search, reply threads, and group-chat forwards. Those are the surfaces that voters actually see, which means impact lives in volume rather than in a single breakout clip. Our editorial coverage on how AI drives election misinformation walks through the mechanics of that long-tail effect across platforms. Platform policies on synthetic political content have hardened since 2023, but enforcement is uneven, especially outside English-language content, and the long-tail problem is harder to measure than viral incidents.

Looking past 2024, the misuse pattern will shift from fake candidate speech to fake voter interactions. Automated agents can already produce thousands of plausible constituent emails per hour, which drowns the signal legislators rely on to measure public opinion. Public comment systems at US federal agencies have reported bursts of what appear to be AI-generated submissions, which forces staff to reweight real voices downward. The next misuse frontier for democracies will not be a single viral deepfake clip. It is the quiet grinding down of representative inputs by synthetic ones. Defending it requires authentication of the sender, not only detection of the content. Several state secretaries of state are exploring verified-constituent portals for comment, which would push authentic voices to the top of the staff queue.

Deepfake Fraud and the New Face of Financial Crime

Shifting from politics to money, financial deepfake fraud is where the misuse of AI shows up in loss statements. The Hong Kong office of the UK engineering firm Arup lost about twenty-five million US dollars after a deepfake video call. The staff member joined what appeared to be a meeting with the CFO and other executives. All of the people on the call except the staffer were AI-generated, as documented in the detailed Insurance Business UK account of the Arup case. The attack worked because the staffer trusted the visual cues of a familiar team, the request had plausible timing, and the transfer channel followed the normal corporate workflow. No single step broke policy, which is what makes the case so widely cited.

Beyond that single incident, the sector-wide pattern is that voice-cloning fraud grew faster than any other category of AI-enabled scam between 2023 and 2026. The identity-verification vendor Veriff tracked the broader trend across its base, with analysts documenting about 3.7 billion dollars in deepfake-related losses in 2026 alone. Elder-fraud call centres, HR onboarding checks, and loan-origination desks are the three most targeted workflows. In every case, the attack path begins with a short audio sample scraped from social media, which the model turns into a passable live voice. The social media link matters because most people assume that samples below thirty seconds are too short to clone, when the real floor is closer to ten.

Looking at defensive practice, banks and corporate treasury teams have begun layering old-world controls on top of new-world attacks. Shared code words for high-value transfers, callback verification over a separate channel, and video calls that require the counterparty to perform a non-scripted action have all seen fresh adoption. The FBI’s Internet Crime Complaint Center public service announcement on AI-enabled fraud gives a short checklist that holds up well in practice. None of these controls stop every attack, but each adds friction, and friction is the lever that moves attacker economics. Our deeper piece on the double-edged impact of AI on cybersecurity covers the broader pattern of what works and what fails here.

Risks of AI-Enabled Cyberattacks and Autonomous Phishing

Stepping back from direct fraud, the misuse of AI in general cybercrime has pushed phishing from a volume game to a personalisation game. Language models now draft spear-phishing messages in dozens of languages, mirror the recipient’s writing style, and chain into convincing follow-up threads. The industry data shows a steep rise in opened phishing lures when AI is used to tailor copy. Click-through rates have risen by around three to four times compared with human-written baselines. That rise is modest in abstract terms, but at the scale of billions of messages it reshapes which organisations get breached first. Security teams that still rely on grammar signals or stylometric mismatch as a filter are the first to lose coverage.

Building on that trend, agent-style attackers are now stringing together reconnaissance, pretexting, credential capture, and lateral movement in a single automated loop. Early public reports describe tools that scrape a target company’s org chart, generate a plausible internal narrative, and run the phishing campaign without a human in the loop. The Verizon Data Breach Investigations Report has been tracking the shift and notes that credential-based intrusions remain the dominant initial access vector. Defence against agentic phishing increasingly looks like phishing-resistant authentication, hardware keys, and short-lived credentials, rather than any single AI-powered filter. Security teams also report measurable reductions in agent-driven account takeover when they add out-of-band verification steps that break the automation loop.

Child Safety, Harassment and Non-Consensual Imagery

Shifting to the hardest category to discuss, generative models have put child safety organisations under acute strain. The Internet Watch Foundation reported that AI-generated child sexual abuse material rose sharply through 2024. Over twenty thousand images were confirmed on a single dark-web forum in one month. The scale is new, the harm is not, and the legal position is clearer than many online debates suggest. Possession and distribution of AI-generated child sexual abuse material is already illegal in most jurisdictions, including the United States and the United Kingdom. The gap is enforcement capacity, which has not scaled with generation volume.

Building on that, non-consensual intimate imagery of adults is now a parallel epidemic. School administrators report incidents in which a classmate has used a photo scraped from social media to produce explicit synthetic images of a victim. Civil remedies vary by state, and criminal penalties are catching up unevenly. Platforms have added hashing and takedown pipelines, but the first wave of harm often happens inside closed group chats that platform policy cannot see. Our tracker piece on the growing fight against explicit AI deepfakes covers the state-by-state legislative response.

Beyond takedowns, the strongest preventative lever is upstream training-data hygiene at model providers, not downstream content moderation on platforms. A model that has never seen a given face cannot easily render one. A training set that excludes explicit material against minors cannot easily generate that material either. Several open-source model releases have added provenance watermarks and training-set filters. Enforcement here is likely to shift from platform liability to model-provider liability, as the EU AI Act’s general-purpose-model obligations begin to bite from 2026 onward. The European Commission’s AI Act compliance timeline shows which obligations apply in which year.

Bias, Discrimination and Decision-System Harm

Shifting from external attackers to institutional misuse, biased decision tools remain the oldest and most litigated form of AI harm. Hiring screens, tenant checks, credit models, and predictive policing tools all encode the data they are trained on, which means they encode past discrimination too. The NIST Special Publication 1270 on managing bias in AI walks through the measurement challenges most teams underestimate. Mere accuracy on a holdout set does not catch disparate impact across protected classes. Audit tooling has matured, but legal exposure has risen faster, which is one reason the compliance team now gets a seat at the model-review table in most mid-sized banks.

Beyond audits, the practical fix is to constrain the question the model is asked, not only the data the model is trained on. A hiring model that predicts whether a candidate will succeed over three years inherits every bias in historical promotion decisions. A model that predicts, say, whether a candidate will complete a specific onboarding task has far less latitude to encode legacy bias. That framing shift is cheap, is well understood, and is still routinely skipped because it slows product teams down. Readers who want more depth can review our fundamentals piece on the dangers of AI bias and discrimination, which maps the common audit gaps clearly.

Data Poisoning, Prompt Injection and Model Attacks

Shifting from decisions to the model itself, a smaller but technically interesting class of misuse targets the model as a system. Data poisoning seeds malicious samples into a training set so the released model behaves badly on specific inputs. Prompt injection smuggles instructions into a model’s context window through a web page, email, or document, so the model acts against its operator’s wishes. Model extraction queries a hosted model enough times to reconstruct a usable copy. Each attack class is well documented in the academic literature, and each has already appeared in commercial settings.

Building on that, prompt injection is the one most product teams underestimate because it does not require the attacker to touch training data at all. A single hostile instruction in a Google doc that an assistant later summarises can rewrite the assistant’s downstream behaviour for that user. The common mitigation of asking the model to ignore malicious instructions does not generalise and has been bypassed in every well-run study. Hardened architectures keep untrusted content out of the privileged prompt and strip tool access when the context contains untrusted input. Those changes are architectural and are not a filter you buy, which is why most mitigations shipped so far have been shallow.

Beyond prompt injection, data poisoning has moved from a theoretical worry to a measured one as public image and text corpora grow in importance. Research teams have shown that a tiny fraction of poisoned examples, in some cases well under one percent, can flip model behaviour on narrow trigger phrases. Teams that fine-tune on scraped data without provenance checks are the most exposed. Open ecosystems that pull from community model hubs are also exposed when a popular adapter has been tampered with. Defence looks like supply-chain hygiene applied to models, with signed artefacts, pinned hashes, and reproducible training runs, which is the direction regulated industries are heading.

AI Misuse in Journalism, Research and Academic Fraud

Shifting from systems to knowledge, generative AI has strained the integrity of journalism, academic research, and peer review. Several news outlets have been caught publishing lightly edited AI output without disclosure, usually after a reporter cuts a corner on deadline. The deeper problem is that AI-generated academic papers are entering the literature faster than retraction mechanisms can respond. A pair of 2024 analyses flagged thousands of suspected AI-authored submissions across minor conferences, and several high-profile journals have begun requiring explicit disclosure of model use. Peer reviewers themselves are also increasingly asked to disclose AI assistance, which is still an uneven standard across fields.

Building on that, academic fraud shows a different pattern from newsroom misuse because the incentives are deeper and the audit cycle is longer. Peer review takes months, replication takes years, and a bad paper that nobody can replicate still accumulates citations in the meantime. Reviewers who use AI to speed up their own write-ups further muddy the signal. Universities are rolling out AI-use policies, but the enforceable part is a shared norm. Technical detectors remain brittle and are easy to confound with light editing. Our ongoing coverage of responsible AI governance frameworks shows how similar norms are forming in enterprise settings.

Weaponisation Concerns: Autonomous Systems and Dual-Use Research

Shifting from civilian misuse to national-security concerns, the misuse of AI in weapons systems is the most consequential and the least transparent. Several militaries have deployed AI-assisted targeting in active conflicts, with the Israel Defence Forces’ use of the Lavender system in Gaza drawing the sharpest public scrutiny. Reporting by +972 Magazine on the Lavender targeting system documented how human-review windows were compressed to minutes. Critics argue that an AI recommendation anchored every subsequent human judgement, which is a textbook automation-bias concern. Defenders argue that volume and speed demanded machine support and that human judgement was never removed from the chain.

Beyond targeting, the dual-use research debate has intensified around biology and chemistry. The RAND Corporation’s 2024 report on biological risk from frontier AI models argued that current frontier models do not yet add meaningfully to the capabilities of a motivated non-state actor. That particular finding is still contested across the policy community of researchers and regulators. The real worry is less about any single model’s answer and more about compounding effects when autonomous agents chain scientific tools together. A model that can only answer questions is bounded; a model that can also run experiments, order chemicals, and read sensor data is bounded differently. Dual-use governance has not caught up to that shift in agent-enabled science and research.

Looking at autonomous weapons specifically, the diplomatic conversation runs through the Convention on Certain Conventional Weapons. Progress on a binding ban or on binding limits is slow, and the camps have hardened. Civil-society groups push for a prohibition on fully autonomous lethal systems, major powers favour non-binding statements of principle, and the pace of military adoption continues regardless. Our deeper feature on the AI arms race walks through the geopolitical dynamics driving both sides. Pacific-rim navies and European air forces have both run exercises with autonomous assets at squadron scale, which is a quieter but more durable signal than any single procurement announcement.

Beyond the state actors, non-state misuse of commercial drones and open-source autonomy stacks is already visible in ongoing conflicts. First-person-view drones adapted with cheap target-recognition models have changed the tactical picture in Ukraine, and the same technology is reaching irregular forces elsewhere. The capability gap between a military-grade autonomous system and a well-funded hobbyist stack keeps narrowing. Export-control regimes designed for 2010 technology are poorly matched to today’s parts list. The near-term policy question is less about banning the capability and more about treating the integrators and the training-data vendors as the regulatory choke points.

How Organisations Detect and Measure Misuse of AI

Shifting from policy to operations, enterprise security teams are developing a measurement discipline around AI misuse that borrows from fraud analytics and from traditional security monitoring. The common baseline is an incident taxonomy aligned to the OECD AI Incidents Monitor’s shared vocabulary, which makes cross-firm comparisons possible for the first time. Teams layer on anomaly detection against prompt and output streams, abuse-reporting pipelines that route to a central triage queue, and red-team exercises that stress-test every public interface. None of these controls catches every misuse case, but together they close the fastest-moving gaps. Measurement itself is also the lever that unlocks board-level attention, which is often the deciding factor on resourcing.

Building on that, the single most useful operational metric is misuse incidents per million interactions, broken down by the five misuse categories above. That rate is easy to track, straightforward to compare across quarters, and legible to a non-technical executive. Teams that have adopted it tend to find that two or three categories dominate their traffic and that targeted controls move the needle quickly. The ones that resist a per-million rate often default to raw counts, which rise with usage and so look like regressions even when controls are working. Our coverage of AI governance trends and regulations highlights the overlap between operational metrics and emerging legal reporting duties.

Beyond metrics, red-team exercises are the single most educational tool for leadership. A red team that successfully phishes an executive using the company’s own voice-cloning stack generates a short, undeniable narrative that moves budget. The Anthropic sleeper-agents paper gave a widely cited methodology for probing model-level deception, which many enterprise red teams now adapt. Shared red-team findings across firms, often brokered through industry groups, have begun to produce real network effects on defence. The remaining weak point is still the human workflow around the model, not the model itself, which is why social-engineering drills still deliver the biggest coverage gains per dollar.

Policy and Regulation: EU AI Act, NIST RMF, ISO 42001

Shifting to the legal picture, three documents dominate practical compliance. These are the EU AI Act in its official journal form, the US NIST AI RMF, and the ISO/IEC 42001 management standard. The EU AI Act is a hard-law regulation with a tiered risk framework and sharp penalties, and it applies extraterritorially to any provider placing a system on the EU market. The NIST AI RMF is voluntary, but it is the de facto US baseline and is referenced by federal procurement, state laws, and major private contracts. ISO/IEC 42001 is a certifiable management-system standard that many large firms are adopting as their internal audit anchor. The three overlap enough that a well-run programme can satisfy all three with a shared control library.

Beyond those three, sector-specific rules now bite in a growing list of jurisdictions. New York City’s Local Law 144 covers automated employment decision tools and requires bias audits with public reporting. Colorado’s SB24-205 covers high-risk AI use in consumer-facing decisions across most regulated industries. The Council of Europe’s Framework Convention on AI is the first binding international instrument on AI and sits above national rules. For a reader orienting against this patchwork, our longer explainer on AI governance trends and regulations maps the obligations onto typical corporate roles.

Implementing a Corporate Governance Playbook for Reducing Misuse of AI

Shifting from external rules to internal practice, a working corporate playbook needs five moving parts. First, an AI inventory that lists every production model, its owner, its risk tier, and its most recent review date. Second, a review process gated on sensitive capabilities such as content generation about real individuals, biometric identification, or high-impact decision support. Third, a monitoring system that reports misuse-rate metrics to a named executive each quarter. Fourth, a disclosed red-team programme with external participation at least annually. Fifth, a disclosure and remediation path for employees, customers, and civil-society reporters who spot a misuse.

Beyond the five parts, the single highest-leverage governance move is to put a named human accountable for each AI feature. That person must have authority to pull the feature if misuse rises above the threshold the board set. That accountability is what turns a policy document into a living programme. Firms that have appointed a chief AI officer or an AI risk officer report faster escalation and cleaner audit trails. Firms that have left accountability diffuse report that misuse incidents stay in the engineering backlog for months. The pattern is strong enough that it now appears in several regulators’ guidance explicitly.

Looking at tooling, most of the market is still immature. AI-governance platforms promise everything from automated model documentation to continuous monitoring, but the capabilities vary widely across vendors. A short list of fit-for-purpose tools covers model-cards generation, structured impact assessment, red-team orchestration, and incident reporting. Teams that try to buy a single platform for all five parts of the playbook tend to miss on at least two capabilities. Teams that assemble a small stack of specialist tools and connect them to their existing GRC system see better outcomes. Our responsible AI governance framework overview shows how to sequence that stack.

The Ethics of Deploying AI When Misuse Is Predictable

Shifting from governance mechanics to the harder normative question, the ethics of deploying an AI feature when misuse is clearly predictable sits at the centre of every product review. The classical defence is that any powerful tool has a dual-use character and that benefits to legitimate users outweigh costs from a small number of bad actors. That defence is coherent, but it also requires honest numbers, which product teams often sidestep. Our editorial piece on the ethical dilemmas created by advanced AI argues that the aggregate harm calculus needs to include dignitary harms, not only measurable financial loss. Dignitary harm is harder to count but is not optional to count.

Building on that, a useful ethical check is a pre-mortem. The team imagines the top three misuse paths and asks whether the launch is still net-positive if each one goes mainstream. The test that most consumer AI features fail is the dignitary one, especially when non-consensual imagery or child safety is on the misuse path. Hardening those paths before launch is cheaper than retrofitting after. The dignitary frame is also what best supports the case for pre-release provenance watermarks, model-card disclosures of known misuse channels, and voluntary participation in incident reporting. Teams that bake these into launch readiness avoid the most reputationally damaging incidents. The engineering cost stays small when it is in the roadmap, not an afterthought.

The Future of AI Misuse Through 2030

Turning to what comes next, the misuse of AI through 2030 will look less like viral deepfakes. The real shift will be a quiet rise in base-rate costs for fraud, harassment, and disinformation. Agent-driven misuse will matter more than single-shot generation, because an agent can run a campaign over weeks without supervision. Multimodal misuse that mixes voice, text, and video will become the default, rather than the current single-channel pattern. The research community expects a short burst of highly damaging incidents each year, with the long tail dominating the overall harm count. Our survey of AI privacy concerns and the connected harms shows why a long-tail view beats a worst-case headline.

Beyond those trend lines, detection and provenance tools will mature, and so will attacker evasion. C2PA-style content credentials give an auditable chain of custody, which is useful for high-trust publishers and institutional contexts. Watermarking of model outputs has improved, but a determined attacker can still strip or obscure watermarks on consumer media. The real near-term win is institutional, not technical: publishers, platforms, and governments adopting a shared provenance standard would do more than any single detection breakthrough. The economics of misuse change the moment it costs the attacker time and friction to produce content a verified channel will accept. That is a policy problem as much as a technology one.

Looking at regulation, the EU AI Act’s general-purpose-model obligations will produce the first wave of real audits in 2026 and 2027. US federal legislation remains unlikely in the near term, but state activity will keep accelerating, and sector regulators will use existing authorities aggressively. International coordination on misuse of ai will probably happen through sector-specific channels for now. The Bank for International Settlements handles financial fraud, while the Internet Governance Forum covers content integrity. A single global AI treaty remains unlikely in the near term. That is unglamorous, but sector-specific coordination tends to produce enforceable rules faster than treaty negotiations. Our editorial on the election cycle’s effect on AI risk concern shows how public attention has already pulled regulators forward.

Beyond regulation, the underappreciated variable is user agency, because the most durable defence against misuse of ai is a public that can recognise, pause, and verify. Media literacy initiatives work when they are specific, repeated, and tied to concrete recent cases. Verification norms, such as calling back on a known channel or waiting an hour on a suspicious appeal, are old controls, but they still work. Platform friction, such as requiring a second factor on high-value transfers or on sensitive content shares, is a quiet lever that compounds. Our overview of related harms through chatbot-related teen safety concerns shows how fast consumer protections catch up with a clear harm narrative. The same pattern will repeat across each misuse category above.

Chart From AIplusInfo

AI misuse by category, as mapped in public incident data

Share of recorded generative AI misuse incidents by category. Toggle to compare DeepMind’s two-year map against OECD AI Incidents Monitor’s broader 2024 to 2026 dataset.

Source: DeepMind map of generative AI misuse and the OECD AI Incidents Monitor.

Key Insights on the Misuse of AI

  • The Google DeepMind two-year misuse map found impersonation and opinion scaling make up about seventy-four percent of mapped cases, which reframes misuse of ai as a social-engineering problem.
  • TechRadar on Veriff 2026 data puts deepfake-enabled fraud losses at about 3.7 billion dollars in 2026 alone, which shifts the harms of ai into a measurable annual line item.
  • The Internet Watch Foundation annual report documented over twenty thousand confirmed AI-generated child sexual abuse images on one forum in a single month, which forced agencies to rebuild takedown pipelines.
  • The OECD AI Incidents Monitor covers a Gartner forecast that forty percent of AI data breaches by 2027 will come from cross-border generative AI misuse, raising jurisdictional complexity for multinationals.
  • Public Citizen’s state deepfake-legislation tracker shows about forty-four US states introduced deepfake-related bills by 2025, which signals how fast state law outruns any federal rule on AI misuse.
  • The FBI Internet Crime Complaint Center announcement reports AI-driven voice-cloning attacks against finance teams at roughly four times the 2023 baseline, which has reshaped controller onboarding across banking.
  • Gartner’s February 2025 release found only twenty-four percent of enterprises had a formal AI incident-reporting process by late 2025, which exposes a measurement weakness most boards still underrate.

Pulling those insights together, the misuse of AI already behaves like a mature, cost-sensitive attacker ecosystem rather than a frontier research problem. The dominant tactics are old ones (impersonation, opinion-shaping, fraud, non-consensual imagery) scaled by cheap generation and cheap targeting. The dominant defences are also old ones, from phishing-resistant authentication to callback verification, now layered with new tooling for provenance and incident response. The gap is measurement discipline, which is improving slowly but has not caught up to the scale of generation. The next five years will reward organisations that treat AI misuse as a continuous operational metric, not a periodic crisis. Technical controls will matter most when paired with disclosed accountability and transparent reporting.

Comparing Categories of AI Misuse Across Dimensions

Shifting from the ranked insights to a side-by-side view, this table compares how the five categories of ai misuse behave across frequency, defence, measurability and policy. Each dimension is one that enterprise and policy teams routinely track and report on. The table is intentionally sparse so that cross-category trade-offs are legible at a glance. The purpose is to make those trade-offs legible without hiding the fact that defences and metrics vary widely across them. Readers can use the table to spot which category of misuse of AI is most relevant to their own sector today. The underlying weights are those used in the misuse exposure calculator above.

DimensionDeepfake impersonationOpinion scaling / disinformationDecision-system biasModel attacks (poisoning, prompt injection)Weaponisation / dual-use
Reported frequencyHighest in incident databasesSecond highest and risingMid, legal exposure driving reportingLow but growing fastLow in public data, higher classified
Primary harm typeFinancial and dignitaryCivic trust and public opinionEconomic exclusionSystem integrity and data lossPhysical and geopolitical
Dominant defenceCallback verification, hardware keysContent provenance, platform policyAudits, framing changes, pre-deploy testingSupply-chain hygiene, hardened promptsTreaty, export control, procurement rules
Measurable in public dataYes, via loss reportsPartial, via platform transparencyPartial, via disparate-impact studiesPartial, via academic benchmarksLow, mostly classified or redacted
Regulatory maturityFast moving at state levelPatchy, platform-ledMature in employment and creditEarly, sector-specificTreaty-stage, slow
Cost to attackerVery low after one-time setupLow per message, high at scaleMid, requires training dataMid, requires technical skillHigh, state-level resources
Public awarenessHigh in US and EU, uneven elsewhereRising since 2023Long-standing but unevenLow, technical audience onlyHigh among policy audiences
Likely 2030 trajectoryAgent-driven, multimodalVerified-sender shiftConverging with audit standardsSupply-chain focus deepensSector coordination over treaty

Real-World Examples of AI Being Misused

Stepping back from framing into practice, three recent incidents show the misuse of AI across different attack surfaces. Each case ties a specific tactic to a measurable outcome. Each one has shaped how practitioners now train and defend around that misuse category.

The Arup Deepfake Video-Call Fraud

Arup, the global engineering firm, deployed a crisis response after about twenty-five million dollars in transfers. A Hong Kong staffer authorised fifteen payments on a deepfake video call impersonating the CFO and other executives. The loss represented a 100 percent increase over the firm’s prior annual fraud losses, which bypassed the firm’s email-only phishing controls. The measurable outcome reshaped how Fortune 500 treasury teams now train on video-call fraud across every region. A limitation is that the case says little about how the attackers acquired the training footage used for the impersonation. The firm has since mandated a second-channel callback on any wire transfer above a defined threshold, which many peers have adopted since.

The Biden New Hampshire AI Robocall

In January 2024, voters across New Hampshire received an AI-cloned robocall imitating President Biden. The Federal Communications Commission ran a Declaratory Ruling banning AI-generated voices in robocalls within weeks of the incident. The agency deployed its Telephone Consumer Protection Act authority to treat the generated voice itself as the illegal element. The measurable outcome was a 6 million dollar civil penalty against the political consultant and a 2 million dollar penalty against the voice-service vendor. A limitation is that the ruling applies only to robocalls under the TCPA, so AI-generated political video on social media still sits under a state-law patchwork. The case still shifted federal enforcement from the harm layer to the capability layer, which has echoed through other agencies since.

Public Comment Flooding at US Federal Agencies

Public comment dockets at several US federal agencies deployed new intake pipelines against large volumes of apparently AI-generated submissions through 2024. These implemented pipelines filtered a reported thirty percent of inbound records. The Government Accountability Office documented the comment-fraud mechanics in detail. The measurable outcome is a new verification layer at multiple agencies that flags duplicates and clusters semantically similar submissions. A limitation is that these systems can over-filter authentic voices who happen to use similar talking points supplied by advocacy groups. The practice is still evolving and remains among the quieter but higher-stakes misuse-of-ai cases for democratic participation.

Recommended by AIplusInfo

Books to go deeper on the misuse of AI

Three widely cited titles that map the political, economic, and technical ground this article covers.

As an Amazon Associate, AIplusInfo earns from qualifying purchases.

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

Book

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

O’Neil’s foundational text on biased decision systems maps the oldest and most litigated class of AI misuse.

Buy on Amazon
AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference

Book

AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference

Narayanan and Kapoor separate genuine AI capability from hype and misuse, directly mapping to this guide’s framing.

Buy on Amazon
Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence

Book

Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence

Crawford’s Yale UP survey covers the political economy of AI misuse, from data labour to environmental harms.

Buy on Amazon

Case Studies in AI Misuse and the Response

Looking at how institutions responded, three case studies show the full arc from problem to policy effect. Each one maps a sustained misuse of AI campaign to a measurable response. Each has influenced the broader governance picture in ways that outlive the single case.

Case Study: Internet Watch Foundation and AI-Generated CSAM

The Internet Watch Foundation, a UK child-protection charity, faced a sharp operational problem in late 2023. Open-source image models began producing photorealistic child sexual abuse material on dark-web forums at a pace that overwhelmed existing hashing pipelines. The IWF AI CSAM report documented more than twenty thousand confirmed AI-generated images in a single month, with roughly three thousand in the most severe category. The organisation deployed a solution with three parts, each shaped by the forum data, which reduced duplicate takedown effort by a measurable percent over the first full quarter. Specialist analysts were trained on model-generated indicators, and the IWF partnered with model providers on training-data filtering and advised UK legislators on updating the Protection of Children Act. The measurable impact was faster takedown response and a visible shift in several major open-model release checklists toward upstream filtering of training content. The IWF now publishes a quarterly data brief that gives platforms a comparable metric and has reduced duplicate takedown effort across member services.

A limitation and ongoing controversy is that technical measures at model providers remain partial, and open-weights releases outside major vendor control continue to pose a dedicated risk pathway. Enforcement capacity, especially for cross-border forums, is also structurally limited, since the originating jurisdiction may not criminalise generated material even where the destination jurisdiction does. The IWF has advocated for a shared international reporting standard, which is still in early draft form. The case shows both the strongest institutional response in any misuse category and the hardest residual problem, since any effective fix requires action by model providers upstream of the harm. The case has shaped UK Online Safety Act implementation and several EU AI Act delegated acts. That is a direct policy outcome, not only an operational one, and shows how the solution scales.

Case Study: NAACP Lawsuit Against xAI's Memphis Facility

The NAACP and the Southern Environmental Law Center faced a clear environmental-justice problem. In 2025 they filed suit against xAI's Colossus data-centre operation in Memphis, alleging that methane-gas turbines caused outsized air-quality harm in majority-Black neighbourhoods. The NAACP's own press release details the plaintiffs' environmental-justice claims against xAI, which centre on the Clean Air Act and on local permitting disclosures. The solution the plaintiffs seek is a halt to additional turbine deployment, binding emissions monitoring, and community remediation, backed by federal and state enforcement authority. The measurable impact so far has been injunctive, with a preliminary ruling slowing turbine expansion by about thirty percent and triggering public disclosure of previously confidential operating-permit filings. The case opens a new legal pathway for environmental-justice plaintiffs. It treats the infrastructure of AI deployment, not only the model outputs, as a locus of misuse of ai where disparate impact can be litigated. The measurable impact already shapes other AI facility-siting negotiations across the US.

A limitation is that the case is still in motion as of this writing, and the final ruling could be narrowed to specific permit violations rather than establishing broader precedent. Industry defenders argue that the facility complies with all state permits. They note that its emissions are a small fraction of regional industrial output. The siting decision, they add, followed standard commercial logic rather than any attempt to shift harm. Those arguments do not resolve the environmental-justice critique, which looks at cumulative impact on a specific community rather than at single-facility compliance. The case has already changed how several AI infrastructure projects negotiate community-benefit agreements, which is a quieter but durable change. Our coverage of the underlying NAACP suit against xAI tracks the filings and the broader civil-rights framing in detail.

Case Study: Clearview AI Enforcement Across the EU

Clearview AI, a facial-recognition vendor, created a cross-border enforcement problem by scraping billions of images from public web sources. The company drew enforcement actions from multiple European data protection regulators between 2021 and 2024, which deployed a coordinated solution for operating a biometric system without legal basis. The European Data Protection Board documented the Dutch DPA's thirty-point-five million euro fine against Clearview AI. Similar enforcement actions followed in France, Italy, and the United Kingdom through the next two years. The regulators treated the entire data pipeline, from scraping to retention to matching, as the actionable misuse, since no European legal basis for mass biometric scraping existed. The measurable impact across the sequence of actions exceeds about forty-five million euros in fines plus binding deletion orders, which has materially constrained Clearview's European business. The case is widely cited as an early template for how the EU AI Act's biometric prohibitions will likely be enforced from 2026 onward.

A limitation is that Clearview is based outside the EU and has repeatedly declined to pay or to comply, which exposes the fundamental cross-border weakness in current data protection enforcement. The regulators have pursued local directors, local processors, and local resellers, which has pushed the practical compliance cost higher even when the fines themselves remain unpaid. Civil liberties groups argue that the real test is extraterritorial reach under the EU AI Act. The European Court of Justice has not yet fully resolved that question, but each new fine builds pressure for a clearer ruling. The case nonetheless rewrote the global playbook on scraped-biometric startups, since any firm entering the European market now faces a known enforcement posture. The broader AI privacy picture on our site traces how the Clearview response rippled into HR, retail, and public-safety procurement.

Common Questions About the Misuse of AI

What is the misuse of AI, and how is it different from an AI mistake?

The misuse of AI is a deliberate human choice to use a model to deceive, defraud, or harm. A model mistake is an unintended error from the system. The distinction matters legally because misuse carries intent, which drives most criminal and civil remedies. Treating every bad AI outcome as misuse muddles both policy and engineering responses.

What are the most common misuse of ai examples right now?

The five most reported categories are political deepfakes, voice-cloning fraud, non-consensual intimate imagery, phishing scaled by language models, and biased decision tools. These account for the bulk of logged incidents in public databases. The exact ranking shifts quarter to quarter, but the mix has stayed consistent since 2023. Each category has its own defensive playbook, which the article maps in detail.

How much money is lost each year to AI-enabled fraud?

Deepfake-related fraud losses crossed about 3.7 billion US dollars in 2026 by Veriff's industry tracking. That figure almost certainly understates the real total because many incidents go unreported for reputational reasons. Elder-fraud, HR onboarding fraud, and corporate wire fraud are the three largest sub-categories. Loss figures are rising faster than reporting infrastructure, so expect revisions upward.

Is AI-generated child sexual abuse material already illegal?

Yes, in the United States, the United Kingdom, and most European jurisdictions, AI-generated child sexual abuse material falls under existing criminal statutes. The real challenge in these cases is enforcement capacity rather than legal theory alone. Several US states and the UK are updating law to make the position explicit for prosecutors. Enforcement is converging on model providers and distribution platforms as the practical choke points.

Which law governs misuse of AI in the European Union?

The EU AI Act governs AI misuse at the EU level through tiered risk categories, prohibited practices, and penalties up to seven percent of global turnover. Member-state data protection authorities also use the General Data Protection Regulation for biometric misuse. Sector regulators layer on their own rules in finance, health, and transport. The combined effect is one of the strictest AI-misuse regimes in the world.

What is the EU AI Act's main response to AI misuse?

The AI Act prohibits specific practices such as social scoring and real-time biometric identification in public spaces, with narrow exceptions. It imposes tiered obligations on high-risk systems and on general-purpose models. Penalties reach up to seven percent of global annual turnover, which is higher than GDPR. Compliance timelines began in 2025 and extend through 2027 for different obligations.

How can organisations detect misuse of their own AI systems?

Operational teams track misuse rate per million interactions, broken down by category. They layer in anomaly detection on prompt and output streams, central triage queues for abuse reports, and periodic red-team exercises. The single most useful control is a named human accountable per feature. Measurement discipline is still the main gap in most mid-sized organisations.

What role does the US government play in regulating AI misuse?

The US government currently uses sectoral regulators, state law, executive orders, and the voluntary NIST AI RMF rather than a federal statute. The FCC and FTC have begun treating specific AI misuse behaviours under existing authority. State activity is accelerating, with over forty states introducing deepfake-related bills in recent sessions. Federal AI misuse legislation remains unlikely to pass in the near term under current conditions.

Can watermarking and content provenance stop AI misuse?

Watermarking helps high-trust institutional contexts but does not stop determined misuse because watermarks can be stripped or ignored. Content Credentials based on C2PA provide auditable provenance, which is useful when publishers and platforms adopt it. The real near-term win is institutional adoption of a shared standard across major channels. Technical measures alone will not close the gap without that institutional layer.

How does AI misuse compare across hiring, lending, and policing systems?

All three domains suffer from decision-system bias, but the regulatory posture differs sharply. Hiring is covered by NYC Local Law 144, Colorado SB24-205, and civil-rights law. Lending falls under the Equal Credit Opportunity Act and sectoral fair-lending rules. Policing varies by jurisdiction, and several cities have banned facial-recognition policing outright. The common defensive move is a required bias audit before deployment.

Are large language models themselves responsible for the misuse of AI?

Models are tools, so responsibility sits with the human or organisation using them for a given harm. Model providers do carry a duty of care, which includes training-data filtering, misuse-mitigation features, and incident reporting. The EU AI Act's general-purpose-model obligations formalise that duty of care from 2026. Litigation is also pushing US model providers in a similar direction without legislation.

What should individuals do to protect themselves from AI misuse?

The highest-value personal controls are phishing-resistant authentication, shared code words with family and finance teams, and separate-channel callbacks on urgent voice or video requests. Treat any emotionally charged inbound message as unverified until confirmed through a known second channel. Media-literacy habits also help, including a short pause before sharing suspicious content online. These controls sound simple, but they close most consumer-grade attack paths.

Where is misuse of AI likely to concentrate in the next five years?

Expect agent-driven misuse, multimodal campaigns mixing voice and video, and continued pressure on child-safety and election-integrity institutions. The weaponisation debate will keep moving slowly through international channels. Sector-specific coordination will do more than any single global treaty in the near term. Verified-sender infrastructure and misuse rate metrics will become operational standards for mature organisations.