Dangers of AI – Lack of Transparency

Introduction

The dangers of AI – lack of transparency have moved from academic worry to boardroom liability in under three years, and 2026 is the year the bill starts to land. A 2024 McKinsey global survey found that only 18 percent of organizations deploying generative AI had a formal board or council reviewing model behavior, even as adoption more than doubled. The gap between what these systems now decide and what deployers can actually explain about their reasoning is widening. Regulators, courts and enterprise buyers are treating that gap as a defect rather than a feature. This article walks through why modern AI became opaque, where opacity turns into concrete harm, which explainability methods actually help, and what a working transparency program looks like. It draws on primary sources, real enforcement actions, and named case studies rather than generic warnings. If you build, deploy, buy or oversee AI, the black box is now your problem, and this piece is a practical map through it.

Quick Answers on the Dangers of AI – Lack of Transparency

What does lack of transparency in AI mean in one sentence?

Lack of transparency in AI means users, deployers, and often regulators cannot see why a model produced a specific output, which breaks the normal chain of accountability, appeal, and safety oversight.

Why are so many modern AI systems opaque by default?

Deep learning systems learn millions of interacting parameters from data, so their reasoning is distributed. Vendors add a second layer of opacity through trade secret protection over training data, weights, and evaluation results.

What can I do about the lack of transparency in AI systems I already run?

Add model cards, per decision reason codes, audit logs, and human review for high stakes outputs. Combine these with independent testing and vendor documentation to build a transparency layer without rebuilding the underlying model.

Key Takeaways

• The lack of transparency in AI is now a regulatory, commercial and safety problem, not a philosophical one.
• The EU AI Act’s Article 13 forces high risk providers to design systems that deployers can actually interpret and audit.
• Explainable AI methods reduce opacity but no post hoc tool converts a large model into a fully transparent system.
• Real world harms are already documented in healthcare, credit, hiring and criminal justice, and courts are increasingly siding with affected people.

What Is the Dangers of AI – Lack of Transparency Problem

The dangers of AI – lack of transparency problem is when users, deployers, and regulators cannot see why an AI system produced a specific output, which blocks accountability, appeals, and safety oversight in decisions affecting health, money, freedom, or livelihood.

Why Modern AI Systems Became Opaque

Beyond raw model complexity, the modern AI stack layered several forms of opacity on top of one another. Vendors treat weights, training data, and evaluation results as trade secrets, which locks buyers out of the very information they would need to audit a decision. Enterprise licensing agreements often forbid the deployer from reverse engineering outputs or running independent evaluations, and public sector procurement contracts have carried the same clauses. The result is that even a large hospital or bank buying an AI system may see nothing beyond an API response and a marketing datasheet.

Deep learning also increased opacity by design rather than by accident. A gradient boosted tree with a few dozen features can be inspected feature by feature, while a hundred billion parameter transformer distributes reasoning across attention heads that have no clean human label. Practitioners call the difference intrinsic interpretability against post hoc explainability, and the second category is inherently a best guess at what happened inside the model. The 2025 arXiv survey Transparent AI: The Case for Interpretability and Explainability catalogs how even leading methods often produce inconsistent attributions for the same input.

Institutional incentives also pushed teams away from transparency during the growth phase of large models. Publishing a full training data provenance report exposes copyright litigation risk, and publishing detailed evaluation results across demographic slices invites lawsuits under civil rights law. Vendors chose thin technical reports over full model documentation, and enterprise buyers accepted the trade off because competitors offered no better disclosure. That equilibrium is now breaking under regulatory pressure, but it explains why the current baseline is a stack of opaque components layered on top of each other.

Inside the Black Box: How Complex Models Hide Their Reasoning

Transition from the market context to the technology, and the picture sharpens further. A modern large language model performs matrix multiplications across billions of parameters for every token it produces, and there is no small subset that a human can read as the reason. Attention heads spread the work across many positions, feed forward layers mix concepts non linearly, and the same weight participates in thousands of unrelated computations. The distribution of reasoning is a feature that lets the model generalize, and it is also the reason opacity is so hard to eliminate.

Post hoc explainability methods reconstruct plausible reasoning after the fact rather than reading the true internal path. SHAP values report how each input feature would have changed the output on average, LIME fits a small local model to approximate behavior around a specific input, and counterfactual explanations describe the nearest change that would flip a decision. Each method is useful and each has known blind spots, including instability where two runs of the same explanation produce visibly different rankings for the same input. Practitioners who treat these outputs as certainties overstate what the model has actually revealed.

Interpretability from the model’s structure is the strongest guarantee, but it comes with real accuracy trade offs on complex problems. A small decision tree scoring loan applications is transparent by construction and easy to audit, while a gradient boosted model on the same task often lifts accuracy by several points. Domain teams weigh that gap against the reputational and legal cost of unexplainable denials, and the trend since 2024 has been to keep interpretable models for high stakes decisions and let large models handle low stakes tasks such as ranking or routing.

The stack also includes a set of documentation practices that reduce opacity even when the model itself remains complex. Model cards summarize intended use, training data, evaluation slices and known limitations, and datasheets for datasets describe collection methods and demographic composition. These artefacts do not open the black box, but they let a reviewer see what the system was designed to do, on which population it was tested, and where it is known to fail. The what is explainable AI primer on AIplusInfo walks through the practitioner toolkit in more depth.

Where Opacity Turns Into Real Risk and Harm

Opacity is not evenly harmful across use cases, and framing the problem as either all or nothing misses the operational reality. A recommender that ranks movies has different stakes from a system that decides whether a patient gets a sepsis alert or a mortgage. The rest of this article focuses on the specific settings where a lack of transparency in AI reliably turns into concrete harm, so that governance decisions can be prioritized rather than diluted.

The pattern is consistent across sectors. Where the AI decision materially affects health, money, freedom or livelihood, opacity strips people of the ability to contest a wrong answer, prevents deployers from catching systemic errors, and denies regulators the evidence they need to enforce rules. The next sections walk through the settings where this pattern is most damaging, drawn from documented cases and regulator actions rather than hypothetical scenarios.

Bias Concealed Behind Unexplainable Decisions

Bias in AI systems predates the deep learning wave, but opacity is what allowed it to survive at scale. A model that offers no reason codes cannot be audited by the people affected, and a system that scores millions of decisions per day can compound a small statistical bias into a systemic pattern before anyone notices. Civil rights organisations have documented dozens of such cases across finance, hiring and housing since 2018, and few of them were caught by the vendors themselves.

The problem worsens when opacity meets protected class variables that never appear directly in the model. Zip code, education history, browser type, and dozens of other proxies often carry demographic signal that regressors can pick up implicitly. Vendors then defend their systems as neutral because no protected attribute was fed in, while affected people see decisions that break down along racial or gender lines. The Brookings review of algorithmic bias mitigation is one of the clearer breakdowns of how this proxy channel operates in credit and insurance.

Where regulators or plaintiffs eventually secure access to model outputs and slicing analyses, the pattern is usually visible within weeks. That inversion is important: opacity is what protected these systems, not their actual performance. The NAACP sues xAI over racial bias filing on AIplusInfo is one of the more recent US examples where legal action forced disclosure that internal governance had missed.

Healthcare AI When Doctors Cannot See the Reasoning

Turn from finance to medicine and the stakes rise sharply. Clinical AI systems now assist in triage, imaging, sepsis prediction, and treatment recommendation, and each of these decisions can move survival probabilities. When the model presents a probability score without the reasoning, the treating clinician must either accept it or override it without knowing which patient features drove the answer. This is the opposite of how evidence based medicine is supposed to work.

The most cited example is the proprietary sepsis alert model deployed by Epic Systems, whose external validation in JAMA Internal Medicine external validation of the Epic sepsis model found sensitivity of 33 percent and a hospital reported positive predictive value near 12 percent in a study of nearly 40,000 hospitalizations. Clinicians had operated the alert for years without independent evidence of its accuracy, in part because the vendor treated the model as proprietary. The result was thousands of alerts that either fired too late or fired on patients who did not have sepsis, with real consequences for care.

Regulators reacted. The FDA moved to strengthen guidance on Software as a Medical Device, hospitals began requiring vendor transparency artefacts before renewing contracts, and the American Medical Association issued policy calls for interpretable clinical AI. The FDA approval and regulation of AI healthcare tools article on AIplusInfo tracks how the agency is now treating adaptive AI as needing ongoing post market monitoring rather than a one time clearance.

Clinicians also carry the risk personally. A malpractice case where a doctor accepted an AI recommendation without the ability to explain it now runs into professional standard of care questions that a transparent system would not create. Hospitals that adopted opaque models to speed up decisions have quietly slowed adoption while they negotiate documentation and reason code access with vendors. The AI health advice study flags risks update covers the parallel research on consumer facing medical chatbots, where the transparency gap is even wider.

Credit and Lending: The Cost of Denials You Cannot Contest

Move to consumer finance and the transparency gap takes a different shape. Under the US Equal Credit Opportunity Act, a lender denying credit must provide adverse action notices with specific reasons, and the Consumer Financial Protection Bureau has warned that vague or generic reason codes generated from opaque AI models do not satisfy the law. Vendors that trained boosted or neural models on thousands of features often struggled to produce the four or five specific reasons the statute requires.

The Apple Card investigation from 2019 through 2021 is the touchstone example. Applicants reported that spouses in the same household with similar or identical financial profiles received wildly different credit limits, and the underwriting model provided no reasoning either applicants or Goldman Sachs staff could interpret. The New York Department of Financial Services closed its investigation without a finding of statutory discrimination, but the probe still forced disclosure of governance gaps and prompted Goldman Sachs to add manual review for anomalous decisions.

Since then the CFPB has published a series of circulars requiring lenders to give specific reasons even when using AI, and several fintech vendors have added explanation modules that convert SHAP values into human readable reason codes. The misleading AI ads cloud consumer trust report on AIplusInfo tracks how consumer regulators are extending the same reasoning duty into marketing and advertising claims about AI features.

Criminal Justice Algorithms and the Right to an Explanation

Turn to the criminal justice system and the transparency deficit meets constitutional rights directly. Risk assessment tools such as COMPAS have informed bail, sentencing, and parole decisions across multiple US states since the early 2010s, and their internal weights have been protected as trade secrets. Defendants have argued in state and federal courts that reliance on a scoring model whose logic they cannot examine violates due process, and the case law is still forming.

The Wisconsin Supreme Court in State v. Loomis allowed the use of COMPAS with cautionary language rather than banning it, but the ruling made clear that judges must not treat a proprietary score as a substitute for their own reasoning. Subsequent audits by ProPublica and academic researchers reported disparate false positive rates across racial groups, and the debate over whether the underlying model is discriminatory is unresolved because the vendor refuses full disclosure. The California Law Review analysis of algorithmic opacity summarizes the constitutional stakes in more depth.

Hiring Systems That Reject Candidates Without Reason

Hiring has become the most heavily regulated frontier for opaque AI in the US, and New York City set an early precedent. Local Law 144 requires employers using automated employment decision tools to complete an annual bias audit and disclose usage to candidates, and enforcement began in July 2023. Compliance has been uneven, with several employers reportedly claiming their tools did not meet the ordinance’s narrow definition, but the direction of travel is clear.

Colorado, Illinois and California followed with expanded requirements that force employers to notify candidates when AI is used, retain evaluation results, and grant appeal rights where practical. The problem is that most hiring AI systems on the market are opaque by construction, so retrofitting them for these disclosure duties has been expensive and slow. The AI law ignored in NYC hiring report on AIplusInfo lays out how uneven enforcement has been through 2025.

Amazon’s now discontinued recruiting model is the historical baseline. Reuters reported in 2018 that the internal tool downweighted resumes containing the word women, apparently because the training data reflected years of male dominated hiring. Amazon reportedly scrapped the model in 2017, but the case became the reference example for how opacity plus biased training data plus scale can produce systemic disadvantage that nobody set out to build.

The Explainable AI Toolkit for Practitioners

Transition from documented harms to the practitioner response, and the tools narrow to a working shortlist. SHAP, developed by Scott Lundberg and Su In Lee, produces feature attributions grounded in game theory, and it has become the default post hoc method for tabular models across finance and healthcare. LIME, developed by Marco Tulio Ribeiro and colleagues, fits a small local model around a specific prediction to explain that decision alone. Both help satisfy documentation duties without redesigning the underlying model.

Counterfactual explanations take a different angle. Instead of listing which features mattered on average, they describe the smallest change to the input that would flip the decision, which is often more useful to the person who wants to appeal or adjust behavior. A denied loan applicant told that raising annual income by 6,000 dollars or reducing revolving debt by 3,400 dollars would have led to approval learns something they can act on, whereas a SHAP bar chart of feature importance is harder to translate into next steps.

Documentation artefacts sit alongside these algorithmic methods. Model cards, popularized by Google Research in 2019, describe intended use, evaluation slices, and known limitations in a short standard format. Datasheets for datasets, from Microsoft Research the same year, describe collection methods, demographic composition, and known problems in the training data. The responsible AI governance framework primer walks through how organizations combine these artefacts into a documentation package.

Newer approaches try to fold interpretability back into training rather than bolting it on afterwards. Concept bottleneck models force the network to route reasoning through named human interpretable concepts, and sparse autoencoders on transformer activations have begun to expose meaningful internal features in large language models. Neither approach has fully solved the black box problem, but they suggest a research direction where interpretability is a first order design objective rather than a compliance afterthought.

Global Regulatory Response to Opaque AI

Regulators have moved from principles to enforceable rules over the past three years. The EU AI Act, in force from August 2024 with staged obligations through 2027, is the most detailed regime and directly targets the lack of transparency in AI. General purpose AI model transparency rules applied from August 2025, and high risk system obligations begin in August 2026. Article 13 of the EU AI Act requires high risk providers to design systems that let deployers interpret outputs, provide detailed instructions for use, and log operations for audit.

The US operates through a patchwork rather than a single act, but the direction is similar. Federal agencies including the CFPB, EEOC, and FTC have issued explicit guidance that opaque AI is not a shield against existing consumer protection, employment, and antitrust laws. State laws in Colorado, California, and New York now impose specific transparency and audit duties on employment, insurance, and consumer AI use. The California leads charge on AI regulation update tracks the most active state legislature on this front.

Other jurisdictions round out the picture. The UK’s approach through 2025 has been sector by sector rather than horizontal, with the FCA and the Information Commissioner’s Office publishing separate guidance for finance and data protection. Canada’s Artificial Intelligence and Data Act is progressing through Parliament, and Brazil’s AI framework mirrors much of the EU model. The UK plans unique AI regulation strategy and white house unveils AI regulation rules updates on AIplusInfo cover the state of play in each region.

Corporate Governance and Ethics Structures That Force Transparency

Corporate governance moves at a different tempo from regulation, but 2025 saw the first wave of board level AI risk committees at large US public companies. These committees typically review new deployments before they go live, mandate model cards and evaluation results, and require an incident response plan for AI specific failures. The pattern echoes how cybersecurity governance matured in the 2010s, and the timelines are compressing because AI incidents are more visible to customers than security incidents were.

Below the board, a working transparency function combines model risk management, legal review, product management, and an independent audit line. The future roles for AI ethics boards update explores how these functions are being staffed at scale. Where the internal audit function was previously financial, the AI extension requires new skills including ML evaluation, disparate impact analysis, and adversarial testing that few audit teams currently have.

Trade Secrets Versus the Public Interest in Model Disclosure

Vendors have historically defended opacity through trade secret law, arguing that disclosing weights, training data, or reason codes would give competitors an unfair advantage. That defense is starting to erode. Courts in the EU have already ordered disclosure in high stakes public sector cases, and the EU AI Act balances trade secret protection against the deployer’s need to understand the system for legal and safety reasons.

In the US, sectoral regulators have chipped away at the same defense. The FTC has treated vendor secrecy as unfair when it prevents consumers from understanding decisions that affect them, and the FDA has required documentation for AI medical devices that vendors previously refused to publish. Enterprise procurement contracts have begun to include audit rights and escrow of model artefacts, so that if the vendor fails, the deployer can still meet regulatory duties.

The public interest side of the balance has strengthened too. Academic research demonstrating disparate impact, journalism exposing hiring failures, and civil society litigation have made a coordinated case that opacity in decisions affecting rights is not sustainable. The AI governance trends and regulations overview on AIplusInfo tracks how the public and private sides of this argument are converging.

Putting AI Transparency Into Practice: Implementation Steps

Turn from principle to practice and the near term work is concrete. A minimum viable transparency layer for a single production model covers four items: a model card, per decision reason codes derived from SHAP or similar, an audit log that records inputs and outputs for a defined retention period, and a human review path for outputs above a defined risk threshold. Teams can typically build this in eight to sixteen weeks without touching the underlying model.

A mature program extends further and takes twelve to eighteen months. It includes vendor management with documented transparency artefacts as procurement requirements, cross functional review boards, disparate impact testing on a defined cadence, and a public facing summary of models in use where the sector allows it. The dangers of AI privacy concerns companion article walks through how transparency and data protection duties reinforce each other in a mature program.

Where legacy systems block transparency retrofit, the practical answer is often to run parallel simpler models and log divergence rather than force the incumbent to explain itself. A logistic regression sitting alongside a boosted model can catch obvious failures in the more complex system, and the simpler model becomes the fallback path when regulators, customers or courts demand a reviewable answer. That pragmatic architecture is now standard practice across the largest US and EU banks.

What Executives Should Track Before AI Goes Live

Executives sponsoring AI programs face a short list of questions that surface the transparency posture quickly. These include which specific decisions the model will inform, what appeal or override path exists for affected people, which documentation the vendor supplies, and what post market monitoring is planned. Every one of these is answerable in a page, and refusal or vagueness from a vendor is a leading indicator of governance risk.

The second question set is internal. Does the deployment plan include disparate impact testing on the population that will actually be scored, and are the evaluation slices the vendor provided representative of that population. Does the incident response plan cover AI specific failure modes such as concept drift and adversarial manipulation. The regulatory science meets AI readiness overview on AIplusInfo covers the kind of playbook mature organizations now attach to every material AI go live.

The Future of Transparent AI Through 2030

Turn to the horizon and the trajectory is easier to see than a year ago. The EU AI Act’s staged enforcement will produce the first concrete fines and market withdrawals by 2027, and the enforcement pattern will influence sectoral rulemaking worldwide. Vendors that ship model cards, evaluation results, and audit interfaces as first class product surfaces will price at a premium, and vendors that refuse will lose material enterprise business.

On the research side, interpretability is now a top three funded area in academic ML, and the interpretability teams inside frontier labs have grown from a handful of researchers to full departments. Mechanistic interpretability work on transformer circuits, sparse autoencoders on activations, and concept guided model design are all producing usable results. None of this eliminates the lack of transparency in AI, but each step narrows the gap between what a model does and what a human reviewer can see.

Executives should plan for a landscape where transparency is table stakes rather than a differentiator by 2028. Procurement processes will require documented model cards, per decision reason codes, and independent audit results, and the absence of these will lock a vendor out of regulated sectors. Organizations that build their internal transparency competence now will have a materially cheaper path to compliance and a materially lower rate of avoidable incidents than those that wait for enforcement to force the change.

Key Insights on the Transparency Crisis in AI

• The Transparent AI arXiv survey confirms that most large models remain functionally opaque despite years of XAI research, and post hoc methods provide only approximate explanations rather than the real internal path a model followed.
• The JAMA Internal Medicine external validation of the Epic sepsis model found sensitivity of 33 percent on 27,697 patients across nearly 40,000 hospitalizations, exposing how opacity delayed independent scrutiny for years.
• The Netherlands SyRI ruling from February 2020 struck down a welfare fraud algorithm partly because its opacity itself violated Article 8 of the European Convention on Human Rights, setting a precedent that lack of transparency can void a system on its own.
• The Brookings report on algorithmic bias documents that models trained without demographic slicing often exhibit multi point disparities in false positive rates once auditors force disclosure, evidence that opacity conceals problems rather than solving them.
• The EU AI Act Article 13 imposes concrete transparency duties on providers of high risk AI systems, with fines that reach 15 million euros or 3 percent of worldwide annual turnover for non compliance under Article 99.
• The California Law Review analysis of algorithmic opacity details how US due process doctrine is being tested by criminal justice risk assessment tools whose logic defendants cannot examine, with unresolved outcomes.
• The Frontiers review of algorithmic discrimination catalogs at least twelve distinct US enforcement actions since 2019 tied specifically to lack of AI transparency in hiring, lending, and housing.

Pull these threads together and a coherent story emerges rather than a scatter of complaints. The lack of transparency in AI is a compounding failure across technology, procurement, and governance, not a single missing feature. Every documented case shows the same anatomy: an opaque model gets deployed for real decisions, harm accumulates below the threshold of internal review, and disclosure comes only after litigation, journalism, or a regulator forces it. That pattern has held across five different sectors for eight years, which is what turns it from an anecdote into a structural risk. The right response is to build transparency into procurement and monitoring from day one rather than treat it as a fix after an incident.

Comparing Transparent and Opaque AI Systems

Dimension	Transparent AI system	Opaque AI system
Reasoning visibility	Per decision reason codes with feature attributions	API returns only a score or class, no reasoning
User participation	Documented appeal path with meaningful review	No appeal or a token review with no effect
Trust posture	Users can inspect the model card and evaluation slices	Users take vendor claims on faith
Decision making rigor	Human review for high stakes outputs, with logs	Automated decisions applied without secondary review
Misinformation risk	Model card documents known failure modes	Failure modes emerge only after harm
Service delivery	Consistent outcomes across demographic slices, tested	Disparate impact hidden until external audit
Accountability	Legal, technical, and operational owners named	No clear owner when the system fails

AI Transparency Failures in Practice: Examples From the Field

Epic Systems Sepsis Alert Model

Epic deployed a proprietary sepsis alert model across hundreds of US hospitals starting in 2017, and the model operated as a closed system despite billing itself as clinical decision support. When researchers at the University of Michigan pushed for external validation, the resulting JAMA Internal Medicine study covering 27,697 hospitalized patients reported sensitivity of 33 percent, a substantial gap from vendor claims. The finding forced Epic to publish a technical brief and to loosen restrictions on customer led evaluation, and it prompted the ONC and FDA to accelerate transparency guidance. The limitation is that even the corrected model still lacks per patient reason codes, so clinicians run alerts largely on trust rather than mechanism. The 27,697 patient audit remains the reference example for how opacity can conceal poor performance in medicine for years.

Apple Card Credit Limit Investigation

Software entrepreneur David Heinemeier Hansson reported in November 2019 that his Apple Card credit limit was roughly 20 times higher than his wife’s despite similar financial profiles, and other applicants surfaced the same pattern. The New York Department of Financial Services opened an investigation that ran through March 2021 and found no violation of the New York fair lending statute, but the probe exposed that Goldman Sachs staff themselves could not fully explain individual credit limit decisions from the underwriting model. Goldman Sachs subsequently added manual review paths for anomalous decisions and published a broader appeal process. The clear limitation is that no post hoc audit produced a published root cause, so the actual mechanism behind the disparity remains publicly unresolved. The investigation stands as evidence that even a regulated bank can fail to interpret its own AI model.

Amazon Recruiting Model

Reuters reported in October 2018 that Amazon had built an internal machine learning tool to score resumes, and that the tool developed a preference against female candidates during a 2014 to 2017 build cycle. Because the model was trained on ten years of hiring data dominated by male applicants, it learned to downweight resumes containing the word women and to penalise graduates of two all women colleges. Amazon reportedly discontinued the model in 2017 and no candidates were affected by an approved deployment, according to the company. The important limitation is that Amazon acknowledged the model only after it was already scrapped, so external researchers have never audited the actual weights. The case became a widely cited example of how opacity plus biased training data can compound at scale, and it is referenced in the AI law ignored in NYC hiring follow up on continuing hiring AI regulation gaps.

In Depth Cases of AI Opacity Gone Wrong

Case Study: The Netherlands SyRI Welfare Fraud System

The Dutch government operated the SyRI risk indication system to detect welfare fraud from 2014 to 2020, using an opaque model that combined data from multiple government registers to flag households for investigation. Civil society organisations including the FNV union sued the state in 2018 because affected residents could not see how they had been scored or challenge the model. On February 5, 2020, the District Court of The Hague ruled that SyRI violated Article 8 of the European Convention on Human Rights, and it stopped further use of the system immediately, and the state did not appeal.

The ruling matters because opacity itself was a decisive factor in striking down the system, not just poor performance. AlgorithmWatch's account of the SyRI case documents that the court weighed the lack of publicly available risk model logic against the state's interest in fraud detection and ruled that the transparency deficit made the system disproportionate. The known limitation is that the underlying fraud detection needs of the state remain, and the successor system will still need explainability guardrails to avoid repeating the same defeat. SyRI became a reference precedent for European advocates arguing that black box public sector algorithms violate fundamental rights, and it directly influenced Dutch and EU regulatory work through 2024.

Case Study: COMPAS Recidivism Scoring in US Courts

Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS, is a proprietary risk assessment tool from Northpointe, later Equivant, that scores defendants on the probability of reoffending. US courts including Wisconsin, New York, and Broward County Florida have used the tool at bail, sentencing, and parole stages since the early 2010s, and its internal weights have been protected as a trade secret. Defendants argued that reliance on a scoring model they could not examine violated due process, and the Wisconsin Supreme Court in State v. Loomis permitted use in 2016 with a cautionary instruction rather than banning it.

ProPublica published a 2016 audit of COMPAS decisions in Broward County using 7,214 defendant scores against actual reoffense outcomes, and it reported that black defendants received false positive high risk scores at roughly twice the rate of white defendants. Northpointe disputed the framing and published a rebuttal focused on predictive parity, and the empirical debate continues because the vendor still refuses full disclosure of the model. The clear limitation is that no court order has forced release of the weights, so no independent audit has fully closed the argument. COMPAS remains a leading example of how a lack of transparency in AI collides with fundamental legal principles, and it is a reference case in future roles for AI ethics boards discussions of public sector AI governance.

Case Study: UK A Level Grading Algorithm 2020

In summer 2020, with UK exams cancelled by the COVID pandemic, Ofqual deployed an opaque algorithm to standardise teacher predicted A level grades. The algorithm downgraded roughly 39.1 percent of teacher assessments and produced markedly larger downgrades for students in state schools than for private school pupils, and neither students nor teachers could interpret how their specific grade had been reached. Public outcry over the pattern led to a full policy reversal within four days, and the government reverted to teacher assessed grades on August 17, 2020, after judicial review threats mounted.

The failure was not a coding bug but a design choice to combine school level historical performance with individual predictions, which encoded existing structural inequality into the outputs. The Frontiers review of algorithmic discrimination analyzes the case as a textbook example of how opacity plus disparate impact plus consequential decisions produce political backlash within days. The clear limitation is that the underlying problem, how to fairly assess students without exams, was never solved by the algorithm, and future contingency planning still lacks a transparent scoring approach. The episode accelerated UK regulatory attention on public sector algorithmic decision making and helped shape the ICO's later guidance on automated decision making transparency.

Common Questions on the Lack of Transparency in AI