by Seyed Milad Mahmood Kashani, Ph.D.
Researcher (Assignor di Ricerca, IUS/01 – Private Law)
University of Naples Federico II
The Transparency Illusion
There is a seductive simplicity to the idea that transparency can solve the problem of algorithmic accountability. If we can see how an AI system reaches its decisions, the argument goes, we can assess whether those decisions are fair, accurate, and legally defensible. This intuition has shaped not only public debate but legislative design. The EU AI Act, the most comprehensive attempt yet to regulate artificial intelligence, places transparency and explainability at the center of its obligations for high-risk AI systems. Yet transparency, as currently conceived in the regulation, is not enough. It may not even be the right goal.
This article argues that the explainability requirements enshrined in the AI Act, while necessary, are structurally insufficient to address the deepest accountability deficits of algorithmic decision-making. It does so not to dismiss the regulation’s ambitions, which are real and important, but to identify what the law leaves undone and what additional conceptual and institutional tools are needed to fill the gap.
What the AI Act Actually Requires
Transparency under the AI Act operates at several levels. Article 13 requires that high-risk AI systems be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret the system’s output and use it appropriately. This obligation is directed primarily at deployers — the hospitals, courts, banks, and public authorities that use AI in consequential decisions — rather than at the individuals affected by those decisions.
Article 50 extends a narrower transparency duty toward natural persons: when individuals interact with an AI system or are subject to emotion recognition or biometric categorisation, they must be informed of this fact. Where a system generates content or makes recommendations, the output must be labelled as AI-generated. These are meaningful requirements, but they operate at the level of disclosure rather than comprehension. Being told that a decision was made with the assistance of an AI system is not the same as being able to understand or challenge that decision.
The regulation does not define “explainability” as a distinct legal concept. It uses the term “transparency” to encompass both the duty to document how a system works and the duty to communicate that information to those who use or are affected by it. This conflation conceals a critical distinction: a system can be perfectly transparent in the sense that its architecture, training data, and decision logic are fully documented, while remaining entirely opaque to the person whose loan application it denied or whose asylum claim it assessed negatively.
The Limits of Interpretability as a Technical Fix
The computer science literature on explainable AI — commonly abbreviated as XAI — has expanded rapidly in response to regulatory and societal demand. Techniques such as LIME (Local Interpretable Model-agnostic Explanations), SHAP (SHapley Additive exPlanations), and attention visualisation in neural networks offer ways to generate post-hoc explanations of individual predictions. A credit scoring model can be queried to identify which features — income, payment history, postal code — contributed most to a rejection. A medical imaging classifier can be prompted to highlight the pixels that most influenced its diagnosis.
These tools are valuable, but they carry a set of limitations that are rarely acknowledged in regulatory discourse. First, most XAI techniques produce approximations rather than faithful representations of how a model actually works. A post-hoc explanation of a deep neural network’s output is not a window into its internal reasoning; it is a simplified reconstruction that may diverge significantly from the actual computational process. Explanations can be accurate or they can be comprehensible — they cannot always be both.
Second, explanations are sensitive to context in ways that undermine their legal utility. The same model can generate different explanations for similar inputs depending on which XAI method is used, which baseline is selected, or how the query is framed. This instability means that an explanation presented to a claimant challenging an automated decision may bear little relationship to what the model actually computed at the moment of that decision. Courts and administrative tribunals are ill-equipped to adjudicate these disputes without access to independent technical expertise that most jurisdictions do not yet have.
Third, and perhaps most fundamentally, explanations answer a question that is often not the one that matters most. Knowing which features drive a particular prediction tells us something about how the model behaves — but it tells us nothing about whether that behaviour is legitimate. A system that correctly identifies that an applicant lives in a neighbourhood with high historical default rates is transparent in the technical sense; but if that neighbourhood is a proxy for race or ethnicity, the explanation illuminates a discriminatory mechanism rather than resolving it.
The Accountability Gap the AI Act Does Not Close
The AI Act’s transparency provisions are designed primarily to support deployers in making informed use of AI outputs. They are not, on their own, designed to give individuals substantive means to contest decisions that affect their rights. This gap is not accidental — it reflects a deliberate choice to build the regulation around a professional compliance model rather than an individual rights model. The deployer receives documentation; the individual receives a notification.
This asymmetry becomes legally significant when considered alongside GDPR Article 22, which grants individuals the right not to be subject to solely automated decisions that produce significant effects, and — where such decisions are permitted — the right to obtain human intervention, express their point of view, and contest the decision. Article 22 creates a meaningful explanation of the logic involved, but neither the GDPR nor the AI Act specifies what makes an explanation meaningful, who is responsible for providing it, or how a recipient without technical expertise can exercise that right in practice.
The proposed AI Liability Directive introduces a rebuttable presumption of causality where non-compliance with AI Act obligations can be demonstrated, which partially addresses the evidentiary barrier that plaintiffs face in litigation. But it does not resolve the prior question of whether individuals have the information necessary to know that a violation occurred, or the institutional support needed to invoke the presumption before a court.
What ‘Meaningful’ Explanation Would Actually Require
Legal scholars working on algorithmic accountability — most prominently in the context of the counterfactual explanation literature — have argued that what individuals need from AI explanations is not a description of the model’s internal logic but actionable information about what would have to change for a different outcome to result. A counterfactual explanation of a rejected loan application might state: if your annual income had been €5,000 higher and your credit utilisation below 30%, this application would have been approved. This kind of explanation does not require exposing proprietary model architecture; it focuses on the relationship between inputs and outputs in a way that is both comprehensible and contestable.
Counterfactual explanations are not without their own limitations — they may identify paths to approval that are practically impossible for the individual to achieve, or they may conceal systemic patterns that affect entire categories of applicants rather than individual cases. But they represent a more rights-oriented conception of explainability than the documentation and disclosure model the AI Act currently mandates. The question for European legislators and courts is whether the right to explanation under GDPR Article 22 can be interpreted to require this kind of actionable, individual-oriented information, and whether the AI Act’s transparency provisions can be read in conformity with that interpretation.
The Institutional Dimension: Who Explains, to Whom, and Under What Conditions
Even a legally robust definition of meaningful explanation cannot function without institutional infrastructure to deliver it. Three gaps stand out. First, the AI Act places the primary obligation to produce explanations on providers, but it is deployers who interact with affected individuals and who are best placed to translate technical documentation into comprehensible communication. The regulation creates obligations for both actors but does not adequately specify how the explanatory function should be allocated between them in practice.
Second, the regulation does not address the capacity asymmetry between sophisticated AI providers and the public authorities, small businesses, and individual users who must interpret and act on the explanations they receive. A hospital employing a thousand clinicians can invest in AI literacy training; a sole trader using an automated credit assessment tool to evaluate clients cannot. Explanatory obligations that are calibrated to the needs and capacities of large deployers will systematically fail smaller actors and the individuals they serve.
Third, independent oversight bodies — data protection authorities, market surveillance authorities, the European AI Office — require both technical expertise and the institutional mandate to audit explanations for accuracy and completeness. The AI Act grants these bodies significant enforcement powers, but the development of the interpretive standards and auditing methodologies they need to exercise those powers effectively is still at an early stage. Regulation without interpretation is aspiration.
Toward a More Substantive Accountability Framework
None of this is an argument against transparency requirements. Mandatory documentation, technical disclosure, and notification obligations are necessary conditions for accountability, and their absence would be a serious regulatory failure. The point is that they are not sufficient conditions. An accountability framework adequate to the risks of high-risk AI would need to complement transparency obligations with at least three additional elements.
It would need a rights-oriented explanation standard — one that specifies not merely that information must be provided, but that it must be provided in a form that enables the recipient to understand the decision and, where appropriate, to challenge it. It would need procedural guarantees for AI-assisted decision-making in public and quasi-public contexts — not merely the right to human review in principle, but the right to a human reviewer who has genuinely engaged with the individual case rather than rubber-stamped an algorithmic recommendation. And it would need ongoing audit and redress mechanisms that allow patterns of discriminatory or erroneous output to be identified at the population level, not only after individual complaints have been lodged.
The AI Act creates architecture for such a framework. What remains to be built — through secondary legislation, technical standards, judicial interpretation, and institutional practice — is the substance that makes that architecture meaningful.
Conclusion: Transparency as a Floor, Not a Ceiling
The EU AI Act’s transparency and explainability requirements represent a serious attempt to bring algorithmic decision-making within the rule of law. They establish disclosure obligations, mandate technical documentation, and create a compliance infrastructure that did not previously exist at European scale. These are genuine achievements.
But transparency, as currently designed, functions as a floor rather than a ceiling. It tells deployers what a system does without always enabling individuals to understand why it did it to them. It creates rights to information without always creating the conditions under which that information can be acted upon. And it places the primary burden of interpretation on actors — public authorities, regulated businesses, individual claimants — who may lack the technical literacy to discharge that burden effectively.
The deeper lesson of the explainability of literature, both technical and legal — is that accountability for algorithmic systems cannot be achieved through information alone. It requires institutional design: bodies with the mandate and the expertise to audit, interpret, and enforce; procedures that make contestation genuinely accessible; and standards of explanation that serve the interests of those affected by AI decisions, not merely those who deploy them. The AI Act points in the right direction. The work of giving that direction content has only begun.
About the Author
Seyed Milad Mahmood Kashani holds a Ph.D. in Private Law with Doctor Europaeus distinction (Excellent cum laude) from the University of Naples Federico II, where he currently serves as a Researcher in Private Law (IUS/01). His research focuses on EU digital regulation, AI governance, GDPR harmonisation, and smart contract law across Italian, German, and European legal frameworks. He has presented his work at international conferences including VUB Brussels and the Universidad de Cádiz.
