Governing the Ungovernable? High-Risk AI Systems and the Legal Architecture of the EU AI Act

by Seyed Milad Mahmood Kashani

Researcher, University of Naples Federico II

A Regulation Born from Urgency

The Artificial Intelligence Act — Regulation (EU) 2024/1689, which entered into force on August 1, 2024, represents the most comprehensive attempt in the history of digital governance to regulate the creation and use of artificial intelligence (AI). The Act is not merely a technical standard or a procedural framework for implementation, but rather an expression of how artificial intelligence, in its most significant developments, should be held accountable, made transparent, and subjected to the rule of law. However, the AI Act did not emerge in a vacuum. It was shaped by years of documented failures — algorithmic systems denying loan applications based on postal codes, recidivism prediction tools producing racially biased outcomes in criminal courts, and medical diagnostic software whose error rates remained unknown to the clinicians relying on them. The question the regulation seeks to answer is deceptively simple: when an automated system makes a decision affecting a human being’s rights, freedoms, or safety, who is responsible — and for what? The EU’s answer is structured around risk. Not all AI applications are equally dangerous, and the law should not treat them as though they were. What the AI Act introduces, and what this article seeks to explain, is a layered framework in which the most significant category — high-risk AI — carries a distinct and demanding set of legal obligations.

What Makes an AI System “High-Risk”?

The concept of high-risk AI is the regulatory core of the Act. Under Articles 6 and 7, read together with Annexes I and III, a system is classified as high-risk if it falls into one of two categories: AI integrated into products already regulated under Union harmonized safety legislation — such as medical devices, machinery, or civil aviation equipment — or AI deployed in a specific set of sensitive areas explicitly listed in the Regulation. These sensitive areas cover an unusually broad range of sectors. They include AI used in biometric identification, critical infrastructure management, education and vocational training, employment and worker management, access to essential public and private services, law enforcement, migration and border control, and the administration of justice. The common thread is not technical complexity — it is the potential for systemic harm to individuals whose life conditions may be profoundly affected by an automated output they have no meaningful means to challenge. This classification is not merely a label. It activates a specific legal regime that applies before a system reaches the market, during its operation, and after harm occurs. Understanding this regime requires close attention to what the law actually demands from those who build and deploy these systems.

The Architecture of Obligations: Providers and Deployers

The AI Act distributes legal responsibility between two principal actors: providers, who develop or place AI systems on the market, and deployers, who use those systems in a professional context. This distinction is highly significant in practice. A hospital using a commercially developed AI diagnostic tool is a deployer, not a provider. A technology company licensing a recruitment algorithm to dozens of employers is a provider. Each carries different obligations, although the line between the two may blur where a deployer significantly modifies an acquired system. Providers of high-risk AI systems are subject to the most stringent requirements. They must implement a quality management system, conduct a conformity assessment, register the system in the EU database for high-risk AI, and affix the CE marking signaling regulatory compliance. Before all of this, however, they must address what the Regulation places at the normative center of the high-risk regime: data governance, technical documentation, transparency, human oversight, accuracy, robustness, and — above all — safety.

Safety as a Legal Obligation, Not a Technical Choice

Article 15 of the AI Act introduces cybersecurity and resilience obligations for high-risk AI systems that deserve particular attention. The provision requires such systems to be designed and developed in a manner that achieves an appropriate level of accuracy, robustness, and cybersecurity — and that these characteristics be maintained throughout the system’s lifecycle. This means that safety cannot be added afterward as an afterthought. It must be embedded into the system from the outset. The Regulation specifically refers to resilience against attempts by third parties to exploit system vulnerabilities, including adversarial attacks — inputs deliberately crafted to manipulate a model’s output. It also addresses risks of error, inconsistency, or unexpected behavior arising from technical limitations or from data that inadequately represents the environments in which the system operates. What is legally innovative here is not the requirement of safety itself — product safety law has always implied a duty to avoid dangerous outputs — but the specificity with which the Regulation addresses threats unique to AI. Traditional product liability models were designed for physical objects that fail predictably. A high-risk AI system may fail invisibly: it may produce statistically accurate outputs overall while systematically discriminating against identifiable groups; it may perform well under known conditions and catastrophically fail under new ones; it may be subtly manipulated without any externally detectable change in behavior. The AI Act’s safety obligations recognize that governing such systems requires a fundamentally different legal vocabulary.

Human Oversight: A Non-Negotiable Condition

Article 14 — concerning human oversight — is perhaps the most philosophically significant provision within the high-risk regime. It requires that high-risk AI systems be designed in such a way that individuals can effectively supervise their operation throughout the period of use. Meaningful oversight (not simply the automatic endorsement of algorithmic recommendations) implies a genuine ability to understand, monitor, or intervene, as well as a real and ongoing capacity to override system outputs. The Regulation identifies several specific capabilities that form part of human oversight, including understanding the system’s capabilities and limitations, identifying and responding to anomalies or unexpected behavior, and deciding on a case-by-case basis whether to use or disregard the system’s output. Deployers will need to designate competent individuals to carry out such oversight. This requirement will have implications not only for how oversight is exercised, but also for training, certification, and institutional accountability in sectors such as healthcare, law enforcement, and public administration. This provision implicitly rejects the logic of automation bias — the well-documented tendency of human operators to defer uncritically to algorithmic recommendations, even when demonstrably flawed. The law does not merely permit human oversight; it structurally imposes the conditions under which such oversight remains possible. This is a sophisticated legislative ambition and, in the history of technology regulation, a relatively unusual one.

The Intersection with Civil Liability

The AI Act does not operate in isolation. It intersects — and was designed to intersect — with the EU’s emerging framework on AI liability, particularly the proposed AI Liability Directive and the revised Product Liability Directive. Together, these instruments alter the evidentiary burden required to demonstrate that an AI system caused harm. Under traditional liability law, a person harmed by a product or service is generally required to prove that the producer acted negligently. In the AI context, this burden is extraordinarily difficult to satisfy. The causal chain between algorithmic design, training data, operational environment, and a specific harmful output is opaque to most injured parties — and often even to regulators. The proposed AI Liability Directive addresses this evidentiary problem through a rebuttable presumption of causality: where an injured party can demonstrate non-compliance with relevant obligations under the AI Act — including those concerning safety, accuracy, and human oversight — causation between that non-compliance and the resulting harm is presumed. The provider or deployer must then rebut the presumption. This constitutes a significant doctrinal innovation, linking the ex ante compliance framework of the AI Act to the ex post liability mechanisms of civil law.

Implementation Challenges and Future Perspectives

The implementation timeline of the AI Act is gradual. Most obligations concerning high-risk AI systems will become fully applicable from August 2, 2026, although some provisions — particularly those concerning general-purpose AI models — entered into force earlier. It is particularly relevant to note that there is an extended transitional period until August 2, 2027, during which high-risk AI systems already integrated into products regulated under European harmonized safety legislation will not be required to achieve immediate full compliance. This is significant for manufacturers of medical devices, machinery, and other regulated sectors, where AI is integrated into products as a component rather than a standalone entity. The European AI Office (established by the European Commission) is responsible for enforcing the Regulation at the EU level and shares comparable competences with national market surveillance authorities in Member States. The practical implementation challenges are considerable. Conformity assessments for high-risk systems require technical expertise that many national authorities have not yet developed. The standards with which AI systems must comply — currently being developed by European standardization bodies — are not yet complete across all relevant sectors. Furthermore, the boundary between what constitutes a high-risk system and what does not remains contested in several areas, generating uncertainty for both developers and deployers. The deepest challenge, however, is conceptual rather than procedural. The AI Act is based on a risk-oriented approach that assumes AI harms are measurable, classifiable, and containable. Yet algorithmic harms are often systemic, diffuse, and slow to emerge — operating at the level of populations rather than individuals, and over timelines that may not be visible at the moment of deployment. The Regulation represents a necessary step. Its sufficiency will depend as much on interpretation and enforcement by institutions, courts, and civil society as on the legislative text itself.

Conclusion: Law as a Design Constraint

The EU AI Act introduces a legal framework that treats high-risk AI not as a product to regulate after the fact, but as a socio-technical system whose design must be constrained from the beginning. The requirements imposed — data quality, transparency, human oversight, accuracy, and safety — are not mere bureaucratic formalities, but structural conditions intended to ensure that automated decision-making processes operate legitimately in contexts where fundamental rights and safety are at stake. Organizations and legal entities operating in Europe must clearly understand that there is no option of non-compliance with the established standards. In the event of violations, both administrative sanctions and civil liability are foreseen. The sanctions regime is structured: breaches of obligations relating to high-risk AI systems may result in penalties of up to €15 million or 3% of global annual turnover, whichever is higher; violations involving prohibited practices — located at the top of the risk hierarchy — may reach €35 million or 7% of global turnover. More profoundly, the Regulation calls for a shift in how developers, deployers, and regulators conceptualize AI: not as a neutral tool that merely amplifies human capabilities, but as a system of technical, commercial, and ethical choices for which someone must always remain accountable. That accountability, ultimately, is what the rule of law requires. The ambition of the AI Act is to ensure that regulated individuals are not once again left to bear alone the costs of technologies they did not design, do not understand, and cannot meaningfully challenge.