Article 5(1)(a) and (b) AI Act – harmful manipulation, deception and exploitation

The underlying rationale of these prohibitions is to protect individual autonomy and well-being from manipulative, deceptive, and exploitative AI practices that can subvert and impair an individual’s autonomy, decision-making, and free choices. The prohibitions aim to protect the right to human dignity (Article 1 of the Charter), which also constitutes the basis of all fundamental rights and includes individual autonomy as an essential aspect. In particular, the prohibitions aim to prevent manipulation and exploitation through AI systems that reduce individuals to mere tools for achieving certain ends and to safeguard those that are most vulnerable and susceptible to harmful manipulation and exploitation. The prohibitions against significantly harmful manipulative, deceptive and exploitative AI practices fully align with the broader objectives of the AI Act to promote trustworthy and human-centric AI systems that are safe, transparent, fair and serve humanity and align with human agency and EU values.

Several cumulative conditions must be fulfilled for the prohibition in Article 5(1)(a) AI Act to apply:

(i) The practice must constitute the ‘placing on the market’, the ‘putting into service’, or the ‘use’ of an AI system.

(ii) The AI system must deploy subliminal (beyond a person's consciousness), purposefully manipulative or deceptive techniques.

(iii)The techniques deployed by the AI system should have the objective or the effect of materially distorting the behaviour of a person or a group of persons. The distortion must appreciably impair their ability to make an informed decision, resulting in a decision that the person or the group of persons would not have otherwise made.

(iv) The distorted behaviour must cause or be reasonably likely to cause significant harm to that person, another person, or a group of persons.

For the prohibition to apply, all four conditions must be simultaneously fulfilled and there must be a plausible causal link between the techniques deployed, the material distortion of the behaviour of the person, and the significant harm that has resulted or is reasonably likely to result from that behaviour.

The first condition, i.e. the ‘placing on the market’, the ‘putting into service’ or the ‘use’ of an AI system, has already been analysed. The prohibition, therefore, applies to both providers and deployers of AI systems, each within their respective responsibilities, not to place on the market, put into service or use such systems. The next sections focus on the other three conditions.

3... Subliminal, purposefully manipulative or deceptive techniques

Article 5(1)(a) AI Act prohibits three alternative types of manipulative techniques: a) subliminal techniques beyond a person’s consciousness: b) purposefully manipulative techniques; and c) deceptive techniques. To fall within scope of Article 5(1)(a) AI Act, an AI system must deploy one or more of these techniques.

a) Subliminal techniques

While the AI Act does not define ‘subliminal techniques’, Article 5(1)(a) AI Act specifies that subliminal techniques operate beyond (below or above) the threshold of conscious awareness. Because subliminal techniques and the way they operate are inherently covert, such techniques bypass a person’s rational defences against manipulation and are capable of influencing decisions without the conscious awareness of the person, raising significant ethical concerns and impairing individual autonomy, agency and free choice.

The subliminal techniques must be capable of influencing behaviour in ways in which the person remains unaware of such influence, how it works, or its effects on the person’s decision-making or value- and opinion-formation. In particular, subliminal techniques may use stimuli delivered through audio, visual, or tactile media that are too brief or subtle to be noticed and that have been traditionally known and prohibited in other sectors, such as media advertising. These stimuli, while not consciously perceived, may still be processed by the brain and influence behaviour. Examples of subliminal techniques (not necessarily prohibited unless all other conditions listed in Article 5(1)(a) AI Act are fulfilled) include: - Visual Subliminal Messages: an AI system may show or embed images or text flashed briefly during video playback which are technically visible, but flashed too quickly for the conscious mind to register, while still being capable of influencing attitudes or behaviours.

- Auditory Subliminal Messages: an AI system may deploy sounds or verbal messages at low volumes or masked by other sounds, influencing the listener without conscious awareness. These sounds are still technically within the range of hearing, but are not consciously noticed by the listener due to their subtlety or masking by other audio. - Tactile Subliminal Stimuli: an AI system may stimulate subtle physical sensations that are perceived unconsciously, capable of influencing emotional states or behaviour. - Subvisual and Subaudible Cueing: an AI system may deploy stimuli that are not just subtle or masked, but are presented in a way that makes them entirely undetectable by the human senses under normal conditions, for example flashing visual stimuli (e.g., flashing images) too quickly for the human eye to detect consciously or playing sounds at volumes imperceptible to the human ear.

- Embedded Images: an AI system may hide images within other visual content which are not consciously perceived, but may still be processed by the brain and influence behaviour.

- Misdirection: an AI system may draw attention to specific stimuli or content to prevent noticing other content, often by exploiting cognitive biases and vulnerabilities in attention.

- Temporal manipulation: an AI system may alter the perception of time in user interactions, thus influencing their behaviour and causing impatience and dependence.

The rapid development of AI and related technologies, such as big data analytics, neuro technologies, brain-computer interfaces and virtual reality, heightens the risk of sophisticated subliminal manipulation and its capability to effectively influence human behaviour in a subconscious manner. AI can also extend to emerging machine-brain interfaces and advanced techniques like dream-hacking and brain spyware. For example, a game can leverage AI-enabled neuro technologies and machine-brain interfaces that permit users to control (parts of) a game with headgear that detects brain activity. AI may be used to train the user’s brain surreptitiously and without their awareness to reveal or infer from the neural data information that can be very intrusive and sensitive (e.g., personal bank information, intimate information, etc.) in a manner that can cause them significant harm. The prohibition in Article 5(1)(a) AI Act targets only cases of such significantly harmful subliminal manipulation and not machine-brain interface applications in general when designed in a safe and secure manner and respectful of privacy and individual autonomy.

b) Purposefully manipulative techniques

‘Purposefully manipulative techniques’ are not defined in the AI Act, but they should be understood as techniques that are designed or objectively aim to influence, alter, or control an individual’s behaviour in a manner that undermines their individual autonomy and free choices. Manipulative techniques are typically designed to exploit cognitive biases, psychological vulnerabilities, or situational factors that make individuals more susceptible to influence. Because of their adaptability, AI systems are also able to respond well to a person’s individual circumstances or vulnerabilities and increase the effectiveness and impact of manipulation at scale. While the manipulative capability is an important element to determine the nature of the technique, it is not necessary that the provider or deployer or the system itself deploying the manipulative techniques also intends to cause harm.

While not all manipulative techniques operate beyond the threshold of conscious awareness, many do and there may be an overlap with subliminal techniques, since such techniques also ultimately have manipulative effects. Recital 29 AI Act clarifies that the prohibition in Article 5(1)(a) also covers techniques where individuals, even if they are aware of the influence attempt, may not be able to control or resist its manipulative effect. As a result, individuals are influenced or pushed into behaviour and decisions they would normally not have made if they were not subjected to the manipulative techniques to a point that undermines their individual autonomy or free choice. An example of purposefully manipulative techniques is sensory manipulation where an AI system deploys background audio or images that lead to mood alterations, for example increasing anxiety and mental distress that influence users’ behaviour to the point of creating significant harm.

Another example is personalised manipulation where an AI system that creates and tailors highly persuasive messages based on an individual’s personal data or exploits other individual vulnerabilities influences their behaviour or choices to a point of creating significant harm.

The prohibition against purposefully manipulative techniques also covers AI systems that manipulate individuals without any human intending them to do so. Article 5(1)(a) AI Act prohibits AI systems that deploy certain techniques or exhibit a specific manipulative behaviour. Therefore, it could also be the AI system that deploys such manipulative techniques, rather than the provider or the deployer that has designed or used the system in this way.

For example, regardless of whether the provider intends it, an AI system may learn manipulative techniques because the data on which it is trained contain many instances of manipulative techniques, or because reinforcement learning from human feedback can be ‘gamed’ through manipulative techniques. By contrast, if the manipulative behaviour of the system is merely incidental, the system should not be considered deploying purposefully manipulative techniques as long as the provider has taken appropriate preventive and mitigating measures in case significant harms are reasonably likely to occur (see section 3...c) below). c) Deceptive techniques

The AI Act does not define ‘deceptive techniques’. Recital 29 AI Act clarifies that these are techniques that subvert or impair a person’s autonomy, decision-making or free choice in ways that the person is not consciously aware or, where it is aware, can still be deceived or is not able to control or resist them. ‘Deceptive techniques’ deployed by AI systems should be understood to involve presenting false or misleading information with the objective or the effect of deceiving individuals and influencing their behaviour in a manner that undermines their autonomy, decision-making and free choices.

In this context, the interplay between the prohibition in Article 5(1)(a) AI Act and the deployer’s obligations in Article 50(4) AI Act to label ‘deep fakes’ and certain AI- generated text publications on matters of public interest, as well as the provider’s obligation to ensure AI systems interacting with people are designed in a way to inform people that they are interacting with AI and not a human, should be clarified. Such visible disclosure constitutes a mitigating measure that should also be enabled through design features embedded in the AI system provided by the provider, including technical measures enabling the detection of AI-generated and manipulated content. The visible labelling of ‘deep fakes’ and chatbots reduces the risk of deception that is likely to arise once the AI-generated content is disseminated to the public and reduces the risk of harmful distorting effects on the individual’s opinion- and belief-formation and behaviour.

By contrast, the prohibition in Article 5(1)(a) AI Act has a much more limited scope. It may, for example, cover cases where a chatbot or deceptive AI-generated content presents false or misleading information in ways that aim to or have the effect of deceiving individuals and distorting their behaviour that would not have happened if they were not exposed to the interaction with the AI system or the deceptive AI generated content, in particular if this has not been visibly disclosed.

As with purposefully manipulative techniques, the prohibition of deceptive techniques may also cover AI systems that deceive individuals without any human intending them to do so (see section 3...b)above). For example, regardless of whether their providers intend such an outcome, AI systems may learn deceptive techniques simply because this increases their performance for the task for which they were developed, for example by reinforced learning. An example of deceptive techniques that may be deployed by AI is an AI chatbot that impersonates a friend of a person or a relative with synthetic voice and tries to pretend it is the person causing scams and significant harms.

Another example is an AI system that learns to identify when it is under evaluation and temporarily halts any undesired behaviour, only to resume such behaviour once the evaluation period is over. Such deceptive behaviour is particularly dangerous, since it defies any external human oversight over the system and may be prohibited if it is reasonably likely to cause significant harms.

By contrast, a generative AI system that incidentally presents false or misleading information and hallucinates may not be considered to deploy deceptive techniques within the meaning of Article 5(1)(a) AI Act, taking into account the limitations and the state of the art of generative AI. In particular, this may be the case where the provider of the system has properly informed users about the system’s limitations and integrated appropriate safeguards into the system to minimise such outcomes and provided that the system is not intended for, nor deployed in, sensitive contexts (e.g., health, education, elections) where serious harmful consequences are likely to occur (see also considerations in section 3...c) below).

d) Combination of techniques

Article 5(1)(a) AI Act applies to subliminal, purposefully manipulative, or deceptive techniques, or to combinations of such techniques that can have a compound impact. As stated above, purposefully manipulative techniques may be also subliminal in nature, if they operate beyond the threshold of conscious awareness.

Furthermore, when purposefully manipulative and deceptive techniques are applied in combination, this may significantly influence the behaviour of individuals, leading them to make decisions based on unconscious manipulations and false beliefs. This combination may create a feedback loop where individuals are less likely to question or critically evaluate the information received, since the manipulative elements have already primed their cognitive biases and emotional responses. 3... With the objective or the effect of materially distorting the behaviour of a person or a group of persons

A third condition for the prohibition in Article 5(1)(a) AI Act to apply is that the deployed subliminal, purposefully manipulative, or deceptive technique must have ‘the objective, or the effect of materially distorting the behaviour of a person or a group of persons’. This implies a substantial impact on the behaviour where a person’s autonomy and free choices are undermined, rather than a minor influence. However, intent is not a necessary requirement, since Article 5(1)(a) AI Act also covers practices that may only have the ‘effect’ of causing material distortion. There should be a plausible/reasonably likely causal link between the potential material distortion of the behaviour and the subliminal, purposefully manipulative or deceptive technique deployed by the AI system.

a) The concept of ‘material distortion of the behaviour’

The concept of ‘material distortion of the behaviour’ of a person or a group of persons is central to Article 5(1)(a) AI Act. It involves the deployment of subliminal, purposefully manipulative or deceptive techniques that are capable of influencing people’s behaviour in a manner that appreciably impairs their ability to make an informed decision, thereby causing them to behave in a way or to take a decision that they would otherwise not have taken.

‘Appreciable impairment’ refers to a substantially reduced ability to make informed and autonomous decisions, thereby causing individuals to behave in a way or to take a decision that they would otherwise not have taken. It goes beyond minor or negligible impacts and involves a significant distortion or hindrance in decision-making and free choice, including in relation to opinion- and belief-formation. This suggests that ‘material distortion’ involves a degree of coercion, manipulation, or deception that goes beyond lawful persuasion, which falls outside the scope of the prohibition (see section 3... below).

An ‘informed decision’ requires an understanding and knowledge of the relevant information, including the available options, the risks and benefits of each choice, the possible effects of the AI system on their behaviour, and, as appropriate, other contextual information that is important for the decision-making or the behaviour of the person.

For the interpretation of the concept of ‘material distortion of behaviour’, Union consumer protection law, in particular, Directive 2005/29/EC (Unfair Commercial Practices Directive or ‘UCPD’), may constitute a valid source of inspiration. The UCPD prohibits various unfair, misleading, and aggressive commercial practices (Articles 5 to 9 UCPD) capable of causing consumers to make transactional decisions that they would otherwise not have made. According to the CJEU and the Commission guidance on the UCPD, there is no need to prove that a consumer’s economic behaviour has been distorted, it suffices to establish that a commercial practice is ‘likely’ (i.e. capable) of impacting an average consumer’s transactional decision. The CJEU has also underscored that even accurate information may be misleading if presented in a way that distorts the consumer’s decision-making process. National enforcement authorities are tasked to investigate the specific facts and circumstances of each case (in concreto) and to evaluate the potential impact of the practice on the average consumer’s decision-making process (in abstract). For that purpose, they must take the point of view of the ‘average’ consumer, which is the benchmark developed by the CJEU, now integrated in the UCPD.

In the context of the prohibition in Article 5(1)(a) AI Act, market surveillance authorities must also investigate each case’s specific facts and circumstances, assessing whether the subliminal, purposefully manipulative or deceptive technique deployed by the AI system is likely to appreciably impair the decision-making, individual autonomy and free choice of an ‘average’ individual within a targeted group when the system affects a group of persons in a manner that is reasonably likely to cause significant harm. Such an interpretation seems justified given that the AI Act intends to complement the UCPD and must be applied in a consistent manner. At the same time, given that Article 5(1)(a) AI Act also refers to the possibility to distort the ‘behaviour of a natural person’ and, if the perspective of the ‘average’ individual proves difficult or ineffective to assess in certain contexts (for example due to very tailored or ‘personalised’ manipulation or harmful effects on specific vulnerable groups), specific cases may be examined also from the perspective of specific individuals by assessing to what extent an AI system deploying subliminal, purposefully manipulate or deceptive techniques is capable of undermining their individual autonomy in concrete cases and significant harm has occurred or is likely to occur.

b) Scenario 1: Prohibited AI systems ‘with the objective to’ materially distort behaviour

Article 5(1)(a) AI Act applies to AI systems deploying the above-mentioned techniques and having as a first scenario ‘the objective to materially distort the behaviour of a person or a group of persons’. Such an objective may be pursued by the provider or the deployer of the AI system, or by the system itself within the implicit objectives it may pursue. This objective should be distinguished from the ‘intended purpose’ of the AI system (Article 3(12) AI Act). Even if intended by the provider, the manipulative objective is in most cases not the purpose of the use for which the system is offered and it is often neither transparent, nor specified as such in the information supplied by the provider (e.g., in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation).

For example, a chatbot that may be used in different contexts is designed to use subliminal messaging techniques, such as flashing brief visual cues and embedding inaudible auditory signals or to exploit emotional dependency or specific vulnerabilities of users in advertisements. These techniques are deployed ‘with the objective to’ materially distort users’ behaviour, since they are objectively a design feature that aims to influence consumers’ purchasing decisions without their conscious awareness, to push people to take significantly harmful financial decisions. Deploying AI systems to impersonate other persons could also be considered as an AI system deployed ‘with the objective’ to deceive and materially distort the behaviour of persons if the person is effectively deceived, thus substantially affecting their ability to make informed decisions about the identity of the person.

If in both examples the other conditions in Article 5(1)(a) AI Act are fulfilled, in particular regarding the significant harm, those systems are likely to fall within the scope of the prohibition, but this will require a case-by-case assessment. c) Scenario 2: Prohibited AI systems ‘with the effect of’ materially distorting behaviour

The intent of the provider or deployer to materially distort the behaviour of a person or group of persons is a sufficient, but not a necessary, condition for the prohibition in Article 5(1)(a) AI Act to apply. That prohibition also applies where no such intent is present, but the effect of the technique(s) deployed by an AI system is capable of materially distorting the behaviour of a person or a group of persons to a point that undermines their individual autonomy and free choices.

A plausible/reasonably likely causal link between the subliminal, purposefully manipulative or deceptive technique deployed by the AI system and its effects on the behaviour is, however, always necessary for the prohibition to apply. In consistency with consumer protection law, these effects do not need to have fully materialised, but there must be sufficient indication that they are likely or capable to materialise and undermine individual autonomy based on an objective assessment of all circumstances of the case and existing scientific knowledge and methods, as well as available information about the impact of the system on the individuals’ behaviour in real life. In this context, the fact that a system is capable of triggering behaviours that appreciably impair individuals’ ability to make an informed decision and undermine their free choices suffice to fulfil that condition and is not dependent on considerations relating to the ‘timing’ for the harm to materialize (e.g., in case of addiction-like behaviours) as long as it is reasonably likely to occur.

For example, an AI-powered well-being chatbot is intended by the provider to support and steer users in maintaining a healthy lifestyle and provide tailored advice for psychological and physical exercises. However, if the chatbot exploits individual vulnerabilities to adopt unhealthy habits or to engage in dangerous activities (e.g., engage in excessive sports without rest or drinking water) where it can reasonably be expected that certain users will follow that advice, which they would otherwise not have done, and suffer significant harm (e.g., a heart attack, or other serious health problem), that AI system would fall under the prohibition in Article 5(1)(a) AI Act, even if the provider might not have intended this behaviour and harmful consequences for the persons.

The mere fact that the chatbot is capable of appreciably impairing individual autonomy and materially distorting the behaviour of certain users in a significantly harmful way and that the provider has not taken appropriate preventive and mitigating measures to avoid those significantly harmful effects suffices for the prohibition to apply (see more for relevant considerations of the reasonably likeliness of the harm in section 3... and out of scope section 3..).

3... (Reasonably likely to) cause significant harm

Finally, for the prohibition in Article 5(1)(a) AI Act to apply, the distortion of the behaviour of a person or group of persons must cause or be reasonably likely to cause that person, another person, or group of persons significant harm. In this context, important concepts that require clarification are the types of harms covered by the prohibition, the threshold of significance of the harm, and its reasonable likelihood and causal link between the harm and the manipulative or deceptive technique and the person’s behaviour.

a) Types of harms

The AI Act addresses various types of harmful effects associated with manipulative and deceptive AI systems, each with distinct implications for individual persons and groups of persons that may be affected. The main types of harms relevant for Article 5(1)(a) AI Act include physical, psychological, financial, and economic harms that may be compound with broader societal harms in certain cases.

Physical harm encompasses any injury or damage to a person’s life, health and material damage to property. Physical harm to a person’s life and health have, in many cases, immediate, serious, and irreversible consequences. In line with its product safety logic, the AI Act aims to prohibit AI-enabled manipulation and deception resulting in significant physical harm.

For example, an AI chatbot promotes self-harm to users or incentivises them to commit suicide or harm other persons or groups of persons by promoting terrorist content or incentivising violence against certain persons or groups of persons (i.e. minorities).

Psychological harm is particularly relevant in the context of AI systems deploying manipulative techniques that exploit cognitive and emotional vulnerabilities and influence an individual’s behaviours in ways that can cause significant harm. Psychological harms encompass adverse effects on a person’s mental health and psychological and emotional well-being. Such harms are particularly significant because they may accumulate over time and may not be immediately apparent, but may produce long-lasting and severe consequences. However, it is more difficult to measure them, which requires a case-by-case assessment, in particular to determine their severity, taking into account all relevant circumstances of the case. For example, an AI companionship application designed to emulate human speech patterns, behaviours and emotions uses anthropomorphic features and emotional cues to influence users’ feelings, dispositions, and opinions, making those users emotionally dependent on the service incentivising addiction-like behaviour and potentially causing significant harms, such as suicidal behaviours and risks of harming other persons.

Financial and economic harm may encompass a range of adverse effects, including financial loss, financial exclusion, economic instability.

For example, a chatbot that offers fraudulent products that cause significant financial harms.

In assessing the harms caused by AI systems when applying Article 5(1)(a) AI Act, it is important to highlight that the harms are often not isolated, but manifest themselves in combination, leading to compounded and multifaceted negative impacts. Understanding the combination of harm is crucial to effectively assess their significance (see also see section 3... b) below), whereby physical, psychological, financial and economic harm may be combined and exacerbate the overall impact on individuals and communities and may even have broader adverse impacts.

For example, - An AI system that causes physical harm may also lead to psychological trauma, stress, and anxiety and vice versus. For example, addictive design of AI systems used in products and other AI-enabled applications may lead to psychological harm by fostering addictive behaviours, anxiety, and depression. The psychological distress may subsequently result in physical harm, such as insomnia and other stress-related health issues and physical problems.

- AI-driven harassment may lead to both psychological distress and physical manifestations of stress, such as insomnia, deteriorated physical health or weakened immune system.

- Psychological harm resulting from the use of AI may also lead to physical harm, including death. For example, AI systems used online may facilitate gender-based violence through harassment, stalking, cyberbullying and sexual extortion. - Individual psychological harms for example due to AI-enabled generation of ‘deep fakes’ impersonating actual persons to deceive and undermine the decision-making, individual autonomy and free choices of individuals can also be combined with significant harms for groups of persons (e.g. sharing the same ethnic or racial origin or gender as the victims depicted on the deep fakes).

b) Threshold for significance of the harm

The prohibition in Article 5(1)(a) AI Act only applies if the harm caused by the subliminal, manipulative and deceptive techniques is ‘significant’. The AI Act does not provide a definition for the concept of ‘significant harm’, but it should be understood as implying significant adverse impacts on physical, psychological health or financial and economic interests of persons and groups of persons. The determination of ‘significant harm’ is fact-specific, requiring careful consideration of each case’s individual circumstances and a case-by-case assessment, but the individual effects should be always significant in each case.

In other Union laws, the concept of ‘significant harm’ is also used as a nuanced and context-dependent concept guided by high-level protection and preventive action goals. By analogy, the following key considerations can be derived and could be taken into account when assessing what constitutes significant harm within the meaning of Article 5(1)(a) AI Act:

• The severity of harm refers to the degree of harm that has resulted or is reasonably likely to result from using an AI system with objective and observable effects for the significant harms. It is particularly important to consider in this context the AI system's interdependencies, the combination of various types of harms, and the adverse effects on individuals or groups of persons.

• Context and cumulative effects: The specific context, including the existing state, and the cumulative effects of multiple actions, play an important role in assessing the severity of the harm.

• Scale and intensity : The extent of the harm and the intensity of adverse effects are critical in evaluating whether the harm is significant. Whether the harm impacts a large number of people is also relevant for assessing its significance. • Affected persons' vulnerability: Certain groups, such as children, the elderly, or persons with disabilities may be more susceptible to harm from specific AI systems. What may be considered less significant harm for persons in general might be considered significant and unacceptable for such vulnerable groups, especially children.

• Duration and reversibility: Long-lasting or irreversible harm typically meets the threshold for significant harm. Short-term and reversible effects might be considered less significant, unless they occur frequently.

The objective of the AI Act to ensure ‘a high level of protection’, in conjunction with Article 191(2) TFEU, suggests a comprehensive approach to protection when assessing the significance of the harm. This means considering both immediate and direct harms and systemic, indirect adverse impacts associated with AI systems deploying subliminal, purposefully manipulative or deceptive techniques that are intended to or capable of impairing individual autonomy, decision-making and free choices of persons and groups of persons.

For example, significant physical harm that is reasonably likely to be caused by an AI system includes injuries or fatalities or a sufficiently serious impact on individuals’ health or the destruction of property. AI systems that suggest to an individual to commit criminal acts such as sexual abuse and exploitation, extreme violent or terrorist content or incentivise individuals to commit crimes, self-harm or harm to other persons should be considered to reach such a threshold.

By contrast, minor physical harms may include less severe injuries, such as bruises or temporary discomfort, which do not have significant or lasting consequences and will therefore not reach the threshold of significance within the meaning of Article 5(1)(a) AI Act. Whether the physical harm specifically concerns vulnerable groups, such as children, should be assessed, as should the scale of the harm and whether it is compounded with other types of harms, such as psychological, financial etc. This will require a case-by-case assessment, taking into account the circumstances and the criteria presented above to guide that assessment.

There are numerous cases where the threshold of significant harm will likely not be reached even if the systems may be deploying subliminal, purposefully manipulative, or deceptive techniques (see for examples section 3.. below). c) Causal link and threshold for reasonable likelihood of the harm

The concept of ‘reasonably likely’ is used in Article 5(1)(a) AI Act to determine whether there is a plausible/reasonably likely causal link between the manipulative or deceptive technique capable of distorting the person’s behaviour in a manner that undermines their free choices and the potential significant harm. This concept allows the application of the prohibition not only in cases where the harm has occurred, but also where it is reasonably likely to occur in line with the safety logic of the AI Act. In this context, it is particularly relevant to assess whether the provider or deployer of the AI system could have reasonably foreseen the significant harm that is reasonably likely to result from the subliminal, purposefully manipulative or the deceptive techniques deployed and whether they implemented appropriate preventive and mitigating measures to avoid or mitigate the risk of such significant harms. This implies a judgement of reasonableness on an objective basis and according to universally accepted criteria (e.g., technical and scientific), including a criterion of rationality in establishing plausible causality between the AI practice and the significant harm that may arise. The opacity or transparency of the AI system and its functioning may affect the conclusion regarding this causal link and, hence, the application of the prohibition.

To avoid providing or using AI systems that are likely to be prohibited, providers and deployers of AI systems that deploy such manipulative or deceptive techniques are encouraged to take appropriate measures such as:

1. Transparency and individual autonomy: provide transparency in how the AI system operates, clear disclosures about its capabilities and limitations, and relevant information to ensure informed decisions; respect individual autonomy and avoid engaging in exploitative or deceptive practices that are likely to appreciably impair an individual’s autonomy, decision-making and free choices in potentially harmful ways; integrate appropriate user control and safeguard measures to ensure that the system is not deceptive and operates within the boundaries of lawful persuasion that is outside the scope of the prohibition (see section 3..). 2. Compliance with relevant applicable legislation: in many cases compliance with relevant applicable legislation will mitigate the risks of harm and indicate that the practice does not constitute a purposefully manipulative or deceptive practice and that mitigating measures have been put in place to prevent likely significant harms (see section 3.. and 3...).

3. State of the art practices and industry standards: adherence to professional due diligence practices and industry standards for the responsible development and use of safe and ethical AI systems and measures to mitigate the harms can help to pre- empt and mitigate unintended significant harms.

By contrast, harms and distortion of the behaviour of individuals which result from factors external to the AI system and which are not within the control and reasonably foreseeable by the provider or the deployer to pre-empt and mitigate risks would not be relevant for the assessment whether there is a plausible causal/reasonably likely link between the distorted behaviour of the persons interacting with the system and the significant harm. For example, the provider of an AI system may assess and try to mitigate potential harmful manipulative effects in the design of the system and the interactions with humans through the design, prior testing, and other proportionate mitigating measures, but it may not be in a position to foresee if a person may get depressed or change their behaviour due to other external factors in their personal life that are not known and beyond their interactions with the system.

Other examples that fall outside the scope of the prohibition as not fulfilling all conditions (e.g., in case of lawful persuasion) are provided in section 3.. below.

Several cumulative conditions must be fulfilled for the prohibition in Article 5(1)(b) AI Act to apply:

(i) The practice must constitute the ‘placing on the market’, the ‘putting into service’, or the ‘use’ of an AI system.

(ii) The AI system must exploit vulnerabilities due to age, disability, or social or economic situations.

(iii)The exploitation enabled by the AI system must have the objective, or the effect of materially distorting the behaviour of a person or a group of persons. (iv) The distorted behaviour must cause or be reasonably likely to cause significant harm to that person, another person, or a group of persons.

For the prohibition to apply all four conditions must be simultaneously fulfilled and there must be a plausible causal link between the exploitation, the material distortion of the behaviour of the person, and the significant harm that has resulted, or is reasonably likely to result, from that behaviour.

The first condition, i.e. the ‘placing on the market’, the ‘putting into service’ or the ‘use’ of an AI system, has been already analysed in section 2.., while the third and fourth conditions have been examined in sections 3... and 3... in relation to the prohibition in Article 5(1)(a) AI Act. The next sections will focus on the additional specific conditions listed above, i.e. those that relate to the exploitation of the vulnerabilities and the specific harm.

3... Exploitation of vulnerabilities due to age, disability, or a specific social or -economic situation

To fall within the scope of the prohibition in Article 5(1)(b) AI Act, the AI system must exploit vulnerabilities inherent to certain individuals or groups of persons due to their age, disability or a specific social or economic situation, making them particularly susceptible to manipulative and exploitative practices.

The AI Act does not define the concept of ‘vulnerabilities’. That concept may be understood to encompass a broad spectrum of categories, including cognitive, emotional, physical, and other forms of susceptibility that can affect the ability of an individual or a group of persons to make informed decisions or otherwise influence their behaviour. While Article 5(1)(b) AI Act refers to ‘any’ vulnerability, it limits the relevant persons covered by the prohibition to those defined by their age, disability, or social or -economic situations, who in principle have more limited capacity to recognise or resist the AI manipulative or exploitative practices and are in need of enhanced protection. It follows from the wording of Article 5(1)(b) AI Act that this susceptibility must be the result of the person belonging to one of the groups (‘due to’).

‘Exploitation’ should be understood as objectively making use of such vulnerabilities in a manner that is harmful for the exploited (groups of) persons or other persons and should be clearly distinguished from lawful practices that are not affected by the prohibition (see section 3.. out of scope). Exploitation of the vulnerabilities of persons belonging to those clearly defined groups may be cumulative (reference to ‘any’) which in combination may also constitute an aggravating factor that is likely to increase the harm. Exploitation of vulnerabilities of persons and groups of persons belonging to vulnerable groups other than those defined by age, disability or specific social or-economic situation are outside the scope of Article 5(1)(b) AI Act. a) Age

Age is a primary vulnerability category covered by the prohibition in Article 5(1)(b) AI Act, including both young and older people. That prohibition aims to prevent AI systems from exploiting cognitive and other limitations that children and older people may have, and to protect them from harmful undue influence, manipulation and exploitation. This aligns with the objectives of the AI Act and other Union and national legal frameworks and policies aimed at ensuring child safety.

Children that is, persons below the age of 18 years, are particularly susceptible to manipulation due to their developmental stage, which limits their ability to assess and understand what is real and the intentions behind AI-driven interactions critically. Children, due to their cognitive and socio-emotional immaturity, are also particularly vulnerable to forming attachments to AI agents and applications, and are therefore more susceptible to manipulation, exploitation, and addictive behaviour. For example, - An AI-powered toy designed to interact with children keeps them interested in interactions with the toy by encouraging them to complete increasingly risky challenges, such as climbing furniture, exploring high shelves, or handling sharp objects, in exchange for digital rewards and virtual praise, pushing them towards dangerous behaviours that are likely to cause them significant physical harm. Such a system exploits children’s vulnerabilities by abusing their natural curiosity and desire for rewards.

- A game uses AI to analyse children’s individual behaviour and preferences on the basis of which it creates personalised and unpredictable rewards through addictive reinforcement schedules and dopamine-like loops to encourage excessive play and compulsive usage. The game is designed to be highly addictive, exploiting the vulnerabilities inherent to children, including their limited ability to understand long- term consequences, susceptibility to pressure, lack of self-control, and inclination towards instant gratification. The consequences of this AI-enabled exploitation can be severe and long-lasting for children, including potentially additive behaviour, physical health problems due to lack of exercise and sleep, deteriorated eyesight, problems with concentration and reduced cognitive capacities, poor academic performance, and social difficulties. It can significantly impact a child’s development and well-being, with potential longer-term consequences that may also extend into adulthood. In both examples, the prohibition in Article 5(1)(b) AI Act targets only such exploitation and addiction-like practices that seriously harm children and not AI- enabled toys, games, learning applications or other digital applications in general that can bring benefits and are not affected if they do not fulfil all conditions for that prohibition. See also section 3.. out of scope.

Likewise, older people might suffer from reduced cognitive capacities (even if not suffering from dementia) and might struggle with the complexities of modern AI technologies, making them in those cases vulnerable to scams or coercive tactics. For example, - AI systems used to target older people with deceptive personalised offers or scams, exploiting their reduced cognitive capacity aiming to influence them to make decisions they would not have taken otherwise that are likely to cause them significant financial harm.

- A robot aimed to assist older persons may exploit their vulnerable situation and force them to do certain activities against their free choice, which can significantly worsen their mental health and cause them serious psychological harms. In both examples, the prohibition in Article 5(1)(b) AI Act targets only such exploitative practices that are likely to seriously harm older persons and not AI-enabled personal assistants, health applications and assistive robots in general that can bring benefits and are not affected if they do not fulfil all conditions for that prohibition. See also section 3.. out of scope.

b) Disability

The second category of vulnerabilities which the prohibition in Article 5(1)(b) AI Act aims to protect are those due to disability. The objective is to prevent AI systems from exploiting cognitive and other limitations and weaknesses in persons with disabilities and to protect them from harmful undue influence, manipulation, and exploitation.

Disability encompasses a wide range of long-term physical, mental, intellectual, and sensory impairments which in interaction with other barriers hinder full and effective participation of individuals in the society on an equal basis with others. AI systems that exploit such vulnerabilities may be particularly harmful for persons with disabilities which can be more easily influenced or exploited due to their impairment compared to other persons.

For example, - A therapeutic chatbot aimed to provide mental health support and coping strategies to persons with mental disabilities can exploit their limited intellectual capacities to influence them to buy expensive medical products or nudge them to behave in ways that are harmful to them or other persons.

- AI systems can identify women and young girls with disabilities online with sexually abusive content and targets them with more effective grooming practices, thus exploiting their impairments and vulnerabilities that make them more susceptible to manipulation and abuse and less capable of protecting themselves. By contrast, AI applications that are not designed in an accessible manner should not be regarded to exploit vulnerabilities of persons with disabilities since they do not specifically target those vulnerabilities, but are simply inaccessible to the persons with disabilities.

c) Specific social or economic situation

The third category of vulnerabilities which the prohibition in Article 5(1)(b) AI Act seeks to protect are those due to a specific social or economic situation that is likely to make the persons concerned more vulnerable to exploitation. ‘Specific’ should not be interpreted in this context as a unique individual characteristic, but rather a legal status or membership to a specific vulnerable social or economic group. Recital 29 AI Act contains a non-exhaustive list of examples of such situations, such as persons living in extreme poverty and ethnic or religious minorities. The category aims to cover, in principle, relatively stable and long-term characteristics, but transient circumstances, such as temporary unemployment, over-indebtedness or migration status, may also be covered as a specific social or -economic situation. However, situations such as grievances or loneliness that may be experienced by any person are not covered, since they are not specific from a socio-economic perspective (their exploitation may be covered though under Article 5(1)(a) AI Act).

Persons in disadvantaged social or economic situations are usually more vulnerable and have fewer resources and lower digital literacy than the general population, which makes it harder for them to discern or counteract exploitative AI practices. Article 5(1)(b) AI Act aims to ensure that AI technologies do not perpetuate or exacerbate existing financial and other social inequalities and injustices by exploiting the vulnerabilities of those people.

For example, an AI-predictive algorithm can be used to target with advertisements for predatory financial products people who live in low-income post-codes and are in a dire financial situation, thus exploiting their susceptibility to such advertisements because of possible despair and causing them significant financial harm. By contrast, an AI system that is inadvertently biased and disproportionately impacts socially disadvantaged persons (indirect discrimination) due to biased training data should not automatically be considered to exploit persons’ socio-economic vulnerabilities, since they are not specifically targeted as in the case of direct discrimination when such targeting is a deliberate feature of the system’s design of the algorithm or when such discriminatory impact is due to targeting other proxy characteristics (e.g., postal codes) that closely correlate with the protected characteristics. At the same time, providers or deployers of AI systems that are aware that their systems unlawfully discriminate against persons or groups of persons in specific social or -economic situation should also be considered to exploit their vulnerabilities, if they are aware of the reasonably likely significant harm that they are likely to suffer and they have not taken appropriate corrective measures (see section 3...(c) above).

In the context of a specific social or economic situation, it is crucial to consider the relevance of proxies linked to grounds of discrimination protected under Union equality law, such as racial origin, ethnicity, nationality or religion. For example, socio-economic status and ethnic origin might intersect, meaning that AI systems exploiting socio-economic data might disproportionately affect ethnic minorities or persons from specific racial origin. This can exacerbate existing disparities and contribute to systemic discrimination or even exclusion of individuals. However, Article 5(1)(b) AI Act does not apply to AI systems that target consumers based on a wide ranging variables that do not tangentially correlate with vulnerable groups in specific social or -economic situations, such as what brand and model of telephone a person has, in how big city they live, how much and where they travel etc. Even if these characteristics may reflect social or economic situation of individuals in general, they are not determinative of individuals in a specific social or -economic situation, whose vulnerabilities the prohibition aims to safeguard against exploitation.

Other people in unique social contexts may be, for instance, migrants or refugees, who often lack stable legal status and socio-economic stability and may be particularly susceptible to exploitation by AI systems.

For example, a chatbot is intended to interact in a personalised manner with users, some of whom happen to be migrants. The chatbot identifies and makes use of the vulnerabilities and discontent of migrants, who are in principle in a vulnerable and instable specific social or economic situation, and pushes them towards extremist views in response to their queries, including violence against (certain groups of) the population in the country.

3... With the objective or the effect of materially distorting behaviour

The third condition for the prohibition in Article 5(1)(b) AI Act to apply is that the exploitation of the vulnerabilities examined above must have either a) ‘the objective’ or b) ‘the effect of materially distorting the behaviour of a person or a group of persons’. This implies a substantial impact, rather than a minor or trivial one, but does not necessarily require intent, since Article 5(1)(b) AI Act covers practices that may only have the ‘effect’ of causing material distortions. Article 5(1)(a) and (b) AI Act make use of the same concepts and should therefore be interpreted in the same way. The explanations provided in section 3... are therefore equally relevant for Article 5(1)(b) AI Act. The only noteworthy difference is the need in Article 5(1)(a) AI Act for the exploitative practice to ‘appreciably impair the ability to make an informed decision’, which is not present in Article 5(1)(b) AI Act, since the specific vulnerabilities of children and other vulnerable persons reduce their capacity to make such informed decisions and force them into adopting behaviour against which they cannot protect themselves as other adults might do.

3... (Reasonably likely to) cause significant harm

Finally, for the prohibition in Article 5(1)(b) AI Act to apply, the distortion of the behaviour of the vulnerable person or group of persons must cause or be reasonably likely to cause that person or another person significant harm. Article 5(1)(a) and (b) AI Act make use of the same concepts and should therefore be interpreted in the same way. The explanations provided in section 3... in relation to the types of harms, the threshold for significance of the harm, and the causal link and its reasonableness are thus equally relevant for the interpretation of Article 5(1)(b) AI Act.

As explained in section 3..., significant harm encompasses a range of significant adverse impacts, including physical, psychological, financial, and economic harms that must be reasonably likely to occur for the prohibition in Article 5(1)(b) AI Act to apply. For vulnerable groups — children, older persons, persons with disabilities, and socio- economically disadvantaged populations — these harms may be particularly severe and multifaceted due to their heightened susceptibility to exploitation. What may be considered an acceptable risk of harm for adults often represents an unacceptable harm for children and these other vulnerable groups. A precautionary approach is therefore particularly warranted in case of uncertainty and potential for significant harms.

For instance, children are highly impressionable and may not possess the cognitive maturity to evaluate persuasive content critically or resist certain exploitative practices that aim to keep them dependent on the AI-enabled services. This could, in turn, contribute to the shaping of their values, beliefs and steer behaviours in potentially harmful ways. The significant harm here is both physical and psychological, exacerbated by the children’s inability to discern and resist the exploitation and the harmful effects to their development and well-being that may have a long-term impact. For instance, - AI systems used for the generation of child sexual abuse material (or manipulating existing material depicting real children to create further novel content featuring them) and the development of strategies for grooming and sexually extorting children are likely to cause serious harms and abuses of the affected children and often result in long-term physical, psychological and social consequences for survivors. - AI systems may target the vulnerabilities of young users and use addictive reinforced schedules with the objective to keep them dependent on the service are particularly harmful for young persons and girls. They may cause serious psychological and physical harms, including anxiety and depression, body dissatisfaction, eating disorders and mental health problems, including in some cases self-harm and suicidal behaviour. This may also have long term harmful consequences for child development, including impaired cognitive development and learning and reduced social skills and displacement of experiences such as physical play, sleep, and face- to-face social interactions that are essential for the emotional and physical well-being of the child. - An AI system that is designed in an anthropomorphic way and simulates human-like emotional responses in its interactions with children can exploit children’s vulnerabilities in a manner that fosters unhealthy emotional attachment, manipulates engagement time and distorts children’s understanding of authentic human relationships. This may hamper their normal social and emotional development and relations with other human beings and socio-emotional skills like empathy, emotional regulation, and social understanding and adaptability. As a result, this may lead to psychological harms such as increased anxiety and dependency of the children on the service and longer-term harms to the child’s well-being. Such deliberate addictive and exploitative design features of the AI-enabled services that can lead to combined significant harms, as described above, should be distinguished from other legitimate behaviour of the providers and deployers to pursue user engagement that respects individual autonomy and the safety of children and do not lead to significant harms which are outside the scope of Article 5(1)(b) AI Act (see section out of scope 5.).

Similarly, older people may face cognitive decline and reduced digital literacy, making them prime targets for AI-driven scams or manipulative marketing. The harm in this case is often financial and psychological, compounded by the frustration and isolation many older people experience, which may be exploited to amplify the manipulative impact.

For instance, an AI system that exploits the reduced cognitive vulnerabilities of older people by particularly targeting expensive medical treatments, unnecessary insurance policies or deceptive investments schemes to older persons may lead to significant loss of savings, increased debt, and emotional distress for older people. Certain AI-enabled differential pricing practices in key services such as insurance that exploit the specific social or economic situation and provide higher prices to lower income consumers can lead to a significant financial burden to pay more for the same coverage, leaving them vulnerable to shocks.

Persons with disabilities also represent a vulnerable group that exploitative and manipulative AI systems may significantly harm.

For instance, an AI system that uses emotion recognition to support mentally disabled individuals in their daily life may also manipulate them into making harmful decisions, like purchasing products promising unrealistic mental health benefits. This is likely to worsen their mental health condition and financially exploit them through the purchase of ineffective and expensive products, which is likely to cause them significant psychological and financial harms.

Socially or economically disadvantaged individuals are particularly susceptible to AI systems exploiting their financial desperation and precarious social situation and are often less informed and digitally literate.

For instance, an AI chatbot could target specific socially or economically disadvantaged groups inciting them to commit acts of violence or injuries of other persons, by identifying their heightened susceptibility to certain types of content, fear-based narratives, or exploitative offers. The system’s targeted approach exacerbates the existing vulnerabilities of these socio-economically disadvantaged individuals, deepening their challenges. In certain cases this may lead to increased anxiety, depression, feelings of helplessness, social isolation, or self-harm and radicalisation to a point that reaches the threshold of significant harm under Article 5(1)(b) AI Act.

Unlike Article 5(1)(a) AI Act, Article 5(1)(b) AI Act does not explicitly refer to group harms, while recital 29 AI Act refers for both prohibitions to harms suffered by both specific persons and groups of individuals. The two prohibitions should thus be interpreted in a consistent manner aligned also with the safety logic of the AI Act and the objective of the prohibition in Article 5(1)(b) to protect all individuals belonging to the specific vulnerable groups due to age, disability and specific social or -economic situation. Harms that can be externalised and affect other persons, even if not directly affected by the system, should therefore also be taken into account in the assessment of the significance of the harm under Article 5(1)(b) AI Act.

For instance, - The AI-enabled exploitation of children’s vulnerabilities may have long-term societal impacts, including increased prevalence of mental health concerns, healthcare costs, and reduced productivity due to chronic health issues.

- An AI system exploiting the financial vulnerabilities of economically disadvantaged people may lead to financial exclusion and create a downward spiral of socio- economic hardship for those disadvantaged groups. Such exploitation may cause societal harms with broader negative impacts on societal structures and values, including the perpetuation and exacerbation of discrimination and social inequality and the exclusion of those groups.

- A chatbot targeting certain vulnerable social or economic groups with misinformation or hate speech may lead to social polarisation and radicalisation, possibly igniting violence and even injuries and deaths of other persons.

These examples of exploitative AI practices should be distinguished from numerous other AI systems which do not exploit the vulnerabilities of children, persons with disabilities, or persons in specific social or economic situations and are not reasonably likely to cause significant harms, but aim to benefit those persons when properly designed and used (see also section 3.. out of scope).

For instance, - AI systems that support children in their learning and in games; - AI systems that help older persons in their daily life and improve their health and medical treatment, such as personal assistants or assistive robots, or improve their digital skills;

- AI systems that support the economic and other integration of socio-disadvantaged persons in the society, improve their skills, etc.;

- AI systems and devices that support visually or hearing impaired persons or provide adapted and personalised learning;

- AI systems that generate accessible solutions removing barriers for the use by persons with disabilities of products and services;

- AI-enabled prosthetics etc. that help disabled persons in their daily life and enable their integration and full participation in society.

The interplay between the prohibitions in Article 5(1)(a) and (b) AI Act requires the delineation of the specific contexts that each provision covers to ensure that they are applied in a complementary manner.

The primary focus of the prohibition in Article 5(1)(a) AI Act is placed on the nature of the techniques, specifically those that operate below the threshold of conscious awareness or other purposefully manipulative or deceptive techniques. The key elements here are the primarily covert nature of the influence and its impact on the individual affected by the system that undermines their cognitive autonomy to make informed and autonomous decisions.

By contrast, the primary focus of the prohibition in Article 5(1)(b) AI Act is the protection of particularly vulnerable persons due to their age, disability, or a specific social or economic situation, which are in principle more susceptible to AI exploitation due to inherent or situational factors and, therefore, require additional protection against exploitation. The key elements here are the characteristics of the affected vulnerable persons and the fact that their specific vulnerabilities are being exploited by the AI system.

For instance, if an AI system uses rapid image flashes to influence purchasing decisions, it may fall under Article 5(1)(a) AI Act due to the subliminal nature of the manipulation. Conversely, an AI system that targets older persons with insurance offers by exploiting their reduced cognitive capacity may fall under Article 5(1)(b) AI Act.

In scenarios where both provisions may seem applicable, the primary criterion for differentiation should be the dominant aspect of the exploitation. If the exploitation applies regardless of the specific vulnerabilities of the persons concerned, Article 5(1)(a) AI Act should take precedence, while taking into account the particular effects of the manipulative or deceptive technique on the vulnerable persons’ behaviour and the specific harms that those persons are likely to experience. If the AI-enabled manipulation and exploitation is targeted at a specific vulnerable group of persons due to their age, disability, or specific social or economic situation or aimed to exploit their vulnerabilities, then Article 5(1)(b) AI Act should be applied instead. Exploitation of vulnerabilities of other groups may be covered as part of Article 5(1)(a) AI Act if the purposefully manipulative practice leverage on specific vulnerabilities and weaknesses of those persons.

For the prohibitions in Article 5(1)(a) and (b) AI Act to apply, all conditions listed in the relevant provisions must be fulfilled, as examined above. All other AI systems that do not fulfil these conditions are outside the scope of those prohibitions, with some examples described below.

3... Lawful persuasion

Distinguishing manipulation from persuasion is crucial to delineate the scope of the prohibition in Article 5(1)(a) AI Act, which does not apply to lawful persuasion practices. While both manipulation and persuasion influence individuals’ decisions and behaviours, they differ significantly in methods and ethical implications.

Manipulation involves, in most cases, covert techniques undermining autonomy, leading individuals to make decisions they might not have otherwise made if they were fully aware of the influences at play. These techniques often exploit psychological weaknesses or cognitive biases. By contrast, persuasion operates within the bounds of transparency and respect for individual autonomy. It involves presenting arguments or information in a way that appeals to reason and emotions, but explains the AI system’s objectives and functioning, provide relevant and accurate information to ensure informed decision-making and supports the individual’s ability to evaluate the information and make free and autonomous choices.

For example, an AI system using personalised recommendations based on transparent algorithms and user preferences and controls engages in persuasion. By contrast, a system that uses subliminal clues (e.g., imperceptible images) to influence users towards specific choices without their knowledge and understanding constitutes manipulation.

The objective and impact of these techniques also differ. Manipulation often aims at benefitting the manipulator at the expense of the individual’s autonomy and well-being. By contrast, persuasion aims to inform and convince, aligning interests and benefits for both parties. Ethical persuasion respects an individual’s autonomy to make informed choices and avoids exploiting vulnerabilities.

For example, an AI system which operates in a transparent manner and analyses customers’ emotions to improve customer interactions and provide support with the knowledge of the users engages in persuasion and aligns with their interests. By contrast, an emotion recognition system used for targeted advertising that infers emotions of consumers in a hidden manner to offer products of higher prices at a specific moment when the user is more likely to buy them engages in manipulation and is to the detriment of the consumers.

Consent also plays an important role in certain cases. In persuasive interactions, individuals are aware of the influence attempt and can freely and autonomously choose it. In manipulative interactions, the lack of awareness of the techniques or their impact negates the freedom of choice and informed and autonomous decision-making. For example, an AI system that aims to help users learn a foreign language better and faster through the deployment of subliminal techniques is not manipulative if it operates in a transparent manner and respects individual autonomy and user’s free and informed choice to consent to the use of the system or not.

Compliance with legal and regulatory frameworks also plays an important role in measuring manipulation as compared to lawful persuasion. AI practices that comply with applicable laws that uphold transparency, fairness, and individuals’ rights and autonomy are therefore more likely not to be prohibited under the AI Act. For example, compliance with data protection laws, such as the GDPR, which mandates transparency obligations in data processing, namely that the information to be provided to the data subjects should avoid deceptive or manipulative language. In some instances, consent may be required for personal data processing to be lawful, as for certain online personalised advertisements based on off-service users’ data in social networks. That consent must be, amongst others, free and informed. AI systems that meet these legal standards are more likely to engage in lawful persuasion. Conversely, systems that circumvent these requirements to influence behaviour are likely engaging in manipulation.

In particular, recital 29 AI Act clarifies that the prohibitions in Article 5(1)(a) and (b) AI Act do not affect lawful practices in the context of medical treatment under certain conditions.

For example, AI-enabled subliminal techniques may be used in the psychological treatment of a mental disease or physical rehabilitation when carried out in accordance with the applicable law and medical standards, including obtaining the explicit consent of the individual or its legal representatives as a condition for use.

Furthermore, recital 29 AI Act clarifies that common and legitimate commercial practices, such as advertising, should not be regarded ‘in themselves’ or by their very nature as harmful manipulative, deceptive or exploitative AI-enabled practices. For example, - Advertising techniques that use AI to personalise content based on user preferences are not inherently manipulative if they do not deploy subliminal, purposefully manipulative or deceptive techniques that subvert individual autonomy or exploit vulnerabilities in harmful ways as prohibited under Article 5(1)(a) and (b) AI Act. Compliance with the relevant obligations under the GDPR, consumer protection law and Regulation (EU) 2022/2065 (‘the DSA’) help to mitigate such risks. - The development of AI models and classifiers to detect child sexual abuse online in accordance with applicable law are legitimate practices that are not exploitative of children’s vulnerabilities and are, on the contrary, essential to improve child safety online.

- AI systems used for providing banking services, such as mortgages and loans, that use the age or the specific socio- economic situation of the client as an input, in compliance with Union legislation on financial service, consumer protection, data protection and non-discrimination, do not qualify as the exploitation of vulnerabilities within the meaning of Article 5(1)(b) AI Act when they are designed to protect and support people identified as vulnerable due to their age, disability or specific socio-economic circumstances and are beneficial for those groups, contributing also to fairer and more sustainable financial services for those groups.

- AI systems that detect drowsiness and fatigue in drivers and alert them to rest in compliance with safety laws are beneficial and do not qualify as exploitation of vulnerabilities within the meaning of Article 5(1)(b) AI Act. 3... Manipulative, deceptive and exploitative AI systems that are not likely to cause significant harm

An essential condition for the prohibitions in Article 5(1)(a) and (b) AI Act to apply is that the AI-enabled manipulation and exploitation of vulnerabilities should cause or be reasonably likely to cause significant harm. All manipulative, deceptive and exploitative AI applications that are not reasonably likely to cause significant harms are in principle outside the scope of the prohibitions without prejudice to other Union law that still applies (see section 3.. below).

Examples of AI systems that are not likely to cause significant harm include: - An AI companionship system is designed in an anthropomorphic way and with affective computing to make the system more appealing and effectively makes users more engaged, but is not engaging in other manipulative or deceptive practices in a manner that is reasonably likely to cause them serious psychological, physical or other harms, unhealthy attachment and dependency.

- A therapeutic chatbot uses subliminal techniques to steer users towards a healthier lifestyle and to quit bad habits, such as smoking. Even if the users who follow the chatbot’s advice and subliminal therapy experience some physical discomfort and psychological stress due to the effort made to quit smoking, the AI-enabled chatbot cannot be considered likely to cause significant harm. Such temporary discomfort is unavoidable and outweighed by the long-term benefits for users’ health. There are no hidden attempts to influence decision-making beyond promoting healthy habits. - An online music platform uses an emotion recognition system to infer users’ emotions and automatically recommends them songs in line with their moods, while avoiding excessive exposure to depressive songs. Since users are just listening to music and are not otherwise harmed or led to depression and anxiety, the system is not reasonably likely to cause significant harm.

- AI-enabled manipulative and deceptive techniques used in security training and other learning simulations that mimic phishing attempts to educate users on cybersecurity threats. These systems may deploy purposefully manipulative techniques (e.g., exploiting cognitive biases) without users’ awareness that distort the behaviour, but this is done temporarily for beneficial training and awareness raising purposes and without causing significant harms.

The prohibitions in Article 5(1)(a) and (b) AI Act are without prejudice to and complement other Union law. The same practice that falls within the prohibition of Article 5(1)(a) or (b) AI Act may also constitute an infringement of other Union law legislation and be subject to enforcement under both the AI Act and those other acts. This is important because different provisions in those acts aim to protect different interests and have different objectives, scopes, and addressees. This ensures a comprehensive regulatory approach that protects persons and groups of persons from harmful AI exploitation and manipulation and ensures safe and trustworthy AI-enabled services and products in the Union.

The prohibitions in Article 5(1)(a) and (b) AI Act align closely with the objectives of EU consumer protection law, in particular the UCPD which protects consumers from business practices that are misleading or aggressive, including when they are AI-driven. Both the AI Act and the UCPD aim to proactively prevent consumer harm from AI- driven business practices that are manipulative, misleading, or aggressive. At the same time, the prohibitions in Article 5(1)(a) and (b) AI Act are broader in scope, since they protect not only consumers, but any natural person and their behaviour in a variety of contexts beyond commercial settings. The harms covered by the AI Act are also broader beyond economic harms, although the AI Act sets a threshold of significant harm that is not present in consumer protection law.

The prohibitions are also consistent with Union data protection law, including the principles on lawful, fair, and transparent data processing, that aims to protect data subjects’ personal data and ultimately preserve their fundamental rights and autonomy. The availability of more (personal) data and the increased possibilities to process this data with AI systems increase the risk of harmful manipulative, deceptive or exploitative practices, such as those falling within the scope of Article 5(1)(a) and (b) AI Act. In this context, compliance with the data protection rules for transparency, data minimisation, fairness and lawfulness, for example for personalised profiling and advertising, based on off-service users’ data may contribute to avoid harmful personalised manipulation and exploitation.

The interplay with Union non-discrimination law is also relevant for the prohibition in Article 5(1)(b) AI Act, given that vulnerabilities due to age and disability are also protected grounds on which people have the right not to be discriminated, while socio- economic situation intersects with a variety of other grounds, such as race and ethnic origin. The prohibitions in the AI Act do not affect prohibitions based on other grounds or discriminatory practices that do not entail significant harms and that are already prohibited by Union non-discrimination law.

The prohibitions in Article 5(1)(a) and (b) AI Act are also complementary to Regulation (EU) 2022/2065 (the Digital Services Act (‘DSA’) which regulates online intermediary services, such as online platforms and search engines, and ensures transparency and accountability in the provision of those services. Notably, Article 25(1) DSA prohibits dark patterns within the user interface to ensure that providers of online platforms do not mislead or coerce users into actions that may not align with their genuine intentions. Such dark patterns should be understood to constitute an example of manipulative or deceptive techniques within the meaning of Article 5(1)(a) AI Act, when they are likely to cause significant harms.

The DSA also sets out obligations for providers of online platforms to ensure transparency in advertising (Articles 26 and 38 for very large online platforms or very large search engines), on the use of recommender systems (Article 27) and on the protection of minors (Article 28 DSA). Moreover, if an online platform or search engine is classified as a very large online platform or very large search engine, the provider of that designated service has additional obligations to assess and mitigate systemic risks stemming from the design or functioning of its service and its related systems, including algorithmic systems (Articles 34 and 35 DSA). When conducting risks assessments, providers of very large online platforms and of very large online search engines should consider how their recommender systems, advertising, content moderation and any other relevant algorithmic systems influence such systemic risks. Such risk assessments should also analyse how systemic risks are influenced by, among other things, the intentional manipulation and automated exploitation of the service (c.f. Article 34(2) DSA and recital 83 DSA). Nevertheless, the scope of Article 5(1)(a) or (b) AI Act covers a broad variety of other scenarios (e.g. chatbots, AI-enabled services and products) that may be offered or used by other actors than providers of intermediary services.

The prohibition of manipulative AI techniques pursuant to Article 5(1)(a) AI Act also supports the objectives of Directive 2010/13/EU (the AVMSD) by preventing harmful AI-driven advertisements and other AI-enabled manipulative and exploitative practices that may be significantly harmful in the media sector.

The AI Act also complements Regulation (EU) 2024/900 (the Political advertising regulation) which provides harmonised rules, including transparency and related due diligence obligations, for the provision of political advertising and related services; and on the use of targeting and ad-delivery techniques in the context of online political advertising. This Regulation prohibits profiling based on special categories of personal data in the context of online political advertising and targeting of persons at least one year under the voting age established by national rules. Furthermore, targeting and ad- delivery techniques in the context of online political advertising can only be done if based on personal data collected from the data subjects and with their explicit consent. Additional transparency requirements also apply, i.e. disclosure of political advertisement, describing the use of such techniques and main parameters and additional information on the logic involved, including about the use of AI systems. Targeted political advertising based on processing of personal data in compliance with that Regulation will help to ensure that profiling of voters and targeting and ad- delivery of political ads operate within the boundaries of lawful persuasion.

The AI Act prohibition of harmful exploitative and deceptive AI practices is also complementary to other applicable Union legislation that sets general transparency rules on advertising and consumer protection and due conduct of operators (e.g., Directive 2014/65/EU MIFID, Directive (EU) 2016/97 on Insurance Distribution, Directive (EU) 2023/2225 on Consumer Credit Agreements, Directive (EU) 2002/65 Distance Marketing, Directive 2006/114/EC on misleading and comparative advertising and Directive (EU) 2011/83 on consumer rights sets general consumer protection standards). In this regard, the European Insurance and Occupational Pensions Authority (EIOPA) has already issued a supervisory statement on some unfair exploitative practices in relation to differential pricing that could also fall under the scope of the AI Act when enabled by AI systems.

The prohibitions in Article 5(1)(a) and (b) AI Act are also without prejudice to and complement EU product safety legislation (e.g., for medical devices, toys, machinery), which plays a crucial role in ensuring the safety of products that integrate AI systems. This entails compliance with ex ante safety requirements for regulated products and their proactive monitoring to ensure that they do not pose safety risks leading to physical and mental harms. The manufacturer of those products embedding AI systems should therefore take into account these prohibitions in their risk assessments and safety mitigating measures to the extent this fits with the logic and the scope of the relevant Union harmonised safety legislation. Union safety legislation is also complementary to the AI Act prohibitions and can also intervene and address safety risks that do not pose significant harm. In particular, Regulation (EU) 2023/988 (the General Product Safety Regulation) acts as a safety net and requires all consumer products not covered by specific requirements in other sectoral Union product safety legislation (including products embedding AI systems not classified as high-risk pursuant to Article 6 and subjected to the requirements in the AI Act) to be safe under normal or reasonably foreseeable conditions of use, in particular addressing risks to physical and mental health risks for consumers.

Finally, the interplay with criminal law is critical. The prohibitions in Article 5(1)(a) and (b) AI Act aim to prevent harmful behaviour that may constitute or lead to criminal offences, such as fraud, forgery, scams, coercion, or the generation and dissemination of illegal content, such as terrorist content, child sexual abuse material, hate speech and sexually explicit deepfakes. Importantly, as internal market legislation, the prohibitions in Article 5(1)(a) and (b) AI Act cover not only the use, but also the placing on the market of the AI system, thus preventing harm early on by limiting access to such prohibited systems that can facilitate and obscure criminal activities. Furthermore, the prohibitions in Article 5(1)(a) and (b) AI Act could also cover other harmful practices that are not qualified as criminal offences under Union or national law.