Article 5(1)(f) AI Act emotion recognition

Emotion recognition technology is quickly evolving and comprehends different technologies and processing operations to detect, collect, analyse, categorise, react, interact and learn emotions from persons. Such technology is also referred to as ‘affect technology’. Emotion recognition can be used in multiple areas and domains for a wide range of applications such as for analysing customer behaviour and targeted advertising and neuromarketing; in the entertainment industry, for example to provide personalised recommendations or to predict reactions to movies; in medicine and healthcare, for example to detect depression, for suicide prevention or to detect autism, in education, for example to monitor attention or engagement of learners (pupils and students at different ages); in employment, for example to accompany the recruitment process, to monitor emotions or boredom of employees, but also well-being applications for ‘making workers happier’; for law enforcement and public safety, for example with lie detectors or emotion screening at big events; and for many other purposes.

Emotion recognition is frequently doubted as to its effectiveness or as to its accuracy. Recital 44 AI Act explains that there are ‘serious concerns about the scientific basis of AI systems aiming to identify or infer emotions, particularly as expression of emotions vary considerably across cultures and situations, and even within a single individual. Among the key shortcomings of such systems are the limited reliability, the lack of specificity and the limited generalisability.’ It further explains that emotion recognition can lead to ‘discriminatory outcomes and can be intrusive to the rights and freedoms of the concerned persons’, in particular the rights to privacy, human dignity and freedom of thought. This plays an important role in asymmetric relationships especially in the context of the workplace and education and training institutions, where both workers and students are in particularly vulnerable positions. At the same time, emotion recognition in specific use contexts, such as for safety and medical care (e.g., health treatment and diagnosis) has benefits.

Several cumulative conditions must be fulfilled for the prohibition in Article 5(1)(f) AI Act to apply:

(i) The practice must constitute the ‘placing on the market’, ‘the putting into service for this specific purpose’ or the ‘use’ of an AI system;

(ii) AI system to infer emotions; (iii) in the area of the workplace or education and training institutions; and (iv) excluded from the prohibition are AI systems intended for medical or safety reasons.

For the prohibition to apply all four conditions must be simultaneously fulfilled. The first element, i.e. the placing on the market, putting into service or use of the AI system, has already been analysed in section 2... The prohibition, therefore, applies to both providers and deployers of AI systems, each within their respective responsibilities, not to place on the market, put into service or use such AI systems. The other conditions related to the prohibition are further described and analysed below. 7... AI systems to infer emotions a) AI systems to infer emotions versus emotion recognition systems

Article 3(39) AI Act defines ‘emotion recognition systems’ as AI systems ‘for the purpose of identifying and inferring emotions or intentions of natural persons on the basis of their biometric data. The prohibition in Article 5(1)(f) AI Act does not refer to ‘emotion recognition systems’, but only to ‘AI systems to infer emotions of a natural person’. Recital 44 further clarifies that that prohibition covers AI systems ‘to identify or infer emotions.’

Inferring generally encompasses identifying as a prerequisite, so that the prohibition should be understood as including both AI systems identifying or inferring emotions or intentions. For consistency reasons, it is also important to construe the prohibition in Article 5(1)(f) AI Act as having a similar scope as the rules applicable to other emotion recognition systems (Annex III, point 1(c), and Article 50 AI Act) and to limit it to inferences based on a person’s biometric data. The definition in Article 3(39) AI Act of emotion recognition systems should therefore be considered relevant in relation to Article 5(1)(f) AI Act.

b) Identification and inference of emotions or intentions

‘Identification’ occurs where the processing of the biometric data (for example, of the voice or a facial expression) of a natural person allows to directly compare and identify an emotion with one that has been previously programmed in the emotion recognition system. ‘Inferring’ is done by deducing information generated by analytical and other processes by the system itself. In such a case, the information about the emotion is not solely based on data collected on the natural person, but it is inferred from other data, including machine learning approaches that learn from data how to detect emotions. c) Emotions

For the purpose of Article 5(1)(f) AI Act, the concept of emotions or intentions should be understood in a wide sense and not interpreted restrictively. Recital 18 AI Act provides some detail, listing emotions ‘such as happiness, sadness, anger, surprise, disgust, embarrassment, excitement, shame, contempt, satisfaction and amusement’. These examples are not exhaustive.

The prohibition should not be circumvented by referring to attitudes, and includes cases where the AI system finds on the basis of the biometric data that a person is showing for example an angry attitude.

Recital 18 AI Act clarifies that emotions or intentions do not include ‘physical states, such as pain or fatigue, including, for example, systems used in detecting the state of fatigue of professional pilots or drivers for the purpose of preventing accidents.’ It further clarifies that emotion recognition systems do not include ‘the mere detection of readily apparent expressions, gestures or movements, unless they are used for identifying or inferring emotions’, which should be understood to also apply to Article 5(1)(f) AI Act. Those expressions can be basic facial expressions, such as a frown or a smile, or gestures such as the movement of hands, arms or head, or characteristics of a person’s voice, such as a raised voice or whispering. However, when these readily apparent expressions or gestures are used for identifying or inferring emotions or intentions, they are covered by the prohibition.

For example, - The observation that a person is smiling is not emotion recognition. - Identifying whether a person is sick is not emotion recognition. - A TV broadcaster using a device that allows to track how many times its news presenters smile to the camera is not emotion recognition.

- Concluding that a person is happy is emotion recognition. An AI system that infers that an employee is unhappy, sad or angry towards customers (e.g., from body gestures, a frown or the lack of a smile) is ‘emotion recognition’. - Systems inferring from voice or body gestures, that a student is furious and about to become violent, is ‘emotion recognition’.

- Using AI recognition systems to infer a professional pilot’s or driver’s fatigue to alert them and suggest when to take brakes to avoid accidents is not ‘emotion recognition’, since emotion recognition does not include physical states such as pain or fatigue.

d) On the basis of their biometric data

According to the definition in Article 3(39) AI Act, only AI systems identifying or inferring emotions or intentions based on biometric data constitute emotion recognition systems.

Personal characteristics from which biometric data can be extracted are physical or behavioural attributes. Physiological biometrics employ physical, structural, and relatively static attributes of a person, such as their fingerprints, the pattern of their iris, contours of their face, or the geometry of veins in their hands. Some modalities are microscopic in nature, but still exhibit biological and chemical structures that can be acquired and identified e.g., DNA and odour. Behavioural biometrics monitor the distinctive characteristics of movements, gestures, and motor-skills of individuals as they perform a task or series of tasks. This means that human movements, such as walking (gait analysis) or finger contact with a keyboard (keystrokes), are captured and analysed. Behavioural biometrics encompass a variety of modalities that exhibit both voluntary and involuntary repeated motions and associated rhythmic timings/pressures of body features ranging from signatures, gait, voice, and keystrokes through to eye tracking and heartbeats, electroencephalography (EEG), or electrocardiograms (ECG). The biometric input can relate to one modality (e.g., facial images) or multiple modalities (e.g., facial information combined with electroencephalogram (EEG)). Recital 18 gives as examples facial expressions, gestures such as movement of hands or characteristics of a person’s voice.

For example, - An AI system inferring emotions from written text (content/sentiment analyses) to define the style or the tone of a certain article is not based on biometric data and therefore does not fall within the scope of the prohibition. - An AI system inferring emotions from key stroke (way of typing), facial expressions, body postures or movements is based on biometric data and falls within the scope of the prohibition.

The AI Act definition of biometric data is therefore broad and includes any biometric data used for emotion recognition, biometric categorisation or other purposes. 7... Limitation of the prohibition to workplace and educational institutions

The prohibition in Article 5(1)(f) AI Act is limited to emotion recognition systems in the ‘areas of workplace and educational institutions’. As clarified in recital 44 AI Act, this limitation is meant to address the imbalance of power in the context of work or education.

a) ‘Workplace’

The notion of ’workplace’ should be interpreted broadly. That notion relates to any specific physical or virtual space where natural persons engage in tasks and responsibilities assigned by their employer or by the organisation they are affiliated to, for example in case of self-employment. This includes any setting where the work is performed and can vary widely based on the nature of the job, spanning from indoor office spaces, factories and warehouses to publicly accessible spaces like shops, stadiums or museums, to open-air sites or cars, as well as temporary or mobile work sites. This is independent from the status as an employee, contractor, trainee, volunteer, etc. The notion of ‘workplace’ in Article 5(1)(f) AI Act should also be understood to apply to candidates during the selection and hiring process, consistently with other provisions of the AI Act addressing the placing on the market, putting into service or use of AI systems in the area of employment, workers management and access to self- employment, since there is an imbalance of powers and the intrusive nature of emotion recognition may already apply at the recruitment stage.

For example, - Using webcams and voice recognition systems by a call centre to track their employee’s emotions, such as anger, is prohibited. If only deployed for personal training purposes, emotion recognition systems are allowed if the results are not shared with HR responsible persons and cannot impact the assessment, promotion etc. of the person trained, provided that the prohibition is not circumvented and the use of the emotion recognition system does not have any impact on the work relationship. - Using voice recognition systems by a call centre to track their customers emotions, such as anger or impatience, is not prohibited by Article 5(1)(f) AI Act (for example to help the employees cope with certain angry customers).

- AI systems monitoring the emotional tone in hybrid work teams by identifying and inferring emotions from voice and imagery of hybrid video calls, which would typically serve the purpose of fostering social awareness, emotional dynamics management, and conflict prevention, are prohibited.

- Using emotion recognition AI systems during the recruitment process is prohibited. - Using emotion recognition AI systems during the probationary period is prohibited. - Using cameras by a supermarket to track its employees’ emotions, such as happiness, is prohibited.

- Using cameras by a supermarket or a bank to detect suspicious customers, for example to conclude that somebody is about to commit a robbery, is not prohibited under Article 5(1)(f) AI Act, when it is ensured that no employees are being tracked and there are sufficient safeguards.

b) ‘Education institutions’

The reference to education institutions is broad and should be understood to include both public and private institutions. There is no limitation as regards the types or ages of pupils or students or of a specific environment (online, in person, in a blended mode, etc.). For example, education and training institutions at all levels fall under the scope of the prohibition in Article 5(1)(f) AI Act, including vocational schools, i.e. schools where students learn skills involving the use of their hands 164 and continuous training. Education institutions are normally accredited or sanctioned by the relevant national education authorities or equivalent authorities. A key feature is that education institutions may provide a certificate (respectively participation is a precondition for obtaining a certificate). The prohibition should be understood to also apply to candidates during the admissibility process.

For example, - An AI-based application using emotion recognition for learning a language online outside an education institution is not prohibited under Article 5(1)(f) AI Act. By contrast, if students are required to use the application by an education institution, the use of such emotion recognition system is prohibited.

- An education institution using AI-based eye tracking software when examining students online to track the fixation point and movement of the eyes (gaze point, e.g., to detect if unauthorized material is used) is not prohibited, because the system does not identify or infer emotions. By contrast, if the system is also used to detect emotions, such as emotional arousal and anxiousness, this would fall within the scope of the prohibition.

- Using an emotion recognition AI system by an education institution to infer the interest and attention of students is prohibited. By contrast, if only deployed for learning purposes in the context of a role-play (for example, for training actors or teachers), emotion recognition systems are allowed if the results cannot impact the evaluation or certification of the person being trained.

- Using an emotion recognition AI system by an education institution during admissibility tests for new students is prohibited.

- Using an AI system that allows to capture students talking to each other via their phones or other channels during online lectures by an education institution is not prohibited, since it does not infer emotions. By contrast, if the system is also used to detect emotions, such as emotional arousal, anxiousness and interest, this would fall within the scope of the prohibition.

- An education institution employing an emotion recognition AI system on both teachers (workplace) and students (education) is prohibited. 7... Exceptions for medical and safety reasons

The prohibition in Article 5(1)(f) AI Act contains an explicit exception for emotion recognition systems used in the area of the workplace and education institutions for medical or safety reasons, such as systems for therapeutical use. In light of the AI Act’s objective to ensure a high-level of fundamental rights protection, this exception should be narrowly interpreted.

In particular, therapeutic uses should be understood to mean uses of CE-marked medical devices. Moreover, this exception does not comprise the use of emotion recognition systems to detect general aspects of wellbeing. The general monitoring of stress levels at the workplace is not permitted under health or safety aspects. For example, an AI system intended to detect burnout or depression at the workplace or in education institutions would not be covered by the exception and would remain prohibited.

The notion of safety reasons within this exception should be understood to apply only in relation to the protection of life and health and not to protect other interests, for example property against theft or fraud.

It follows from this narrow interpretation of the exception that any use for medical and safety reasons should always remain limited to what is strictly necessary and proportionate, including limits in time, personal application and scale, and should be accompanied by sufficient safeguards. Such safeguards could include, for example, prior written and motivated expert opinion relating to the specific use case. The necessity should be assessed on an objective basis in relation to the medical and safety purpose, and not refer to the employer’s or educational institution’s ‘needs’. This assessment should inquire whether less intrusive alternative means exist which would achieve the same purpose.

Employers and educators should only deploy emotion recognition systems for medical and safety reasons in case of an explicit need. Data collected and processed in this context may not be used for any other purpose. This is particularly important given that the use of AI management software at work has proven to potentially negatively impact workers’ health and safety. Continuous monitoring via wearables, for instance, may increase work-stress while affecting productivity.

Since Recital 18 AI Act excludes from the definition of emotion recognition systems physical states, such as pain or fatigue, a number of AI systems used for safety reasons would already not fall under that definition, including, for example, systems used in detecting the state of fatigue of professional pilots or drivers for the purpose of preventing accidents.

Other laws, including data protection rules, remain applicable to emotion recognition systems that fulfil the conditions of the exception in Article 5(1)(f) AI Act.

Emotion recognition systems that classify as high-risk systems pursuant to Article 6(2) and Annex III(1)(c) AI Act will need to comply with the high-risk requirements in Chapter III Section 2 AI Act and the transparency obligation of Article 50(3) AI Act. For example, Emotion recognition may be deployed for medical reasons to assist employees or students with autism and improve accessibility for those who are blind or deaf. Such uses would fall within the exception for medical reasons in Article 5(1)(f) AI Act. By contrast, emotion recognition for assessing students’ or employees’ wellbeing, motivation levels, and job or learning satisfaction do not qualify as ‘use for medical reasons’ and would be prohibited.

An employer would be prohibited from deploying AI-enabled devices or digital assistants at the workplace for measuring anxiety based on measured stress levels or for measuring boredom of employees, unless the elevated stress level/lack of concentration would pose a specific danger, for example when deploying dangerous machines or dealing with dangerous chemicals. In the latter case, the employer may not use the data for other purposes, such as assessing the employee’s work performance.

Article 2(11) AI Act provides that the Union or Member States may keep or introduce ‘laws, regulations or administrative provisions which are more favourable to workers in terms of protecting their rights in respect of the use of AI systems by employers’. Collective bargaining agreements which are more favourable to workers may also be allowed or encouraged.

For example, Member States may adopt laws providing that the use of emotion recognition systems in the area of work may not be applied for medical purposes.

As mentioned before, out of scope are:

- AI systems inferring emotions and sentiments not on the basis of biometric data, - AI systems inferring physical states such as pain and fatigue.

Emotion recognition systems used in all other domains other than in the areas of the workplace and education institutions do not fall under the prohibition in Article 5(1)(f) AI Act. Such systems are, however, considered high-risk AI systems. At the same time, such systems may be prohibited in certain cases by virtue of Article 5(1)(a) and (b) AI Act (harmful manipulation and exploitation), or by virtue of other Union legislation. All other applicable legislation, such as Union data protection law, consumer protection etc. continue to apply to such systems.

For example, Emotion recognition systems used in a commercial context for addressing customers do not fall under the prohibition of Article 5(1)(f) AI Act, whether based on biometric data or not. Hence, examples such as AI systems that enable emotion recognition based on keystroke or based on voice messages of customers (e.g., chat messages, use of virtual voice assistants), used in online marketing for applications for displaying personalized messages and for advertisement purposes including in smart environments (‘intelligent billboards’) are not covered by the prohibition. Nevertheless, such practices may be covered by the prohibitions of harmful manipulation and exploitation in Article 5(1)(a) and (b) AI Act, if all conditions for the application of those prohibitions are met.

a) Other systems out of scope

‘Crowd control’ generally refers to the control and monitoring of the behaviour of groups to maintain (public) order and event safety. It is often associated with large crowd events (e.g., soccer or football games, concerts, etc) or specific places, such as airports or trains. Crowd control systems can operate without inferring emotions of individual persons, when for example analysing the general noise and mood level at a given place. In that case, the system would not fall within the scope of Article 5(1)(f) AI Act, because it does not infer emotions of (a concrete) natural person.

However, there may be instances where such crowd control systems infer emotions of individuals, for example whether there are many angry faces. Normally, such AI systems would not fall under the prohibition of Article 5(1)(f) AI Act, since they are typically not used in the workplace or in education institutions.

Also out of scope are systems that are used in the medical field for example care robots, or medical practitioners using emotion recognition systems during an examination at their workplace, and voice monitors that analyse emergency calls.

Such systems will often screen persons that are there in a work context, for example the security staff at a football stadium or at a central station (where such systems are used to recognize aggressive behaviour), or employees in the medical field. In such cases, deployers must employ safeguards to avoid the screening of employees. However, it cannot be completely avoided that such systems also infer the emotions of those employees. Since the primary objective of the system is not targeted at assessing employees’ emotions, these systems should be considered to be outside the scope of the prohibition. Deployers of such systems remain responsible to ensure that employees are not adversely affected by their use.