Article 5(1)(g) AI Act: biometric categorisation for certain ‘sensitive’ characteristics

A wide variety of information, including ‘sensitive’ information, may be extracted, deduced or inferred from biometric information, even without the knowledge of the persons concerned, to categorise those persons. This may lead to unfair and discriminatory treatment, for example when a service is denied because somebody is considered to be of a certain race. AI-based biometric categorisation systems for the purpose of assigning natural persons to specific groups or categories, relating to aspects such as sexual or political orientation or race, violate human dignity and pose significant risks to other fundamental rights, such as privacy and non-discrimination. They are therefore prohibited by Article 5(1)(g) AI Act.

Several cumulative conditions must be fulfilled for the prohibition in Article 5(1)(g) AI Act to apply:

(i) The practice must constitute the ‘placing on the market’, ‘the putting into service for this specific purpose’ or ‘the use’ of an AI system;

(ii) The system must be a biometric categorisation system;

(iii) individual persons must be categorised;

(iv) based on their biometric data;

(v) to deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex-life, or sexual orientation.

For the prohibition to apply, all five conditions must be simultaneously fulfilled. The first condition, i.e. the placing on the market, the putting into service or the use of the AI system, is analysed in section 2.. The prohibition, therefore, applies to both providers and deployers of AI systems, each within their respective responsibilities, not to place on the market, put into service or use such AI systems. The other conditions for the application of the prohibition are further described and analysed below.

The prohibition does not cover the labelling or filtering of lawfully acquired biometric datasets, including for law enforcement purposes.

8... Biometric categorisation system

‘The categorisation of an individual by a biometric system is typically the process of establishing whether the biometric data of an individual belongs to a group with some pre-defined characteristic. It is not about identifying an individual or verifying their identity, but about assigning an individual to a certain category. For instance, an advertising display may show different adverts depending on the individual that is looking at it based on their age or gender.’ Persons may also simply be categorised for statistical reasons, without being identified and without the objective to identify them.

Article 3(40) AI Act defines a biometric categorisation system as an AI system for the purpose of assigning natural persons to specific categories on the basis of their biometric data, unless it is ancillary to another commercial service and strictly necessary for objective technical reasons. As explained in section 7...d), ‘biometric data’ is defined in Article 3(34) AI Act. In particular, biometric data comprises behavioural characteristics that are based on biometric features. The scope of biometric categorisation excludes categorisation according to clothes or accessories, such as scarfs or crosses, as well as social media activity.

Biometric categorisation may rely on categories of physical characteristics (e.g., facial features and form, skin colour) based on which persons are assigned to specific categories. Some of these categories may be of a special ‘sensitive’ nature’ or characteristics protected under Union non-discrimination law, such as race. However, biometric categorisation may also be based on DNA or on behavioural aspects, such as keystroke analysis or a person’s gait.

To fall outside the scope of the definition of biometric categorisation under the AI Act, two conditions – being ‘ancillary to another commercial service and strictly necessary for objective technical reasons’ – must be cumulatively fulfilled.

According to recital 16 AI Act, a purely ancillary feature is a feature that is intrinsically linked to another commercial service, meaning that the feature cannot, for objective technical reasons, be used without the principal service, and the integration of that feature or functionality is not a means to circumvent the applicability of the rules of the AI Act.

For example, the following uses of AI are permitted under Article 5(1)(g) AI Act: - Filters categorising facial or bodily features used on online marketplaces to allow a consumer to preview a product on him or herself could constitute such an ancillary feature, since they can be only used in relation to the principal service which consists in selling a product.

- Filters integrated into online social network services which categorise facial or bodily features to allow users to add or modify pictures or videos could also be considered to be ancillary feature, since such a filter cannot be used without the principal service of the social network services consisting in the sharing of content online.

In contrast, examples of uses that would be prohibited include: - An AI system that categorises persons active on a social media platform according to their assumed political orientation, by analysing the biometric data from the photos they have uploaded on the platform, to send them targeted political messages. While such a system may only be ancillary to the political advertising, it would not be ‘strictly necessary for objective technical reasons’, hence the conditions for excluding it from the definition of biometric categorisation are not fulfilled. - An AI system that categorises persons active on a social media platform according to their assumed sexual orientation by analysing the biometric data from photos shared on that platform and on that basis serves those persons advertisements would qualify as biometric categorisation within the meaning of the AI Act. Also in this case there is no strict necessity for this ‘ancillary service’, hence the exclusion from the prohibition does not apply.

8... Persons are individually categorised based on their biometric data

The use of biometric data for the categorisation of natural persons is an essential element for the prohibition to apply (see above section 8... and 7...d)).

Furthermore, for the prohibition to apply, natural persons must be ‘individually’ categorised. If this is not the purpose or outcome of the biometric categorisation, the prohibition does not apply, for example if a whole group is categorised without looking at the individual.

Examples of individual categorisation include:

- AI systems that conduct ‘Attribute Estimation’ (count demographics), including for example ‘age, gender, ethnicity’, on the basis of for example bodily features, such as face, height, or skin, eye and hair colour (or a combination thereof). - AI systems capable of categorising individuals and singling them out based on a specific feature (e.g., a scar under the right eye), or because they have a tattoo on their right hand.

These use-cases are examples for individual biometric categorisation. For these examples to fall within the prohibition of Article 5(1)(g) AI Act all conditions of that provision must be fulfilled.

8... To deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation

Article 5(1)(g) AI Act prohibits only biometric categorisation systems which have as their objective to deduce or infer a limited number of sensitive characteristics: race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation.

For example, systems prohibited under Article 5(1)(g) AI Act include: - a biometric categorisation system that claims to be capable of deducing an individual’s race from their voice (This is different from a system that categorises persons according to skin or eye colour, or a system that analyses the DNA of victims of crimes in view of their origin. Those systems would not be prohibited). - a biometric categorisation system that claims to be capable of deducing an individual’s religious orientation from their tattoos or faces.

The prohibition in Article 5(1)(g) AI Act does not cover AI systems engaged in the labelling or filtering of lawfully acquired biometric datasets, such as images, based on biometric data, including in the area of law enforcement. This is further explained in recital 30 AI Act.

The labelling or filtering of biometric datasets may be done by biometric categorisation systems precisely to guarantee that the data equally represent all demographic groups, and not, for example, over-represent one specific group. If the data used for training an algorithm are biased against a specific group (i.e. systematic differences in the data exist between groups due to the way the data are collected, or data is historically biased), the algorithm may replicate this bias, possibly resulting in unlawful discrimination against persons or groups of persons. For this reason, labelling on the basis of some protected sensitive information may be necessary for high-quality data, precisely to prevent discrimination. The AI Act may even require labelling operations to conform to the AI Act’s requirements for high-risk AI systems. Such labelling or filtering of biometric data is therefore explicitly exempted from the prohibition in Article 5(1)(g) AI Act. The prohibition only applies where biometric data is categorised to infer race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation.

Examples of permissible labelling or filtering include:

- the labelling of biometric data to avoid cases where a member of an ethnic group has a lower chance of being invited to a job interview because the algorithm was ‘trained’ based on data where that particular group performs worse, i.e. has worse outcomes than other groups. - the categorisation of patients using images according to their skin or eye colour may be important for medical diagnosis, for example cancer diagnoses.

Article 5(1)(g) AI Act also provides that the prohibition in that provision does not apply to the labelling or filtering of lawfully acquired datasets in the area of law enforcement. For example, this covers the use by a law enforcement authority of an AI system that allows labelling and filtering of a dataset suspected of containing child sexual abuse material. In a first step, law enforcement would use the support of AI systems to detect and redact sensitive data from images. Furthermore, filtering and labelling according to gender, age, biometric data such as eye and hair colour, scars and marking could help with identifying the victims or creating links with other cases. Similarly filtering and labelling abusers’ hands based on specific characteristics such as length of fingers or any distinguishing markings or tattoos to help with identifying possible suspects is permitted.

AI systems intended to be used for biometric categorisation according to sensitive attributes or characteristics protected under Article 9(1) GDPR on the basis of biometric data, in so far as these are not prohibited under this Regulation, are classified as high- risk under the AI Act.

Article 5(1)(g) AI Act further restricts the possibilities for a lawful personal data processing under Union data protection law, such as the GDPR, LED, EUDPR. In particular, Article 5(1)(g) AI Act excludes the possibilities for biometric categorisation of natural persons, based on their biometric data, as defined in the AI Act, to infer race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation, subject to the exception for labelling or filtering of lawfully acquired biometric data sets, including in the area of law enforcement, as described above. Moreover, the prohibition in Article 5(1)(g) AI Act is consistent with Article 11(3) LED, which explicitly prohibits any ‘profiling’ that results in discrimination on the basis of special categories of personal data, such as race, ethnic origin, sexual orientation, political opinion, or religious beliefs.