Article 5(1)(d) AI Act – individual risk assessment and prediction of criminal offences

Recital 42 AI Act explains the background and rationale of the prohibition in Article 5(1)(d) AI Act, namely, that natural persons should be judged on their actual behaviour and not on AI-predicted behaviour based solely on their profiling, personality traits or characteristics.

Several cumulative conditions must be fulfilled for the prohibition in Article 5(1)(d) AI Act to apply:

(i) The practice must constitute the ‘placing on the market’, ‘the putting into service for this specific purpose’ or the ‘use’ of an AI system.

(ii) The AI system must make risk assessments that assess or predict the risk of a natural person committing a criminal offence.

(iii)The risk assessment or the prediction must be based solely on either, or both, of the following:

(a) the profiling of a natural person, (b) assessing a natural person’s personality traits and characteristics.

For the prohibition to apply all three conditions must be simultaneously fulfilled. The first condition, i.e. the placing on the market, putting into service or use of the AI system, has been already analysed in section 2.. The prohibition, therefore, applies to both providers and deployers of AI systems, each within their respective responsibilities not to place on the market, put into service, or use such AI systems for this specific purpose. The other two conditions for the prohibition to apply are analysed below. 5... Assessing the risk or predicting the likelihood of a person committing a crime

Risk assessments to assess or predict the risk of an individual committing a criminal offence are often referred to as individual ‘crime prediction’ or ‘crime forecasting’. While there is no generally agreed definition of ‘crime prediction’ or ‘crime forecasting’, these terms refer in general to a variety of advanced AI technologies and analytical methods applied to large amount of often historical data (including socio- economic data, but also police records, etc.) which, in combination with criminology theories, are used to forecast crime as a basis to inform police and law enforcement strategies and action to combat, control, and prevent crime.

Crime prediction AI systems identify patterns within historical data, associating indicators with the likelihood of a crime occurring, and then generate risk scores as predictive outputs. For example, such systems may be used for planning police task forces, for monitoring high-risk situations, and for conducting controls of persons predicted as likely (re-)offenders. Such systems bring opportunities for law enforcement authorities, especially those with scarce resources, increasing efficiency, and enabling a proactive approach for detecting, deterring, and anticipating criminal offences. However, such use of historical data on crimes committed to predict other persons’ future behaviour may perpetuate or even reinforce biases, and may result in crucial individual circumstances being ‘overlooked’ when these circumstances are not part of the data set or considered in the algorithms on which the particular AI system operates. This may also undermine public trust in law enforcement and the justice system in general.

Such risk assessments and predictions are, in principle, forward-looking and concern future criminal offences (not yet committed) or crimes that are assessed as a risk of being committed at the moment, including in cases of an attempt or preparatory activities undertaken to commit a criminal offence. They can be made at any stage of the law enforcement activities, such as during prevention and detection of crimes, but also during the investigation, prosecution and execution of criminal penalties (including when judicial authorities assess the risk of re-offending e.g., in the context of making decisions on the imposition of pre-trial detention) as well as part of the individuals’ plan for re-integration into society after serving a criminal sentence.

The prohibition in Article 5(1)(d) AI Act does not outlaw crime prediction and risk assessment practices as such. It only applies to AI systems for making risk assessments to assess or predict the risk of a natural person committing a criminal offence, where also the third condition referred to above is met. Moreover, as noted, the prohibition does not apply in the situations described in the express exclusion contained in the last phrase of Article 5(1)(d) AI Act.

5... Solely based on profiling of a natural person or on assessing their personality traits and characteristics

The third condition for the prohibition in Article 5(1)(d) AI Act to apply is that the risk assessment to assess or predict the risk of a natural person committing a crime must be based solely on a) the profiling of the person or b) on the assessment of their personality traits and characteristics.

The prohibition in Article 5(1)(d) AI Act applies, irrespective of whether the AI system profiles or assesses the personality traits and characteristics of only one natural person or a group of natural persons simultaneously, since the prohibition aims to protect every individual in respect of whom the risk of committing a criminal offence is being predicted or assessed.

a) Profiling of a natural person

Unlike Article 5(1)(c) AI Act, Article 5(1)(d) explicitly uses the term ‘profiling’. Article 3(52) AI Act defines that term by reference to its definition in Article 4(4) GDPR. The concept of profiling includes the objective to ‘evaluate certain personal aspects’ as one of its core elements. In the context of Article 5(1)(d) AI Act, the profiling is done for the purposes of assessing or predicting the risk of a person committing a crime.

The concept of so-called group profiling may also be relevant in this context. That concept refers to the construction and the application of a descriptive profile for a given group, for example categories of perpetrators of criminal offences (e.g., terrorists, gangsters etc.) constructed on historic data about previously committed crimes by other persons. Those group profiles may be used later to assess and predict the risk of other persons committing similar offences. Whenever an AI system makes prediction and applies such a (group) profile to a specific individual, this constitutes profiling of the person and may therefore fall within the prohibition of Article 5(1)(d) AI Act. b) Assessment of personality traits and characteristics

The prohibition also applies if the risk assessment to assess or predict the risk of the person committing a criminal offence is only based on assessing the person’s personality traits and characteristics. Such an assessment or prediction is often included in the concept of profiling, but it could also be seen as an alternative, if profiling as defined in Article 4(4) GDPR cannot be established.

As mentioned in section 4...c), personality traits and characteristics constitute a broad category of characteristics related to a particular natural person, for which there is no generally agreed taxonomy. Recital 42 AI Act provides examples of personality traits and characteristics which may be assessed for predicting the risk of a person committing an offence, such as ‘nationality, place of birth, place of residence, number of children, level of debt or type of car’. This is only an illustrative and not an exhaustive list. c) ‘Solely’

Article 5(1)(d) AI Act provides that the risk assessments covered by that provision are only prohibited where they are based ‘solely’ on the profiling of a person or the assessment of their personality traits and characteristics. It is clear from Recital 42 AI Act that ‘solely’ is intended to apply both to profiling or to the assessment of personality traits and characteristics.

The condition that the risk assessment must be based ‘solely’ on profiling or assessing personality traits and characteristics may not be fulfilled in a number of situations.

As is evident from the last phrase of Article 5(1)(d) AI Act, such a situation arises, in any event, where the AI system is used to support the human assessment of the involvement of a person in a criminal activity, which is already based on objective and verifiable facts directly linked to a criminal activity. As Recital 42 makes clear, in this context one should think, in particular but not necessarily exclusively, of a situation in which a reasonable suspicion in respect of the natural person concerned already exists. After all, in such cases there will normally have been a human assessment, which will normally be based on relevant objective and verifiable facts.

However, there can also be other situations, which will always need to be assessed on a case-by-case basis. On the one hand, the use of the term ‘solely’ leaves open the possibility of various other elements being taken into account in the risk assessment, which makes that it is no longer based on profiling or assessing personality traits or characteristics alone. On the other hand, in order to avoid circumvention of the prohibition and ensure its effectiveness, any such other elements will have to be real, substantial and meaningful for them to be able to justify the conclusion that the prohibition does not apply. A reading of the prohibition of Article 5(1)(d) AI Act together with the exclusion contained in the last phrase thereof suggests that, in particular, the existence of certain pre-established objective and verifiable facts may justify that conclusion.

For example, - A law enforcement authority uses an AI system to predict criminal behaviour for crimes such as terrorism solely based on individuals’ age, nationality, address, type of car, and marital status. With that system, individuals are deemed more likely to commit future offences that they have not yet committed solely based on their personal characteristics. Such a system may be assumed to be prohibited under Article 5(1)(d) AI Act.

- National tax authorities use an AI predictive tool to review all taxpayers’ tax returns to predict potential criminal tax offences to identify cases requiring further investigation. This is done solely on the basis of the profile built by the AI system, which uses for its assessment personality traits, such as double nationality, place of birth, number of children, and opaque variables, especially inferred information that is predictive and therefore non-objective and hard to verify. Such a system will normally fall under the prohibition of Article 5(1)(d) AI Act, since there is no reasonable suspicion of the involvement of a particular person in a criminal activity or other objective and verifiable facts linking that to that criminal activity. This is also an example that falls within the scope of social scoring prohibited under Article 5(1)(c) AI Act involving unfavourable treatment with data from unrelated social contexts. - A police department uses AI-based risk assessment tool to assess the risk of young children and adolescents being involved in ‘future violent and property offending’. The system assesses children based on their relationships with other people and their supposed risk levels, meaning that children may be deemed at a higher risk of offending simply by being linked to another individual with a high-risk assessment, such as a sibling or a friend. The parents’ risk levels may also impact a child’s risk level. The risk assessments result in police ‘registering’ these children in their systems, monitoring them with additional inspections, and referring them to youth ‘care’ services. Such a system is also likely to fall under the prohibition of Article 5(1)(d) AI Act.

5... Exclusion of AI systems to support the human assessment based on objective and verifiable facts directly linked to a criminal activity

Article 5(1)(d) AI Act provides, in its last phrase, that the prohibition does not apply to AI systems used to support the human assessment of the involvement of a person in a criminal activity, which is already based on objective and verifiable facts directly linked to a criminal activity. Although, as noted, the situation described in this express exclusion is not necessarily the only one in which the prohibition does not apply, including that situation expressly in that provision offers legal certainty by delineating the scope of the prohibition and by making it clear that, where that situation is at issue, the prohibition does not in any event apply.

Where the system falls within the scope of the exclusion and is therefore not prohibited, it will be classified as a high-risk AI system (as referred to in Annex III, point 6(d), AI Act) if intended to be used by law enforcement authorities or on their behalf and therefore subject to the requirements and safeguards, including human oversight (Article 14 and Article 26 AI Act). These requirements include that the human oversight must be assigned to persons with the necessary competence, training and authority who should be able to properly understand the capabilities and limitations of the AI system, correctly interpret its output and address the risk of automation bias. Those persons should have clear procedures, training and the necessary competence and authority to meaningfully assess the outputs of the AI system. In this specific case, their human assessment should ensure that any AI prediction or assessment of the risk of a person committing a crime is based on objective and verifiable facts linked to a criminal activity. Those persons should also intervene in order to avoid negative consequences or risks, or stop the use of the AI system if it does not perform as intended.

Furthermore, the concept of ‘human intervention’ has been subject to CJEU case-law, in particular in the context of solely automated decision-making predicting the risk of air passengers being involved in serious crimes. That case-law may also be relevant for the application of the concept of ‘human assessment’ as used in Article 5(1)(d) AI Act. In the Ligue des droits humains case, the CJEU examined the legality of the use of an advanced AI system for the systematic processing of passenger name record (PNR) data of air travellers to assess their likelihood of being involved in terrorism and other serious crimes.

The CJEU interpreted the rule in Directive (EU) 2016/681 (‘PNR Directive’) that prohibits adverse legal decisions based solely on automated processing and required individual human assessment and review for any positive matches by non-automated means to identify false positives and ensure non-discriminatory results. According to the CJEU, that human assessment, subject to which any results of automated processing of PNR data must be based, must rely on objective criteria to evaluate whether a positive match concerns someone who might be involved in this specific case in terrorist offenses or serious crime, and to ensure the non-discriminatory nature of automated processing.

As to the content of the exclusion, one of its central elements is that the AI system is used to support human assessment, rather than involving the AI system itself making the risk assessment as occurs in the situations covered by the prohibition. However, for the exclusion to apply, that human assessment must, in addition, already be based on objective and verifiable facts directly linked to a criminal activity. 5... Extent to which private actors’ activities may fall within scope

Besides law enforcement authorities that are in principle the main deployers of AI crime predictive systems, the activities of private entities may also be covered by the prohibition in Article 5(1)(d) AI Act in some cases. That follows from the fact that, based on its wording, the prohibition does not apply exclusively to law enforcement authorities. Moreover, otherwise the prohibition might be easily circumvented, which would call into question its effectiveness.

That being so, the prohibition may be assumed to apply, in particular, when private actors are entrusted by law to exercise public authority and public powers for the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties. Private actors may be also explicitly requested on a case-by- case basis to act on behalf of law enforcement authorities and carry out individual crime risk predictions. In those cases, the activities of those private actors could also fall within the scope of the prohibition, if the applicable conditions are fulfilled and the exclusion does not apply.

For example, a private company providing advanced AI-based crime analytic software may be asked by a law enforcement authority to analyse a large amount of data from multiple sources and databases, such as national registers, banking transactions, communication data, geo-spatial data, etc., to predict or assess the risk of individuals as potential offenders of human trafficking offences. If all the criteria for Article 5(1)(d) are met, such a use case could be prohibited.

Furthermore, the prohibition may apply to private entities assessing or predicting the risk of a person of committing a crime where this is objectively necessary for compliance with a legal obligation to which that private operator is subject to assess or predict the risk of persons committing specific criminal offences (e.g., in case of anti- money laundering, terrorism financing).

For example, a banking institution has an obligation under Union anti-money laundering legislation to screen and profile customers for money-laundering offences. If the bank uses an AI system to fulfil its obligations, that should be done based only on the data as specified in that law which are objective and verifiable to ensure that the persons singled out as suspect are reasonably likely to commit anti- money laundering offences. The predictions must also be subject to human assessment and verification in compliance with that legislation in order to ensure the accuracy and appropriateness of such assessments. Compliance with that legislation will ensure that the use of individual crime prediction AI system for anti-money laundering purposes fall outside the scope of the prohibition in Article 5(1)(d) AI Act.

However, having regard to the focus on risk assessments relating specifically and exclusively to the commission of criminal offences that is evident from the wording of the prohibition as well as to the purpose of the prohibition as explained in Recital 42, if a private entity profiles customers for its regular business operations and safety or to protect its financial interests (e.g., detecting financial irregularities) without the purpose of assessing or predicting the risk of the customer committing a specific criminal offence, the activities of the private entities should not be considered to fall under the scope of the prohibition of Article 5(1)(d) AI Act.

In other words, in the absence of private parties having been entrusted by law certain specific law enforcement tasks, acting on behalf of law enforcement authorities or being subject to specific legal obligations as described above, the use of AI systems for making risk assessments in the context of private entities’ ordinary course of business and with the aim of protecting their own private interests, whilst the fact that those risk assessments may relate to the risk of criminal offences being committed merely as a purely accidental and secondary circumstance, is not deemed to be covered by the prohibition.

Location-based or geospatial or place-based crime predictions is based on the place or location of crime or the likelihood that in those areas a crime would be committed. In principle, such policing does not involve an assessment of a specific individual. They therefore fall outside the scope of the prohibition.

Examples of location-based or geospatial predictive or place-based crime predictions - AI-based predictive policing system provides a score of the likelihood of criminality in different areas in a city based on previous criminality rates by area and other supporting information such as street maps, to highlight elevated risk of specific types of criminalities e.g., burglaries, knife crime etc. and help law enforcement authorities determine where to deploy less or more police patrols/presence to carry out community policing to disrupt and deter criminal activity.

- A customs authority uses AI risk analytic tools to predict the likelihood of the location of narcotics or illicit goods, for example on the basis of known trafficking routes. - A police department uses AI-driven systems to detect and locate gunshots in real time. The system employs acoustic sensors in urban areas to identify gunfire sounds and triangulate their location, providing officers with actionable data to aid detection and investigation of crimes.

It may, however, not always be evident how to distinguish location-based crime predictive systems from individual predictive systems assessing the risk of a person committing a crime. To the extent that an AI system carries out location-based predictive policing and then considers the risk score of the location as an aspect in the profiling of a person, that system should be considered person-based and in principle covered by Article 5(1)(d) AI Act, although it may fall outside the scope of the prohibition on other grounds.

For example, if location-based or geospatial information or place-based information is linked to information relating to an individual (e.g., the place of residences of a given person) and the AI system would assess the risk that that person is likely to commit crime based solely on profiling of the individual concerned, including based on his or her residence where crime is high, that system should be considered person-based. 5... AI systems that support human assessments based on objective and verifiable facts linked to a criminal activity

Article 5(1)(d) AI Act provides that the prohibition does not apply to AI systems used to support the human assessment of the involvement of a natural person in a criminal activity, which is already based on objective and verifiable facts directly linked to a criminal activity. In such a case, the individual crime risk assessments and predictions would also not be based solely on profiling or the assessment of personal characteristics and would therefore not be prohibited.

Examples of AI systems falling outside the scope of the prohibition for this reason include:

- The use of an AI system for profiling and categorisation of actual behaviour, such as reasonably suspicious dangerous behaviour in a crowd that someone is preparing and is likely to commit a crime, and there is a meaningful human assessment of the AI classification. In this case, the risk assessment made by the human with the support of AI is not solely based on the personal traits or the profiling, but on objective and verifiable facts linked to the threatening criminal behaviour of that person that has been reviewed by a human before action taken.

- The police is investigating the risk of a possible armed robbery and suspects two individuals. Several verifiable and objective facts are present on which that suspicion is based, such as verifiable participation and dialogues in dark web chat groups for purchasing arms. An AI system combining geospatial predictive or place-based policing information and Automated Number Plate Registration (ANPR) information of vehicles belonging to the suspects supports the human assessment in the investigation based on verifiable and objective facts directly linked to a specific criminal activity.

- The use of an AI system that assesses the risk whether a prisoner should receive the benefit of an early release. The AI profile of the affected person or assessment of their personality traits and characteristics only support the human assessment of objective and verifiable facts related to past criminal offences and demonstrated behaviour relevant to rehabilitation.

- A judge conducts a pre-trial detention hearing for a person accused of a serious criminal offence to assess whether non-custodial measures can be applied. The decision is based on an assessment of the existence of valid grounds to imposed pre- trial detention, such as the likelihood that the suspect or accused person will commit another offence if not detained or that they will abscond or obstruct the proper conduct of the investigation. To assist in this process, the judge uses an AI risk assessment tool trained on data including past criminal history of individuals in similar cases, as well as factors such as age group, social behaviour, income, and employment status. - AI system is used to support the assessment of a human officer to assess the risk of an individual serving a non-custodial sentence violating release conditions or absconding based on past criminal behaviours and objective facts that give grounds for suspicion such as adherence to conditions of release, psychological assessment outcomes and recommendations from other community services the individual may be using. Based on this information, the officer decides whether to maintain the status quo or revise the conditions of release.

- AI systems used by custom authorities to assess the risk of goods entering the EU not complying with the legislation applicable at the border (e.g., which may include bans on import of illicit drugs, export sanctions contravention or other illegal activity) to identify situations where a customs control should be carried out. The AI system assesses objective and verifiable information provided to the customs related to the goods and their supply chains (e.g., nature and value of the goods, container number, means of transport for concealment of other goods, prior knowledge relating to the compliance of goods of the particular description and origin with requirements for their importation to or exportation from the Union). In certain cases, it may also process information about the prior involvement of the importer or exporter in irregularities related to import of goods, their affiliation to criminal organizations or a criminal record for drug trafficking. Such systems are out of scope of the prohibition because any prediction of a likelihood of a natural person to be involved in an import or export of illicit goods is not solely based on profiling, but on objective and verifiable information related to the goods and the importer or exporter’s prior involvement in criminal activity and subject to a human review to determine whether or not the situation requires a customs control or risk mitigation action. 5... AI systems used for crime predictions and assessments in relation to legal entities

The prohibition in Article 5(1)(d) AI Act applies only to individual predictions and risk assessments of natural persons, thus typically excluding crime predictive systems that profile legal entities such as companies or non-governmental organisations. For example, - A tax or customs authority is using an AI system to analyse large amounts of data on transactions and tax declarations and customs data of companies for assessing the risk of a company committing tax or customs fraud constituting a criminal offence. - AI systems used to assist customs authorities to help identify situations where an instruction not to send illicit goods to the EU should be issued to legal entities.

At the same time, there may be borderline cases where a natural person acts via a legal entity as a ‘sole trader’ or as an independent professional (e.g., a lawyer). In such circumstances, the prohibition in Article 5(1)(d) AI Act may apply provided that all conditions are fulfilled, since the AI system profiles a specific natural person and assesses or predicts their risk of committing a criminal offence, even if this is done for purposes in relation to the commercial activity undertaken by the natural person. 5... AI systems used for individual predictions of administrative offences

The prohibition in Article 5(1)(d) AI Act applies only for the prediction of criminal offences, thus excluding administrative offences from its scope, the prosecution of which is, in principle, less intrusive for the fundamental rights and freedoms of people. For example, a public authority using AI in the context of an administrative investigation to assess the risk of potential offenders of committing minor offences (such as petty traffic offences) or irregularities in tax, procurement or expenditure processes would not fall within the scope of the prohibition in Article 5(1)(d) AI Act, even in cases where information might be gathered for possible involvement of the natural persons in criminal offences as a result of the administrative investigations and checks.

Whether an offence is administrative or criminal in nature may depend on Union or national law. For offences that are not directly regulated by Union law, the national qualification of the offence is subject to scrutiny by CJEU since ‘criminal offence’ is a concept that has autonomous meaning within the EU law and should be interpreted consistently across Member States. The CJEU has concluded, in a different context, that the classification of the offences by the Member States is not conclusive in that respect. Relevant criteria used to assess the nature of the offence (criminal or not) may be found in relevant case-law of the CJEU and the European Court of Human Rights (ECtHR).

The interplay of the prohibition in Article 5(1)(d) AI Act with the LED and GDPR is relevant when assessing the lawfulness of personal data processing under Union data protection law, such as the GDPR and the LED. In particular, Article 5(1)(d) AI Act imposes a specific prohibition for law enforcement authorities, other public authorities and private entities falling within the scope of the prohibition to assess or predict the risk of a natural person committing a criminal offence, based solely on the profiling of a natural person or on assessing their personality traits and characteristics. With regard to LED, Article 5(1)(d) AI Act is without prejudice to Article 11(3) LED, which prohibits profiling resulting in (direct or indirect) discrimination.

The interplay of the prohibition in Article 5(1)(d) AI Act with Directive (EU) 2016/343 on the presumption of innocence is also relevant, since both acts are concerned - directly in the case of the Directive and indirectly in the case of the AI Act (see its Recital 42), with the fundamental right to be presumed innocent until proven guilty according to law. While the Directive applies from the moment that a person is suspected or accused of having committed a criminal offence, the AI Act has a broader scope of application and applies already at the stage of prediction and crime prevention before a formal criminal investigation is opened against a particular person and even in cases when such predictions and risk assessments are made by private actors falling within the scope of Article 5(1)d) AI Act and not by competent law enforcement authorities, including judicial authorities.

Even in cases where the prohibition in Article 5(1)(d) AI Act does not apply, it is important to emphasize that applicable Union and national law remains fully applicable, including in particular data protection, criminal procedural and police law and safeguards that may further restrict or impose additional conditions on the use of individual crime predictive AI systems.