Article 5(1)(c) AI Act - social scoring

AI systems enabling ‘social scoring’ practices may lead to discriminatory and unfair outcomes for certain individuals and groups, including their exclusion from society, as well as social control and surveillance practices that are incompatible with Union values. The prohibition of ‘social scoring’ aims to protect, in particular, the right to human dignity and other fundamental rights, including the right to non-discrimination and equality, to data protection, and to private and family life, as well as relevant social and economic rights, as applicable. It also aims to safeguard and promote the Union values of democracy, equality (including equal access to public and private services), and justice.

Several cumulative conditions must be fulfilled for the prohibition in Article 5(1)(c) AI Act to apply:

(i) The practice must constitute the ‘placing on the market’, the ‘putting into service’ or the ‘use’ of an AI system.

(ii) The AI system must be intended or used for the evaluation or classification of natural persons or groups of persons over a certain period of time based on: (a) their social behaviour; or (b) known, inferred or predicted personal or personality characteristics. (iii)The social score created with the assistance of the AI system must lead or be capable of leading to the detrimental or unfavourable treatment of persons or groups in one or more of the following scenarios:

(a) in social contexts unrelated to those in which the data was originally generated or collected; and/or (b) treatment that is unjustified or disproportionate to their social behaviour or its gravity.

For the prohibition in Article 5(1)(c) AI Act to apply, all three conditions must be simultaneously fulfilled. The first condition, i.e. the placing on the market, the putting into service or the use of the AI system, has been already analysed in section 2.. The prohibition therefore applies to both providers and deployers of AI systems, each within their respective responsibilities, not to place on the market, put into service or use such AI systems. The remaining criteria for the prohibition on ‘social scoring’ are further described and analysed below.

4... ‘Social scoring’: evaluation or classification based on social behaviour or personal or personality characteristics over a certain period of time a) Evaluation or classification of natural persons or group of persons

The second condition for the prohibition in Article 5(1)(c) AI Act to apply is that the AI system is intended or used for the evaluation or classification of natural persons or groups of persons and assigns them scores based on their social behaviour or their personal or personality characteristics. The score produced by the system may take various forms, such as a mathematical number (for example, from 0 to 1), a ranking, or a label.

The scope of the prohibition is broad covering evaluation and classification practices in both the public and the private sector (see section 4...). At the same time the evaluation or classification concerns only natural persons or groups of natural persons, thereby excluding in principle legal entities (see section 4.. out of scope).

While ‘evaluation’ suggests the involvement of some form of an assessment or judgement about a person or group of persons, a simple classification of persons or groups of persons based on characteristics, such as their age, sex, and height, need not necessarily lead to evaluation. The scope of ‘classification’ is therefore broader than ‘evaluation’ and can also cover other types of classifications or categorisations of natural persons or groups of persons based on criteria that do not necessarily involve a particular assessment or judgement about those persons or groups of persons and their characteristics or behaviour.

The term ‘evaluation’ also relates to the concept of ‘profiling’, which is regulated by Union data protection legislation and constitutes a specific form of evaluation. While no direct reference is made in Article 5(1)(c) AI Act to that concept or that legislation, they may also be relevant for the prohibition contained in that provision, as well as for other prohibitions in the AI Act, when the evaluation occurs in an automated fashion by an AI system based on personal data. Profiling means the use of information about an individual (or group of individuals) and evaluating their characteristics or behaviour patterns in order to place them into a certain category or group, in particular to analyse and/or make predictions about, for example, their ability to perform a task; interests; or likely behaviour’. Profiling of natural persons under EU data protection law, when conducted through AI systems, may therefore also be covered by Article 5(1)(c) AI Act. For example, in the SCHUFA I judgment, the CJEU examined a creditworthiness scoring system used in Germany. In that case, the ‘score’ generated by the computer programme was a ‘probability value’ concerning the ability of a person to meet payment commitments, which was qualified by the CJEU as ‘profiling’. More specifically, the system established ‘a prognosis on the probability of a future behaviour of a person (‘score’), such as the repayment of a loan, based on certain characteristics of that person. The establishment of scores (‘scoring’) is based on the assumption that, by assigning a person to a group of other persons with comparable characteristics who have behaved in a certain way, similar behaviour can be predicted. According to the CJEU, this activity met the definition of ‘profiling’ within the meaning of Article 4(4) GDPR. This form of profiling may also be considered to constitute an evaluation of persons based on their personal characteristics within the meaning of Article 5(1)(c) AI Act, which will be prohibited if done with AI systems and provided that the other conditions for the application of that provision are fulfilled.

b) Over a certain period of time

The prohibition in Article 5(1)(c) AI Act requires that the evaluation or classification is based on data that spans over ‘a certain period of time’. This suggests that the assessment should not be limited to a one-time or an at once rating or grading with data or behaviour from a very specific individual context. At the same time, it is important that this condition is assessed, taking all circumstances of the case into account to avoid circumvention of the scope of the prohibition.

For example, an authority for migration and asylum implements a partly automated surveillance system at refugee camps built on a range of surveillance infrastructure, including cameras and motion sensors. If the analysed data spans a period of time and specific individuals are evaluated (such as migrants) for example to ascertain whether they are at risk of trying to abscond, then this would qualify as ‘over a certain period of time’ and the prohibition in Article 5(1)(c) AI Act may apply if all other conditions are fulfilled.

c) Based on their social behaviour or known, inferred or predicted personal or personality characteristics

The practices of ‘evaluation’ and ‘classification’ prohibited under Article 5(1)(c) AI Act must be based on the AI-enabled processing of data (often extensive) in relation to either i) the social behaviour of individuals or groups of persons or ii) their known, inferred or predicted personal and personality characteristics, or both. The data may be directly provided by the persons or indirectly collected, i.e. through surveillance, obtained from third parties or through inferences from other information.

Regarding the first scenario, ‘social behaviour’ is a broad term that can generally include actions, behaviour, habits, interactions within society, etc., and usually covers behaviour related data points from multiple sources. This could include behaviour of individuals and groups of individuals in social and private contexts, such as participation in cultural events, volunteering, etc., but also social behaviour in business contexts, for example the payment of debts, behaviour when using certain services, as well as relations with public and private entities, government, police, and the law (for example, whether a person obeys traffic rules). Social behaviour data from multiple contexts and data points may be collected in a centralised way by the same entity, but is most often collected in a distributed way and combined from different sources, which may involve increased monitoring and the tracking of individuals (so called ‘dataveillance’).

The second scenario is where the scoring is based on personal or personality characteristics, which may or may not involve specific social behavioural aspects. ‘Personal characteristics’ may include a variety of information relating to a person, for example sex, sexual orientation or sexual characteristics, gender, gender identity, race, ethnicity, family situation, address, income, household members, profession, employment or other legal status, performance at work, economic situation, financial liquidity, health, personal preferences, interests, reliability, behaviour, location or movement, level of debt, type of car etc. ‘Personality characteristics’ should be in principle interpreted as synonymous with personal characteristics, but may also imply the creation of specific profiles of individuals as personalities. Personality characteristics may be also based on a number of factors and imply a judgement, which may be made by the individuals themselves, other persons, or generated by AI systems. In the AI Act, personality characteristics are sometimes referred to as personality traits and characteristics; those concepts should be interpreted consistently.

‘Known, inferred or predicted’ personal or personality characteristics are different types of information and personal data that need to be distinguished. ‘Known characteristics’ are based on information which has been provided to the AI system as an input, and which is in most cases verifiable information. By contrast, ‘inferred characteristics’ are based on information which has been inferred from other information, with the inference usually made by an AI system. ‘Predicted characteristics’ are those which are estimated based on patterns with less than 100% accuracy. The concepts of ‘inferred’ (or derived) data are also used in the context of profiling in Union data protection law and may therefore be a source of inspiration for interpreting those concepts used in Article 5(1)(c) AI Act. The use of these different types of data may have different implications for the accuracy and the fairness of the scoring practices and therefore may be taken into account, in particular where the processing is opaque or relies on data points whose accuracy is more difficult to be verified.

4... The social score must lead to detrimental or unfavourable treatment in unrelated social contexts and/or unjustified or disproportionate treatment to the gravity of the social behaviour a) Causal link between the social score and the treatment

For the prohibition in Article 5(1)(c) AI Act to apply, the social score created by or with the assistance of an AI system must lead to a detrimental or unfavourable treatment for the evaluated person or group of persons. In other words, the treatment must be the consequence of the score, and the score the cause of the treatment. Such a plausible causal link may also exist in cases where the harmful consequences have not yet materialised but the AI system is intended to or capable of producing such an adverse outcome. This is particularly relevant given that the prohibited practice in Article 5(1)(c) AI Act also covers the ‘placing on the market’ of such AI systems.

Article 5(1)(c) AI Act does not require the evaluation or classification performed by the AI system to be the sole cause of the detrimental or unfavourable treatment. It therefore also covers AI-enabled scoring practices that may also be subject to or combined with other human assessments. At the same time, the AI output must play a sufficiently important role in producing the social score. For example, in the case where a public authority deploys an AI system to assess the trustworthiness of persons and combines its output with a human assessment of additional facts, this AI-enabled social scoring practice will fall within the scope of the prohibition only if the AI-generated score plays a sufficiently important role in the final decision, provided the other conditions for detrimental or unfavourable treatment are fulfilled as described below (see section 4... b).

A score may lead to detrimental or unfavourable treatment even if it is produced by an organisation(s) different from the one that uses the score. For example, a public authority may obtain a score for a natural person’s creditworthiness assessment produced by another company specialised in creditworthiness and risk assessments, which are based on information about the individuals and their behaviour from a variety of sources.

b) Detrimental or unfavourable treatment in unrelated social contexts and/or unjustified or disproportionate treatment

The final condition for the prohibition in Article 5(1)(c) AI Act to apply is that the use of the social score must result (or be capable of resulting) in detrimental or unfavourable treatment either:

i. in social context(s) unrelated to the contexts in which the data was originally generated or collected, or ii. unjustified or disproportionate to the social behaviour or its gravity.

These conditions are alternative and may apply also in combination. An analysis on a case-by-case basis is necessary to assess if at least one of them is fulfilled, since many AI-enabled scoring and evaluation practices may not fulfil them and therefore be outside the scope of the prohibition. In particular, this may not be the case where the AI-enabled scoring practices are for a specific legitimate evaluation purpose and comply with applicable Union and national laws that specify the data considered as relevant for the purposes of evaluation and ensure that the detrimental or unfavourable treatment is justified and proportionate to the social behaviour (see section 4.. out of scope).

‘Unfavourable treatment’ means that as a result from the scoring, the person or group of persons must be treated less favourably compared to others without necessarily requiring a particular harm or damage (for example, in the case of scoring practices where people are singled out for additional inspections in case of fraud suspicious). By contrast, ‘detrimental’ treatment requires the person or group of persons to suffer certain harm and detriment from the treatment. Unfavourable or detrimental treatment may also be discriminatory and prohibited under EU non-discrimination law or imply the exclusion of certain individuals or groups, but that is not a necessary condition for the prohibition to apply. Article 5(1)(c) AI Act could therefore cover unfair treatment beyond EU non-discrimination law that applies only to certain protected groups (e.g., age, ethnic and racial origin, sex, religion).

Scenario 1: Detrimental or unfavourable treatment in unrelated social contexts

In the first scenario described under Article 5(1)(c)(i) AI Act, the detrimental or unfavourable treatment resulting from the score must take place in social context(s) unrelated to the contexts in which the data was originally generated or collected. This implies not only that the persons may be treated in an unfavourable or detrimental manner due to the social score, but also that the data about their social behaviour or their known, inferred or predicted personal or personality characteristics are generated or collected in social contexts unrelated to the one in which the scoring takes place. The data collected or generated from these unrelated contexts must be subsequently used by the AI system for the scoring of the persons without an apparent connection for the purpose of the evaluation or classification or in a manner that leads to generalised surveillance of the persons or the groups of persons. In most cases, this happens against the reasonable expectations of the persons and in violation of Union data protection law and possibly other applicable rules that specify the types of data and sources considered relevant and necessary for the evaluation or classification. Whether this condition is fulfilled will require a case-by-case assessment, taking into account the purpose of the evaluation and the contexts from which the data has been collected and generated. Examples of detrimental or unfavourable treatment in unrelated social contexts prohibited under Article 5(1)(c)(i) AI Act - National tax authorities use an AI predictive tool on all taxpayers’ tax returns in a country to select tax returns for closer inspection. The AI tool uses relevant variables, such as yearly income, assets (real estate property, cars etc.), data on family members of beneficiaries, but also unrelated data, such as taxpayers’ social habits or internet connections, to single out specific individuals for inspections. - A social welfare agency uses an AI system to estimate the probability of fraud by beneficiaries of household allowances that relies on characteristics collected or inferred from social contexts with no apparent connection or relevance for the assessment of fraud, such as having a spouse of a certain nationality or ethnic origin, having an internet connection, behaviour on social platforms, or performance at the workplace, etc. By contrast, data that is relevant for the allocation of the benefits and lawfully collected could be used to determine the risk of fraud, since public authorities pursue a legitimate aim in verifying if social benefits are correctly allocated.

- A public labour agency uses an AI system to score unemployed individuals based on an interview and an AI-based assessment for determining whether an individual should benefit from state support for employment. That score is based on relevant personal characteristics, such as age and education, but also variables collected or inferred from data and contexts with no apparent connection to the purpose of evaluation, such as marital status, health data for chronic diseases, addiction, etc. These unacceptable scoring practices may be distinguished from lawful practices that evaluate persons for specific purpose in compliance with Union and national law, in particular when such laws, in compliance with EU law, specify the data considered as relevant and necessary for the purposes of evaluation (see section 4.. out of scope). Scenario 2: Unfavourable or detrimental treatment disproportionate to the social behaviour

Another alternative scenario under Article 5(1)(c)(ii) AI Act where an AI scoring system may be prohibited is if the treatment resulting from the score is unjustified or disproportionate to the gravity of the social behaviour. The severity of the impact and the interference with the fundamental rights of the person concerned resulting from the social scoring compared to the gravity of the social behaviour of the person should determine whether such treatment is disproportionate for the legitimate aim pursued, taking into account the general principle of proportionality. This requires a case-by- case assessment, which should consider all relevant circumstances of the case, as well as general ethical considerations and principles for fairness and social justice related to the assessment of the social behaviour and the proportionality of the detrimental treatment. The treatment may also be ‘unjustified’, such as lacking a legitimate aim. Sectoral Union or national legislation setting specific criteria and procedures that regulate such potential detrimental or unfavourable treatment may also be relevant as part of this assessment.

Examples of unjustified or disproportionate treatment compared to the social behaviour prohibited under Article 5(1)(c) ii) AI Act - A public agency uses an AI system to profile families for early detection of children at risk based on criteria such as parental mental health and unemployment, but also information on parents’ social behaviour derived from multiple contexts. Based on the resulting score, families are singled out for inspection and children considered ‘at risk’ are taken from their families, including in cases of minor transgressions by the parents, such as occasionally missing doctors’ appointments or receiving traffic fines. - A municipality uses an AI system to score trustworthiness of residents based on multiple data points related to their social behaviour in a variety of contexts. The generated score for residents considered ‘less trustworthy’ is used for blacklisting, i.e. withdrawal of public benefits, other serious punitive measures, and increased control or surveillance. Among the factors considered in the assessment are insufficient volunteering and minor misbehaviour, such as not returning books to the library on time, leaving rubbish on the street outside the day of collection, and a delay in the payment of local taxes.

These unacceptable social scoring practices may be distinguished from lawful practices that evaluate persons for a legitimate specific purpose in compliance with Union and national law, in particular where those laws ensure that detrimental or unfavourable treatment is justified and proportionate to the social behaviour (see section 4.. out of scope).

Both alternatives under Article 5(1)(c)(i) and (ii) AI Act may also be fulfilled simultaneously.

Examples of unjustified or disproportionate treatment under Article 5(1)(c)) i) and ii) AI Act - A tax authority uses an AI system to detect child benefit fraud by profiling and assigning beneficiaries suspected of fraud to categories such as ‘deliberate intent/gross negligence’ using criteria such as low income, dual nationality, social behaviour, etc. Based on the risk score, a beneficiary’s file is inspected and, in many cases, their childcare benefit ceased, they receive notice to repay the received benefits, and no longer qualify for standard debt collection arrangements. Such scoring causes many families to be heavily indebted and leads to unjust, discriminatory and detrimental treatment of individuals and groups of individuals, driving many families into severe financial hardship.

- A public authority uses an AI system to control fraud in the student housing grant process that considers among the indicators the internet connections, family status or level of education of beneficiaries as distinguishing factors for fraud risk, which do not seem relevant, nor justified.

- A government introduces a comprehensive AI-based system that monitors and rates citizens based on their behaviour in various aspects of life, such as social interactions, online activities, purchasing habits, and punctuality in paying bills. People with a lower score face restricted access to public services, higher interest rates on loans, and difficulty in traveling, renting apartments, and even finding jobs. The system leads to excessive surveillance of individuals and detrimental treatment in contexts unrelated to the social behaviour used to determine the social score (e.g., job opportunities are influenced by social media activity), while also imposing excessive penalties for minor infractions (e.g., social and financial disadvantages for relatively minor offences).

These unacceptable social scoring practices may be distinguished from lawful practices evaluating persons for legitimate specific purposes that do not fulfil these conditions and are in compliance with Union and national law, in particular when those laws ensure that detrimental or unfavourable treatment is justified and proportionate and data from related social contexts is used (see section 4.. out of scope).

The prohibition under Article 5(1)(c) AI Act may also cover cases where awards or preferential treatment are given to certain individuals or groups of persons, since this implies less favourable treatment of other individuals (e.g., in cases of support employment programmes, (de-)prioritisation for housing or resettlement). 4... Regardless of whether provided or used by public or private persons

As already noted, Article 5(1)(c) AI Act prohibits unacceptable AI-enabled social scoring practices regardless of whether the AI system or the score are provided or used by public or private persons. While scoring in the public sector may have very significant consequences for people due to an imbalance of power and a dependence on public services, similarly harmful consequences may also occur in the private sector, where scoring practices are also increasingly undertaken by companies and other entities.

For example, - An insurance company collects spending and other financial information from a bank which is unrelated to the determination of eligibility of candidates for life insurance and which is used to determine the price of the premium to be paid for such insurance. An AI system analyses this information and recommends, on that basis, whether to refuse a contract or set higher life insurance premiums for a particular individual or a group of customers.

- A private credit agency uses an AI system to determine the creditworthiness of people and decide whether an individual should obtain a loan for housing based on unrelated personal characteristics.

These unacceptable social scoring practices may be distinguished from lawful practices evaluating persons for specific legitimate purposes that do not fulfil these conditions and are in compliance with Union and national law, in particular when those laws ensure that detrimental or unfavourable treatment is justified and proportionate and data from related social contexts is used (see section 4.. out of scope).

In the case of checks by competent market surveillance authorities, it is on the provider and the deployer of the AI system, each within their responsibilities, to demonstrate that the AI practice is legitimate and justified, including by providing transparency of the functioning of the AI system, and information about the types of data and data sources, ensuring that only data related to the social context in which the score is used are processed for the purpose of the evaluation or classification, the system is performing as intended, and any resulting detrimental or unfavourable treatment is justified and proportionate to the social behaviour. Compliance with applicable legislation and appropriate and proportionate safeguards built in the system and applied during its operation will help to avoid the prohibition from applying, while enabling the use of AI systems for the evaluation or classification of persons for legitimate and beneficial purposes (e.g., to improve the effectiveness of processes, quality of service, safety, etc.) (see section 4.. out of scope).

Compliance with the requirements for high-risk AI systems (e.g. in the area of essential public services and benefits, credit-scoring and creditworthiness assessment, migration etc.) may also help to ensure that AI systems used for evaluation and classification purposes in those high-risk areas do not constitute unacceptable social scoring practices that providers and deployers should consider when implementing their respective obligations (e.g. on risk management, transparency, data governance, fundamental rights impact assessment, human oversight, monitoring, etc.).

The prohibition in Article 5(1)(c) AI Act only applies to the scoring of natural persons or groups of persons, thus excluding in principle scoring of legal entities where the evaluation is not based on personal or personality characteristics or social behaviour of individuals, even if in some cases individuals may be indirectly impacted by the score (e.g., all citizens in a municipality in case of allocation of budget). However, if legal entities have been evaluated based on an overall score that aggregates the evaluation or classification of a group of natural persons based on their social behaviour or personal or personality characteristics and this score directly affects those persons (e.g. all employees in a company, students in a specific school whose behaviour has been evaluated), the practice may fall within the scope of Article 5(1)(c) AI Act if all other conditions are fulfilled. This will depend on a case-by-case assessment.

AI-based social scoring as a ‘probabilistic value’ and prognosis should also be distinguished from individual ratings by users which assess the quality of a service (such as a driver in an online car-sharing platform or a host in an online platform for accommodation). Such ratings are the mere aggregation of individual human scores that do not necessarily involve AI, unless the data are combined with other information and analysed by the AI system for evaluating or classifying individuals fulfilling all conditions in Article 5(1)(c) AI Act.

Furthermore, the scoring of natural persons is not at all times prohibited, but only in the limited cases where all of the conditions of Article 5(1)(c) AI Act are cumulatively fulfilled, as analysed above. Recital 31 AI Act, in particular mentions that the prohibition ‘should not affect lawful evaluation practices of natural persons that are carried out for a specific purpose in accordance with Union and national law’. For example, credit scoring and risk scoring and underwriting are essential aspects of the services of financial and insurance businesses. Such practices, as well as other legitimate practices (i.e. to improve the quality and efficiency of services, to ensure more efficient claims handling, to perform specific employee evaluations, fraud prevention and detection, law enforcement or scoring of users’ behaviour on online platforms), are not per se prohibited, if lawful and undertaken in line with the AI Act and other applicable Union law and national law, which must comply with Union law.

In other words, AI systems which evaluate or classify individuals for the purposes of generating a social score in a lawful manner and for a specific purpose in the related context as that in which the personal data used for the score were collected are not prohibited, provided that any detrimental or unfavourable treatment from using the score is justified and proportionate to the gravity of the social behaviour.

Compliance with sectoral Union legislation, such as in the field of credit-scoring, anti- money laundering, etc., that specifies the type of data that can be used as relevant and necessary for the specific legitimate purpose of evaluation and ensures that the treatment is justified and proportionate to the social behaviour may thus ensure that the AI practice falls outside the scope of the prohibition in Article 5(1)(c) AI Act. Examples of legitimate scoring practices in line with Union and national law that are outside the scope of Article 5(1)(c) AI Act:

- Financial credit scoring systems used by creditors or credit information agencies to assess a customer’s financial creditworthiness or outstanding debts, providing a credit score or determining their creditworthiness assessment, which are based on the customer’s income and expenses and other financial and economic circumstances, are out of scope of Article 5(1)(c) AI Act if they are relevant for the legitimate purpose of the credit scoring and if they comply with consumer protection laws specifying the type of data and the necessary safeguards to ensure the fair treatment of consumers in creditworthiness assessments.

- Companies have a legitimate interest to evaluate customers for financial fraud and those practices are not affected by the prohibition, if the evaluation is based on relevant data such as transactional behaviour and metadata in the context of the services, past history and other factors from sources that are objectively relevant to determine the risk of fraud and if the detrimental treatment is justified and proportionate as a consequence of the fraudulent behaviour.

- Information collected through telematic devices that show that a driver is speeding or not maintaining safe driving practices used by an insurer that offers telematics-based tariffs in relation to a policyholder’s high-risk driving behaviour may be used to increase the premium of that policyholder due to the higher risk of an accident caused by that driving behaviour, provided the increase in the premium is proportionate to the risky behaviour of the driver.

- The collection and processing of data that is relevant and necessary for the intended legitimate purpose of the AI systems (e.g. health and schizophrenic data collected from various sources to diagnose patients) is out of scope of Article 5(1)(c) AI Act, in particular because it process relevant and necessary data and typically does not entail unjustified detrimental or unfavourable treatment of certain natural persons. - Online platforms profiling users for safety reasons on their services based on data which is relevant for the context and purpose of assessment is out of scope of Article 5(1)(c) AI Act, when the evaluation does not result in detrimental treatment that is disproportionate to the gravity of the user’s misbehaviour.

- AI-enabled targeted commercial advertising is out of scope, where it is based on relevant data (e.g., users’ preferences), it is done in line with Union law on consumer protection, data protection and digital services, and it does not result in detrimental or unfavourable treatment disproportionate to the gravity of the user’s social behaviour (e.g., exploitative and unfair differential pricing).

- AI systems using data collected in refugee camps (e.g., behavioural compliance) for decisions about resettlement or employment is not affected by the prohibition, given that this data is relevant for the purpose of assessment and provided that the procedures under applicable Union migration law are fulfilled to ensure the treatment is justified and proportionate.

- AI-enabled scoring by an online shopping platform which offers privileges to users with a strong purchase history and a low rate of product returns, such as a faster returns application process or returnless refunds are out of scope of Article 5(1)(c) AI Act, given that the advantages are justified and proportionate to reward positive behaviour and other users continue to have access to the standard return process. - AI-evaluation and scoring of individuals by police and other law enforcement authorities that collect data about individuals’ social behaviour from multiple contexts are out of scope of Article 5(1)(c) AI Act where those data are relevant for the specific purposes of the prevention, detection, prosecution and punishment of criminal offences, and where the detrimental treatment is justified and proportionate in accordance with substantive and procedural Union and national criminal and police law. It is also relevant to consider in this context the prohibition in Article 5(1)(d) AI Act, which imposes additional and more specific conditions for AI-enabled risk assessments and predictions of the likelihood of a person committing a criminal offence which must not be solely based on profiling or the assessment of personality traits (see section 5).

Providers and deployers should carefully assess whether other applicable Union and national legislation applies to any particular AI scoring system used in their activities, in particular if there is more specific legislation that strictly regulates the types of data that can be used as relevant and necessary for specific evaluation purposes and if there are more specific rules and procedures to ensure justified and fair treatment.

AI-enabled social scoring practices by private parties acting as traders in business-to- consumer relations may also be in breach of Union consumer protection law, i.e. Directive 2005/29/EC on unfair business-to-consumer commercial practices (the ‘UCPD’). The UCPD prohibits commercial practices if they are contrary to the requirements of professional diligence and materially distort or are likely to materially distort the economic behaviour of the average consumer or average member of the group with regard to the product (Article 5 UCPD). The scoring practices may also be found misleading (Articles 6-7 UCPD) subject to case-by-case assessment of the impact of the commercial practice on the consumer’s transactional decision.

Social scoring, whether by public or by private parties, may also be in breach of Union data protection laws, for example as regards the legal ground for processing (lawfulness), the data protection principles (e.g., data minimisation and necessity, fairness, transparency), and any other obligations, including the rules on solely automated individual decision-making, where relevant.

Where the evaluation or classification is based on one of the grounds protected from discrimination (e.g., age, religion, racial or ethnic origin, sex etc.) or results directly or indirectly in discrimination of those groups, such a practice will also be subject to Union non-discrimination law.

The Consumer Credit Directive (EU) 2023/2225127 may also be relevant in this context. Article 18(3) CCD requires that the assessment of creditworthiness is carried out based on relevant and accurate information on the consumer’s income and expenses and other financial and economic circumstances which is necessary and proportionate to the nature, duration, value and risks of the credit for the consumer. That information may include evidence of income or other sources of repayment, information on financial assets and liabilities, or information on other financial commitments. The CCD explicitly prohibits special categories of personal data to be included in the information and to obtain information from social networks. The European Banking Authority’s Guidelines on loan origination and monitoring further specify the relevant information for the purpose of creditworthiness assessments. This specification of the type of data in these sectoral laws for specific evaluation purposes are relevant considerations to be taken into account when determining whether a practice falls within the scope of the prohibition in Article 5(1)(c) AI Act.

Similarly, AI systems used for the evaluation and classification of persons for anti- money laundering and terrorism financing purposes should also comply with relevant Union legislation on these matters.