What is Replika?
Replika is an application based on artificial intelligence developed by Luka Inc., which allows users to interact with a customizable chatbot via text or voice (an interaction that may involve extensive sharing of data, both personal and non-personal, potentially including sensitive data). The service presents itself as an “empathetic virtual companion,” which can be configured as a friend, romantic partner, mentor, or coach, with the aim of offering emotional support and encouraging personal reflection.
This type of simulated interaction, powered by advanced language models (LLMs) that learn from conversations, may lead some users to form an emotional bond with the chatbot. This raises legitimate concerns regarding privacy, ethics, and possible impacts on psychological well-being, especially for vulnerable individuals.
The Case and the Alleged Violations
In 2023, the Italian Data Protection Authority (Garante), following reports and preliminary investigations that also identified potential risks for minors and vulnerable individuals, imposed a provisional restriction on data processing concerning Replika. In the 2023 ruling, the Garante highlighted not only the lack of filters to prevent access by minors and the presence of inappropriate (and unsuitable) responses from the chatbot, but also serious shortcomings in the privacy notice, which was deemed opaque and lacking essential elements such as purposes, data retention periods, and legal bases for processing, with particular concern regarding underage users.
The investigation initiated in 2023 concluded in 2025 with a final decision dated April 10, which confirmed the initial concerns and found violations of the GDPR regulations.
In the April 10, 2025 decision, the Garante identified three categories of violations:
● Unlawful data processing (lack of clear indication of legal bases);
● Inadequate privacy notice (provided only in English and lacking clear information on data processing purposes and retention periods);
● Absence of adequate mechanisms to protect minors (including the lack of effective age verification systems).
For completeness, as also noted by the Garante, Luka Inc. attempted to make improvements between 2023 and 2025, implementing a new privacy policy, introducing age gate mechanisms and a feedback system to report inappropriate content.
However, according to the Garante, these measures remain insufficient. The system still allows users to change their date of birth after registration (“without any verification by the data controller”) and does not reliably prevent minors from accessing the service. Therefore, the Garante imposed an administrative fine of 5 million euros and ordered the company to comply by making corrective changes to the systems, privacy notices, and data processing methods.
Training of AI Systems
Let us now turn to another crucial point that emerges from the case and the Garante’s decision concerning the training of AI systems.
AI-based chatbots operate through language models that learn in multiple phases. In simple terms (without claiming completeness or technical detail), it all begins with a pre-training phase during which models read enormous amounts of text—books, articles, conversations—to learn the rules of language: how sentences are formed, the meaning of words, and their contextual usage. Next, they are fine-tuned for more specific tasks, such as adopting an empathetic tone, responding formally, or translating between languages. The final stage involves human input, where expert evaluators guide the AI regarding the preferred type of responses, thereby improving the quality of interactions.
A specific GDPR-related issue arises when real user conversations are used to improve these responses. Among the various concerns, the Garante found that Replika trains its model using fragments of user conversations.
According to the Garante, this activity was carried out without a clear privacy notice and without collecting consent.
The GDPR requires that the processing of personal data, such as for the purpose of training AI systems, must be based on an explicit legal basis and accompanied by a clear, comprehensible, and complete privacy notice. The lack of transparency on how data is (re)used—especially when involving minors or vulnerable individuals—represents a significant risk, both in terms of data protection and from an ethical and social standpoint.
Article in collaboration with AW LEGAL
AW LEGAL is a law firm specializing in Intellectual Property, Privacy, and Legal Tech.
