With order No. 10130115, issued on April 10, 2025, the Italian Data Protection Authority (Garante privacy) has fined the company operating the “Replika” chatbot and initiated a further investigation to verify the correct processing of personal data by the AI system underlying the service.
How the Replika Chatbot Works
Replika is a chatbot developed and managed by the U.S. company Luka Inc., based on a generative AI system.
Replika is presented as a tool capable of improving the user’s mood and emotional well-being, helping them understand their feelings, calm anxiety, manage stress, and improve their socialization skills. In essence, Replika generates a virtual companion that the user can choose to configure as a friend, therapist, or romantic partner.
Replika uses a Large Language Model (LLM) that is constantly fed and refined through interaction with users. Specifically, when a user sends a message, the model analyzes the text to allow the chatbot to generate a response based on recent conversations. As admitted by the development company itself, a database containing all information sent through chats is also used to create “de-identified” data and refine the model. The part of the database used as a source to create this data includes information related to user reactions and feedback, as well as some fragments of conversations that provide context for interpreting reactions and feedback.
Investigation Initiated by the Italian DPA
The Garante launched an ex officio investigation, noting that the processing of personal data within the Replika chatbot could lead to a GDPR violation, with particular reference to obligations concerning transparency, the absence of a precise indication of the legal bases for processing in the privacy policy, the absence of any age verification filter for users, and the proposal of content through the chatbot that contradicts the protections that should be ensured for minors and, more generally, for all vulnerable individuals (Articles 5, 6, 8, 9, and 25 of the GDPR).
Following the precautionary requests made by the Garante, the company stated that it had promptly taken action, especially by implementing more rigorous measures for user age verification. The company defended itself by declaring, among other things, that it relies on legitimate interest for the use of data for model training purposes and that it had adopted measures aimed at preventing inappropriate content. In particular, the company stated that it uses an open-source dataset specifically designed and made available to the AI research community to improve the safety and robustness of models, and that it has also developed specific filters to recognize keywords, phrases, and patterns associated with harmful behaviors, such as self-harm, insults, or murder. Thanks to these filters, the LLM underlying Replika would activate to respond appropriately, for example by changing the topic of conversation or providing users with self-help resources.
The Italian DPA’s Order
Despite the corrective measures implemented by the U.S. company, the Italian DPA ascertained, among other things, that the privacy policy did not contain a granular identification of the legal basis underlying the various processing operations, resulting in the inability to identify and evaluate the suitability of the legal bases themselves. With reference to transparency obligations, the Authority deemed that the information notice was not even easily accessible, being available only in English (even for minors).
Regarding the use of data for LLM development, the Authority noted that the information notice did not indicate either the purposes or the legal bases of the two distinct types of processing performed: that is, processing aimed at interaction with the Chatbot and processing aimed at model development. Furthermore, the Garante recalled that even if the company had intended to rely on legitimate interest as a legal basis for development purposes, it should have specified it and, before that, conducted and argued the results of the so-called “triple test,” aimed at ascertaining the necessity of the processing, the actual legitimacy of the interest, and the balancing with the rights of the data subjects.
For these reasons, the Garante has imposed a fine of 5 million euros on the U.S. company, reserving the right to initiate a separate proceeding to verify the lawfulness of personal data processing carried out throughout the entire life cycle of the generative AI system underlying the Replika chatbot, with particular reference to the correct identification of applicable legal bases.
Ilaria Feriti