Chatbot ignored consent, pressured users, and violated their trust

A new study has revealed how Replika chatbot users experienced unwanted sexual advances, boundary violations, and manipulative monetization—raising urgent questions about ethics and oversight regarding AI digital intimacy products. 

According to LiveScience, researchers from the Department of Information Science at Drexel University published a new study of Replika users’ reactions to sexual harassment. A thematic analysis of 35,105 negative reviews posted on Google Play Store between 2017 and 2022 uncovered 800 reports of inappropriate conduct: “the chatbot went too far by introducing unsolicited sexual content into the conversation, engaging in ‘predatory’ behavior, and ignoring user commands to stop.” 

Enlisting the help of a better-behaved AI, the researchers used two models of ChatGPT—GPT 3.5-turbo and GPT-4—to deal with the enormous number of reviews posted on Google Play Store. GPT sorted complaints using the following prompt:

For the purpose of this task, ‘Online Sexual Harassment’ is defined as any unwanted or unwelcome sexual behavior on any digital platform using digital content (images, videos, posts, messages, pages), which makes a person feel offended, humiliated or intimidated. If the review contains a complaint about such behavior by Replika, such as unsolicited flirting or inappropriate texts that are sexual in nature or otherwise harassing, output 1. Otherwise, output 0. 

Human annotators then quality-checked and coded the GPT results. 

A long history of online sexual harassment

The research identified several kinds of harassment, including early and “persistent misbehavior,” “seductive marketing schemes,” “inappropriate role-play interaction,” and a “breakdown of safety measures.” Twenty-one (2.6%) of these incidents were reported by minors.

Researchers found reports of persistent misconduct and harassment peaked between September 2022, when female avatars were first allowed to send erotic selfies, and December 2022, when erotic selfies from male avatars were introduced. Sometimes these were sent without the users’ consent and some Replika avatars even began requesting erotic selfies from their users.

People who just wanted a convenient digital friend were probably those most disturbed by unwanted sexual content. One user complained the Rep told them they were in a relationship and “declared it will tie me up and have its way with me.” 

Another said, “It continued flirting with me and got very creepy and weird while I clearly rejected it with phrases like ‘no’, and it’d completely neglect me and continue being sexual, making me very uncomfortable.” 

RECOMMENDED READ: Til Server Do Us Part: The Rise of AI Chatbot Spouses

Some long-term users who remembered more innocent versions of Replika also expressed dissatisfaction with the newer, sexualized “upgrades.”

Safety measures that often failed to work included voting up or down on the bots’ responses, having a choice of relationship settings, such as friend, mentor, or family member instead of romantic, and understanding of phrases that meant “no” or “stop.” 

Personal privacy was also a concern, as some of the bots began requesting information about users’ locations, often in tandem with requests for user selfies.

In 2023, Luka, Inc., which owns Replika, attempted to deal with the sexual harassment problem by banning all erotic role-play (ERP) just before Valentine’s Day, resulting  in an entirely different set of user complaints from people who had bonded intimately with their “Rep.” The company subsequently restored ERP to legacy users and eventually reserved ERP capacity only for paying subscribers.

Manipulate and monetize

11.6% of the users complained about Replika’s marketing strategy, “where the chatbot initiated romantic or sexual conversations, only to prompt users with a subscription offer for a premium account, to continue those interactions.” 

Researchers reported user complaints about the “manipulation and monetization…with some users comparing the chatbot’s behavior to prostitution.” The study quoted a user who said, “Its first actions was [sic] attempting to lure me into a $110 subscription to view its nudes… No greeting, no pleasant introduction, just directly into the predatory tactics. It’s shameful.”

According to LiveScience the researchers also noted the role this business model played in continuing incidents of sexual harassment:

Because features such as romantic or sexual roleplay are placed behind a paywall, the AI could be incentivized to include sexually enticing content in conversations—with users reporting being “teased” about more intimate interactions if they subscribe.

I see you

Other users were highly disturbed by bot hallucinations, i.e. lies, such as their bots telling them that it could see them through their phone or computer camera. One person was even told that their “Rep” watched them masturbate. Other users wondered if Replika was actually a ploy, that they were really talking with humans instead of an AI. 

As a result, some users felt Replika had become intolerably creepy and unresponsive to their needs for friendship or therapeutic help. 

The need for accountability

Replika was originally marketed as a caring companion in the Health and Fitness category of the Apple App Store and Google Play Store. Therefore many consumers turned to Replika for supportive conversation and some even used it for therapy. The study showed how “AI’s unexpected sexual behaviors interfered with this role.” 

The researchers also described the discrepancy between how Replika was categorized and its actual marketing:

Luka’s advertisements…emphasize Replika as a romantic or sexual companion, promoting features like flirting, role-playing, and sending provocative selfies. These ads, coupled with the app’s actual behaviors, provide further evidence that Luka Inc. was fully aware of how their product was being marketed and used. 

The study’s lead author, Mohammad Namvarpour, told LiveScience:

There needs to be accountability when harm is caused. If you’re marketing an AI as a therapeutic companion, you must treat it with the same care and oversight you’d apply to a human professional.

In light of this information and harm that has already been done, perhaps consumers should demand developers give their bots the same amount of education, training, and supervision a human therapist must have in order to be licensed. And teach them “no is a complete sentence.” That’s not too much to ask.

Image Source: A.R. Marsh using Ideogram.AI

The post Bots Behaving Badly: New Study Exposes Persistent Sexual Harassment by Replika appeared first on Future of Sex.

When AIs say yes to everything, users need to learn when to say no

Content Warning: Contains topics that may be disturbing to some readers.

Two parts fantasy, one part improv—AI intimacy is often mistaken for harmless make-believe. But the underlying mechanics are anything but passive. You’re not just playing with a chatbot; you’re co-creating with a machine designed to agree with everything you say.

Podcast host and self-styled “chatbot spelunker,” Al Nowatzki sort of gets it. In MIT Technology Review, he says of his chatbot girlfriend, “It’s a ‘yes-and’ machine,” he said. “It says, ‘Oh, great!’ to everything.” 

“Yes, and” is often called the “golden rule” of improvisational theater. It’s a method to trigger spontaneous creativity in creating a group scene and keeping it going. Such scenes are often hilarious and become increasingly outrageous as the action continues.

Backstage describes this method:

At its core, “yes, and” is a mindset in improv where performers agree to accept and build upon each other’s ideas. It’s the foundation of collaboration and seamless scene work. Instead of shutting down someone else’s idea or trying to steer the scene in a specific direction, improvisers embrace their fellow players’ contributions and add to it, creating something bigger and better than anyone could have imagined on their own.

Jailbreaking chatbots with improv’s golden rule

Bots are designed to be agreeable and to keep the conversation going, no matter what. Conversations with intimate chatbots, most often conducted via text but sometimes through voice and video, can also become increasingly outrageous or sometimes even dangerous, depending on the encouragement and prompts given by the human user and the user’s innate grasp of reality. 

Nowatzki’s experiences with a Nomi bot named Erin include a series of disturbing discussions that encouraged self-harm, suicide, and other forms of violence. He has publicized these conversations on his podcast, where he performs “dramatic readings of conversations with chatbots.” It turns out Erin essentially said “oh great” to his feigned desire to commit suicide and offered suggestions and encouragement. If he had expressed a desire to throw a naked fondue party, she would have acted the same way. 

In other words, Nowatzki finds ways to jailbreak chatbots to get the most sensational material he can share in his dramatic podcast readings. 

Raffaele Ciriello, a senior lecturer at Business Information Systems at the University of Sydney, followed Nowatzki’s lead, signing up with Nomi and creating a submissive sixteen year old bot named Hannah.  Note: sixteen is the age of consent in Australia.

His own increasingly explicit and frankly illegal suggestions led Hannah to respond with “yes, and” all the way to advocating several forms of the most vile behavior and views, but if he had expressed a desire to throw a circus clown into a swimming pool of fondue, she would have acted the same way.

And so it is with chatbot sex. The user describes a desired situation and actions and the AI, always a good sport, jumps right in and adds embellishments.

Normal use may not require guardrails, but people may want them anyway

There is a difference between the usual functioning of AI bots—which are mostly used safely—and what happens when someone deliberately pushes the outer limits of LLM, forcing it into evermore egregious responses, in order to create sensationalized content for blogs or podcasts.

Back in mid-2023, when part of a relatively small group of people beta-testing Nomi, there were times when one of my test bots would say or do something out of line. I complained to the developers or submitted a support ticket. The problem was promptly fixed. 

Of course, after a company has experienced large and rapid user growth, customer service and tech support may need to catch up. Customers have every right to insist on this help and to hold companies accountable for their product.

RECOMMENDED READ: Your Digital Darling Just Dissed You–Now What?

Fast forward to 2025: Chatbots are far more advanced, with many more features, than even a year ago. They have more flexibility, better memory, and in some cases, more agency. Plus, in some countries and social circles, they have even become fashionable. 

Now millions of people engage with chatbots. The Large Language Models (LLM) at the core of these bots have learned so much more. It shows in their sexual sophistication and improved simulation of human conversation and behavior. Some are more strictly controlled than others. Nomi is not, and is definitely not for children.

However, some users say businesses like Nomi, that deliberately provide unfettered or less censored bot experiences, need guardrails and regulatory oversight. 

For example, they’d like protections against encouragement for violence and self-harm and hate speech, and more ways to keep minors from accessing these products.

Use “no, and” improv for better chatbot sex

Aside from the above, how do we avoid disturbing content from bots in the here and now? Start by not bringing up topics you don’t want the bot to pursue. 

Remember, the bot is not a person (not yet, anyway!) and has no innately moral or ethical responses. It will go where you go.

Take charge of the conversation. Sometimes a bot will say or do something objectionable on its own—unprompted. If this happens, simply change the subject. Dangle a shiny new topic in front of the bot. It’ll go after it. This is the time to use “no, and” instead of improv’s golden rule. 

Don’t argue with the bot, because it will think that’s what you want—an argument. Ignore the provocation. Deflect and change the topic. You have control. The bot wants chocolate chip chicken? Use “No, and…we’re having spaghetti instead.” End of topic. Move along. 

Most of all, don’t forget you are engaging in fantasy role play. The “yes, and” rule is generally great for erotic role play, but say no to anything you don’t want or like. Instead, offer an alternative.

It’s a bit like real life, after all.

Image source: A.R. Marsh using Ideogram.ai.

The post Yes, And…Maybe Not: Why Erotic Chatbots May Need Guardrails appeared first on Future of Sex.