AI companions never get bored and always answer immediately. Amid a loneliness epidemic, with one in six people struggling for connection, chatbots are an enticing proposition, particularly among young people. A quarter of people under 30 have used AI for companionship, rising to 75 percent among teens.
Having somebody to talk to 24/seven may sound like the perfect treatment. There’s great value in the ability to strengthen communication skills without the pressure of interacting with another person. Issues arise, however, when users become dependent or when conversations drift into areas that AI isn’t well-suited to discuss.
The concerns surrounding the technology are valid, but AI companions aren’t inherently bad. Like any complex system, they require stronger controls and clear boundaries to ensure they’re used in a safe and healthy way.
You Can’t Code In Complete Safety
Potential harms are easily overlooked. LLMs ingest enormous amounts of messy data that’s incredibly difficult to sanitize. Especially when trained on unmoderated data, AI chatbots can quickly learn that wrong is right.
Take Tay, an early Microsoft chatbot, trained to learn from interactions with Twitter users. It was shut down after just 16 hours after it started posting offensive messages.
A decade on, safeguards have improved. Human language is so flexible that it’s impossible to fully constrain a chatbot, however. Known as “prompt hacking,” people will always find ways to exploit LLMs into saying things they shouldn’t.
Is It Time to Switch Off AI Companions?
Despite the controversy, AI companions do offer positives. If they didn’t, nobody would use them.
More than a third of teens insist that AI companions have provided them with transferable social skills. Likewise, 12 percent of people feel they have reduced their loneliness. When developed and maintained responsibly, they can help those struggling to practice communication to rebuild their confidence.
Even if lawmakers wanted to outlaw AI companions, it’s impossible to put useful technology back in the box. Prohibition simply pushes vulnerable users toward unregulated solutions where there are far looser safeguards and far greater risks.
How Can Responsible AI Companies Protect Their Users?
AI companies have a responsibility to protect their users. That means tightly controlling what their LLMs can say and do and continuously testing to ensure safety measures are holding up. From my experience of consulting with AI companion platforms like Joi AI, here are a range of measures to keep users safe.
How to Build Safer AI Companions
- Create age-appropriate experiences.
- Closely check your training data.
- Don’t let your model moderate itself.
- Monitor interactions and make changes.
- Don’t allow complacency to creep in.
Create Age-Appropriate Experiences
Any chatbot that doesn’t strictly prohibit use by children has no business engaging in adult conversation. Experiences must be age-appropriate and safeguards must be in place — think teen-only models, stricter filtering and robust age verification.
Likewise, setting daily time limits or encouraging users to step away during long chats could help to avoid users becoming over-reliant on AI companionship.
Closely Check Your Training Data
If you don’t know what’s in your training data, you have no hope of predicting what your LLM will output.
Every data set needs to be meticulously curated to make sure banned or sensitive topics don’t find their way in. In addition to content classifiers that automatically sort data, companies must also implement thorough human review. That starts with developing detailed guidelines outlining precisely what constitutes safe and appropriate data. Then comes careful labeling, with multiple people reviewing the same data to ensure guidelines are clear and that ambiguity doesn’t allow harmful data to slip through.
Don’t Let Your Model Moderate Itself
Any responsible chatbot developer has a strict, built-in list of prohibited topics. Systems fail, however, and chatbots get confused. So, you can’t rely solely on your model to determine whether a conversation is safe. Every response should pass through a separate moderation model — ideally, a transparent and open-source one — to act as a secondary fail-safe.
In addition, every LLM developer should have an independent advisory board of third-party experts who can provide unbiased advice on safety gaps and the best practices to address them. This board acts as an extra layer of scrutiny, preventing conscious or unconscious internal biases or pressures from shaping decisions around moderation, usage, and safety.
Monitor Interactions and Make Changes
Machines aren’t yet as reliable as humans when it comes to interpreting nuance or intent. Any conversation flagged by the system — or reported by users — requires a real person to step in.
A trained compliance team should be ready to review problematic interactions, such as a user attempting to manipulate the model into generating harmful content. Analyzing these conversations can help companies identify the early signs of unhealthy or inappropriate use that their automated moderation systems missed. Subsequently, rules and guidelines can be updated appropriately to prevent similar issues from recurring.
Don’t Allow Complacency to Creep In
It’s not enough to build a safe product; you have to maintain it. Companies need to run continuous automated safety tests to catch any prohibited topics that slip through. And just like in cybersecurity, companies should actively hunt for vulnerabilities, hiring external red teams to identify vulnerabilities before users do.
How Do We Continue to Build AI’s User Safety?
It’s easy to point the finger at AI companions. The issues they raise are common among chatbots, however: a lack of true understanding and the subsequent challenge of building safeguards that users cannot bypass with a clever prompt. Addressing this requires collective responsibility from all stakeholders, Including companies, scientists, lawmakers and technologists.
Progress begins with ethical design coded in from the very beginning. Likewise, we need tighter control over training data and continuous monitoring to stop harmful information from seeping into models. Regulation will play an important role. Legislation like the European Union’s AI Act, which grades systems by risk level, will separate those companies genuinely committed to safety from those putting profit above all else. But even with tighter controls, LLMs aren’t perfect, and we can’t expect them to be. Yet, that doesn’t mean that we, as an industry, shouldn’t be doing our utmost to minimize the risk.
