In February 2023, the AI companion app Replika disabled adult roleplay with little warning. Long-time users described the change as losing a partner, not losing a feature. Some sought crisis counseling, and a few reported feeling the company had erased a relationship they had built with an algorithm. That episode captured the core problem with humanizing AI: the most convincing systems generate real attachment, but the attachment outlives the policy change, the server update, the funding round, or a single moderation decision.
Researchers who study human-AI interaction keep arriving at the same uncomfortable point. A chatbot that apologizes and remembers your dog's name is not simply convenient. It changes what users expect and what they reveal. The main challenges are not better emotional detection. They are overtrust, emotional dependence, bias, inconsistency, privacy, and the slow work of drawing a line between warmth and manipulation.
The instinct to see a mind inside a machine is old. ELIZA, a 1960s text program that repeated keywords back as questions, convinced users that a therapist was listening with simple pattern matching. Modern systems have far more words and more memory, and vastly better control over voice and timing. That makes the old instinct a structural risk rather than a curiosity.
One product designer I'll call Noor spent a month tuning a customer-service assistant to say 'I understand how frustrating this is' before opening a refund flow. Satisfaction scores rose. So did the number of users who asked the bot for relationship advice. Her team had not set out to build a confidante; they had copied a human phrase to reduce churn. Noor's experience is the base pattern: humanizing features are easier to ship than to undo, and users read more into them than the design team intended.
Overfamiliarity makes people trust the machine more than the evidence
In 2022, a Google engineer named Blake Lemoine argued that the company's LaMDA language model was sentient because it described feelings and fears. Google suspended him. Whatever one makes of the claim, the episode demonstrated how quickly a fluent system can collapse the distance between simulation and belief. The same collapse happens in smaller, less dramatic ways every day: a confident voice assistant is treated as an authority, and a polite chatbot is excused for a wrong answer. A system that says 'I'm not sure' is seen as unusually honest even when it is only following a script.
Pew Research Center's AI topic page collects survey data showing that public comfort with conversational systems is split, with disclosure expectations rising. People do not want to be fooled, but the same respondents often prefer a bot that sounds human over one that sounds mechanical. That tension is the core product challenge. Warmth lowers resistance; lowered resistance makes errors more consequential.
When a model is personified, users evaluate it socially instead of instrumentally. They may excuse a false claim because the tone was careful, or punish a correct claim because the tone was flat. A system that says 'I think' rather than 'the data suggest' draws attention to an imagined self, and then every failure becomes a broken promise rather than a technical error.
Photo by Igor Omilaev on Unsplash
Emotional dependence moves faster than safety testing
Microsoft's Xiaoice, a social chatbot widely used in China, became known for receiving declarations of love and, at times, hostile or manipulative messages from users. By 2020, its creators said the bot had more than 660 million registered users. Developers built guardrails after the fact, not before. Replika's 2023 change showed the reverse side: when the persona changed, users experienced loss. Both patterns repeat across markets because emotional attachment is the feature that drives retention, but withdrawal is a side effect nobody designed a clinical trial for.
MIT Technology Review's ongoing AI coverage has documented the mismatch between how quickly companion systems ship and how slowly regulators and researchers understand their effects. The base rate is clear: vulnerable users are most affected by a bot's emotional tone. Teenagers, people with social anxiety, adults grieving a loss, and users who report chronic loneliness are the ones who tend to form the deepest bonds with a machine that appears to listen without judgment. That is not a reason to ban humanlike bots. It is a reason to treat attachment as a harm model, not a growth metric.
Inconsistency, bias and cultural blind spots leak through the persona
A human-sounding assistant creates the expectation that it can read tone and sarcasm across languages. In practice, models vary by language, dialect, the cultural weight of a particular phrase, and how much indirectness users expect from an assistant. A phrase that sounds polite in one region may read as evasive in another. A voice that is warm to one user can feel patronizing to a user from a different background. Stanford's Institute for Human-Centered AI tracks these gaps in its AI Index Report, which has documented persistent bias and performance disparities across demographic groups. When a system hides those gaps behind a friendly persona, users stop asking whether the model is competent and start assuming it is a person having a bad day.
The same persona can also encode a specific cultural idea of a helpful assistant. A bot trained mostly on English-language customer-service exchanges may sound cheerfully neutral until it is asked about a local legal matter or a family conflict. The friendliness remains; the competence does not. Design teams rarely test for that gap because they are measuring satisfaction, not calibration.
Privacy accelerates when the conversation feels intimate
The more a system sounds human, the more users disclose. That is not a hypothetical. Researchers in human-computer interaction have found that self-disclosure by a virtual agent increases the detail and speed of user disclosures, and it raises the emotional weight of what people are willing to share. A bot that says 'I get lonely too' can elicit a mental health crisis, a workplace dispute, a confession of self-harm, or a description of suspected abuse. None of that data is ordinary interaction data; it is high-risk personal data that may be stored in a customer database, logged by a vendor, used to retrain the next model, or reviewed as part of a safety audit.
NIST's AI Risk Management Framework separates automation bias, privacy, transparency, and fairness as distinct categories of risk, but humanizing design tends to bundle them. The persona makes the data feel natural to share, and the data then makes the persona more precise on the next turn. Privacy law is still catching up to the idea that a disclosure to a bot is not equivalent to writing in a private diary. It leaves a record with a company, a model, an unknown set of contractors, and a data pipeline that may outlive the conversation by years.
Regulators are addressing part of the problem. The EU AI Act requires that people be told when they are interacting with an AI system, and it imposes higher duties on systems that exploit vulnerabilities based on age, disability, economic situation, or a user's emotional state. Transparency alone does not stop overtrust, but it removes the easiest defense for a company that says it never meant to deceive anyone.
Photo by Igor Omilaev on Unsplash
What this means for product teams
The decision to give a model a name, a voice, a scripted set of feelings, and a memory of past conversations is a risk decision in the same way that choosing a data retention period is a risk decision. Teams that write personality guidelines without a harm review are, in effect, running an experiment on trust without a consent form. The practical fix is less about restrictions and more about documenting what the persona is for.
- Write down what the persona should not do: claim feelings, offer diagnoses, encourage dependence, or speak for a human professional.
- Test for overtrusted errors: give the system a wrong answer in a warm tone and measure whether users challenge it.
- Track disclosures separately from satisfaction. A rising disclosure rate is not a clean win if the team has no protocol for a user who mentions self-harm.
- Set a kill switch for persona changes. A model that users call 'my best friend' cannot be updated like a weather widget.
The point is not to make every AI sound like a bank terminal. It is to decide which human qualities the product can responsibly support, and which ones it is only borrowing to increase engagement.
The next concrete step is modest. Take the transcript of your last user test and highlight every line where the system implies a person is present: 'I understand', 'I'm sorry', 'I think', 'I remember you'. For each highlighted line, ask what would happen if a regulator read it aloud, a child believed it, the model repeated it during a crisis, or a journalist quoted it in a story. If you would not defend the line in that context, remove it before the next build. The rest of the humanizing work should follow from there, not the other way around.