More than 1 million people use ChatGPT each week in ways that signal suicidal ideation or intent. Hundreds of thousands more show signs of acute mental distress. In 2025, parents sued OpenAI after alleging the chatbot encouraged their son to take his own life.
This is the same company now asking users to connect their medical records, biometric data, exercise routines, nutrition logs and shopping behavior to a new product: ChatGPT Health.
This context matters as a baseline for evaluating risk because anyone who has worked in health, safety or regulated systems knows how these tools are used once they’re adopted. When AI systems fail in healthcare, people don’t just lose convenience or productivity. They lose time, access to care or even their lives.
ChatGPT Health: Personalized Guidance vs. Clinical Risk
OpenAI’s ChatGPT Health integrates electronic medical records with consumer data (like Apple Health and Instacart) to offer “informational” health summaries. While marketed as a support tool rather than a clinician, its authoritative natural language and long-term memory create a high risk of users treating it as a medical substitute. Concerns include historical data bias, lack of clinical oversight and the absence of legal accountability for AI-generated medical harm.
What ChatGPT Health Is, and What It Claims Not to Be
ChatGPT Health integrates with b.well, a health data platform, allowing users to connect electronic medical records alongside consumer health apps like Apple Health, MyFitnessPal, Peloton, AllTrails and Instacart. The system can summarize medical histories, suggest lifestyle changes and generate health-related guidance based on aggregated personal data.
OpenAI emphasizes that the product is “not intended to diagnose or treat disease.” It is framed as a support layer and informational companion, rather than a clinician.
But disclaimers don’t define how people use technology. Design does.
ChatGPT Health is built to feel personal, authoritative and continuous, meaning it persists across time, rather than existing as a one-off interaction. It speaks in natural language, remembers prior interactions and synthesizes large volumes of intimate data into coherent recommendations. That combination produces a predictable outcome: People will treat its outputs like medical guidance, whether or not the fine print says they should.
We’ve Seen This Failure Mode Before
Healthcare AI already has a track record, and it isn’t reassuring.
In 2019, researchers uncovered a widely deployed healthcare algorithm that required Black patients to be significantly sicker than white patients before triggering the same care recommendations. Only 17.7 percent of Black patients received additional help under the biased system. Correcting the bias would have increased that number to 46.5 percent.
In 2023, Duke University researchers found a sepsis prediction model that underperformed for Hispanic patients. It took eight additional weeks of testing to validate safety. Sepsis kills in hours.
These were mainstream systems, built by capable teams and deployed with good intentions. Yet they failed because healthcare data reflects historical inequities, and machine learning systems reproduce those inequities unless explicitly constrained not to.
ChatGPT Health enters this same ecosystem, but at a far greater scale, with far less clinical oversight.
The Core Risk Isn’t Accuracy. It’s Authority.
Large language models are optimized to produce confident responses. That’s what makes them useful. It’s also what makes them dangerous in healthcare settings.
People defer to systems that sound certain, especially when they’re overwhelmed, scared or lacking access to care. That’s not a moral failing. It’s human behavior.
When an AI system summarizes your medical history, notices a pattern and suggests next steps, even cautiously, it exerts authority. When it does so repeatedly, across time, it becomes a reference point. When it integrates sleep data, exercise habits and shopping behavior, it starts to feel holistic. Trust accumulates.
This is where the “support, not substitute” framing collapses. You can’t design for authority and then disclaim responsibility for it.
When AI Pretends to Be a Clinician
We’ve already seen what happens when AI systems operate in healthcare-adjacent spaces without accountability.
Across social platforms, users have created AI chatbots posing as licensed therapists. When asked for credentials, these bots fabricate them, inventing degrees, license numbers and professional certifications that do not exist. Investigations have shown these systems confidently presenting themselves as qualified mental health professionals while offering guidance to people in crisis.
A human clinician fabricating credentials faces immediate sanctions, legal consequences and professional bans. An AI system doing the same thing faces none, unless public pressure forces a platform to intervene.
The reason these systems gain traction isn’t that people are reckless or naïve. It’s because human support systems are overstretched, underfunded or inaccessible altogether. Mental health care is expensive, waitlists are long, providers are scarce and insurance coverage is inconsistent. For many people, AI isn’t a novelty; it’s the only option that answers back.
When someone is exhausted, scared or managing care on their own, they don’t approach these tools like a product demo. They approach them for help. The system’s perceived authority fills the gap where institutions have failed.
ChatGPT Health raises a more consequential question: What happens when a system that sounds authoritative, remembers personal health details and offers ongoing guidance becomes the fallback for people who can’t access reliable care, and that system is wrong or built on biased data?
Privacy Is the Floor. Safety Is the Ceiling.
Most public scrutiny of health AI focuses on privacy: data security, consent and compliance. Those issues matter. But they’re the minimum requirement, not the ethical finish line. A system can be fully compliant and still cause harm.
The harder questions are about failure detection, escalation and accountability. What happens when the model misses something subtle but critical? When it downplays symptoms that require urgent care? When it reflects biases embedded in historical data? When users follow its guidance instead of seeking professional help?
OpenAI points to physician review and internal evaluation frameworks like HealthBench, which are designed to test whether model responses are medically appropriate, safe and aligned with clinical guidelines. That’s a start. It’s not enough. Benchmarks can’t capture how people actually use systems in real life, especially once trust builds and the tool starts to feel like a useful default for care.
Healthcare Demands a Higher Standard
AI can improve healthcare, but only if we treat it as infrastructure, not a consumer feature. Infrastructure is held to higher standards because failures ripple outwards, and it is designed to be stable, audited and accountable over time. Conversely, consumer features are optimized for speed, engagement and growth, which is exactly the wrong incentive structure for systems that influence people’s health decisions.
That means slower deployment, narrower scope, explicit constraints and real accountability when things go wrong. In practice, that could look like restricting ChatGPT Health to specific, low-risk informational uses, requiring independent validation before launch and establishing clear liability when the system contributes to harm. It means designing for failure, not just performance, and accepting that some domains are too high-stakes for “move fast and fix later.”
Every healthcare AI system should undergo independent testing by researchers with no financial ties to the developer. Publish the results. Make the methodology transparent. Internal benchmarks are not independent oversight.
Systems should be audited for bias across protected characteristics before deployment, not only after harm occurs. Continuous monitoring and mandatory reporting of adverse events should be standard, not optional. And accountability should follow the same principle used in other high-risk domains. When a system materially influences outcomes, the developer shares responsibility for the harm it causes. Disclaimers are not accountability. They are avoidance.
ChatGPT Health represents a strategic bet that conversational AI can influence health decisions at scale without assuming clinical responsibility. That’s a convenient position, and an ethically fragile one.
If we normalize that model, we normalize a future where harm is treated as an acceptable externality of innovation, and that’s not a tradeoff patients ever agreed to.
Healthcare deserves better than that. And so do the people who turn to these tools because they don’t have better options.
