The integration of large language models (LLMs) into the healthcare landscape, like the emergence of ChatGPT Health, marks a pivotal moment in the history of medicine. We are entering an era where the barriers between complex medical information and the individual are finally dissolving. At Profusa, where we have spent years pioneering tissue-integrating biosensors for the continuous monitoring of body chemistries, we view this shift with optimism. This development represents the democratization of data in its most tangible form, putting the power of informed decision-making directly into the hands of the patient.
As we open this new chapter, however, we must acknowledge that every revolutionary tool carries a set of unique risks. As a biotech community, our goal should be to ensure these tools are as successful as possible, which requires us to be vocal about the ethical and safety risks that must guide their evolution. If we ignore these challenges, we risk failing fast at a moment that could result in real human harm and set the digital health movement back by at least a decade.
Is AI in Healthcare Safe?
Although tools like ChatGPT Health offer the potential to democratize medical data, they carry significant risks if not properly regulated. To ensure patient safety and ethical integrity, the integration of large language models (LLMs) into medicine must meet three critical standards:
- Validated Data Sets: AI must be trained on clinically reviewed, evidence-based data rather than unverified internet sources like social media.
- Real-Time Monitoring: Systems should incorporate “closed-loop” data from wearables and biosensors to track how a patient’s chemistry responds to treatment in real time.
- Ethical Consent: Patients must maintain absolute ownership of their biological data, with transparent frameworks for how their information is used to train AI models.
It’s All About the Data Sets
In the world of AI, there is a simple truth — the model is only as good as its foundation. While the interface of a chatbot may feel like magic, its utility is strictly bound by the quality of the data it ingests. If tools like ChatGPT Health are to move beyond mere novelty and become true clinical assets, we must demand a higher standard for “validated” data sets. This means data that is clinically reviewed, evidence-based and sourced from trusted medical and scientific institutions rather than places like Reddit.
Currently, most LLMs are trained on a broad swath of the internet. In a medical context, this is inherently dangerous. For example, if a model treats a peer-reviewed oncology study with the same degree of importance as an unverified anecdote from a social media forum, the outcome is not just “noisy.” It’s potentially lethal. The AI cannot inherently discern what is scientifically rigorous versus what’s simply popular.
To be truly effective, these data sets must evolve. They should not just include static information like diagnoses, symptoms and treatment modalities. They must incorporate real-time views of change, tracking how people’s chemistries and health markers respond to treatments or lifestyle shifts. This “closed-loop” approach, which can be achieved through wearables or biosensors that continuously collect and update health data, allows us to see how people are changing for the better or worse in response to the decisions they make. Without this real-time validation, we’re essentially giving patients a high-tech compass that hasn’t been calibrated.
The Ethics of Ownership and Consent
As we build these massive training sets, we run head-first into the fundamental premise of modern health data: The individual owns their information. Patients should have the absolute power to determine who, what, where and how their biological data is used.
This creates a significant ethical quandary. AI models require large numbers of data sets to reach the level of accuracy we require for medical safety. But how do we aggregate this data while maintaining the sanctity of consent? For instance, if an LLM provides advice on a specific cancer treatment, was that advice learned from patients who gave explicit permission for their records to be part of a training set?
If we build these tools on the backs of patients without their knowledge, we have breached the trust that is the bedrock of the patient-provider relationship. We must ensure that the push for big data collection is governed by transparent consent frameworks, clear opt-in participation, and disclosure on how their health data will be used. Without this, the big data push risks becoming a vacuum that sucks up private health journeys without a clear, ethical mandate.
The Untrained Hand and the Golden Rule of Healthcare
In medicine, our North Star is the Hippocratic Oath: Primum non nocere — First, do no harm. This must remain the number one priority for AI developers and biotech leaders alike.
The irony of the current moment is that, while we want to empower individuals to take control of their own healthcare, we’re placing powerful, potentially misleading advice into untrained hands. A patient may not have a team of highly trained professionals to vet the AI’s suggestions. This is particularly concerning for younger patients or those with complex, multi-system disorders.
When AI provides a recommendation, it often does so with an authoritative, clinical tone. If that advice is misguided, perhaps because the system missed a subtle contraindication in the data, the patient may act on it with a false sense of security. To counter this, LLMs need a validation layer where advice is cross-referenced against verified clinical guidelines before it reaches the user. We cannot treat an AI health suggestion with the same casualness we treat an AI-generated recipe or travel itinerary.
The Evolving Privacy Threat in AI Healthcare
Although the general public has become somewhat desensitized to data privacy in a social media context, health privacy is a different beast entirely. We’re no longer just worried about our names and addresses being leaked; we’re worried about the nefarious use of our search history and health inquiries.
Consider the so-called Mole Scenario. If a person uses a health LLM to analyze a photo of a mole or to search for symptoms of melanoma, where does that data go? The concern isn’t just about a data breach. It’s about the secondary use of that data. Could an insurance company access those logs to determine that a person has a “predisposition” to a disorder before a clinical diagnosis is even made? Could your digital paper trail of health concerns impact your future premiums or your ability to get coverage? These are not hypothetical fears; they’re the privacy challenges of the next decade.
Safeguarding the Future
The vision of Profusa has always been about bringing clarity to the body’s internal chemistry through continuous monitoring. We believe that tools like ChatGPT Health can be the “brain” that interprets the “signal” we provide. But for this partnership to work, the brain must be trained on truth, not noise.
We must advocate for curated, scientifically verified data sets, transparent consent models and rigorous validation of AI advice. If we prioritize these ethical safeguards now, we can ensure that these tools fulfill their promise of making a direct, positive impact on human life. If we don’t, we risk the very technology meant to uphold the Golden Rule of health tripping it up from the start.
Let us move forward with the excitement this new chapter deserves, but with the caution that medicine demands.
