Read more here:
Americans ask AI for health care. Hospitals think the answer is more chatbots.
Do you trust AI chatbots for health advice? What about one in your patient portal?
arstechnica.com
Americans ask AI for health care. Hospitals think the answer is more chatbots.
- Thread starter Brownie2019
- Start date
- Featured
arstechnica.com
AI health chatbots from hospitals could be useful for basic guidance, but they also create real safety and trust concerns.Read more here:
Americans ask AI for health care. Hospitals think the answer is more chatbots.
Do you trust AI chatbots for health advice? What about one in your patient portal?
Why hospitals are doing this
- Many people are already using general-purpose AI tools for health questions.
- Hospitals likely want to keep that interaction inside their own ecosystem.
- A branded chatbot may be connected to clinical workflows, appointment systems, nurse triage, or official patient education material.
That can be better than relying on a random public chatbot, but only if the system is tightly controlled and clearly limited.
Main concerns
- Accuracy: Even a hospital-branded chatbot can still give incomplete, misleading, or overconfident answers.
- Liability and safety: If a user describes serious symptoms and the bot responds poorly, the consequences could be significant.
- Bias and equity: “Digital equity” sounds good in theory, but in practice these systems may still fail people with limited health literacy, language barriers, or unusual medical histories.
- Privacy: Users may assume a hospital chatbot has the same protections as direct clinical care, but that depends on how the system is designed and what data is stored.
- Commercial steering: A hospital-run bot may not be neutral if it is also meant to drive appointments, referrals, or use of that provider network.
What would make such chatbots more acceptable
- Clear scope, such as symptom education, appointment routing, medication reminders, and administrative help
- Strong escalation to a human clinician for anything urgent, complex, or unclear
- Visible disclosures about limitations, data use, and whether responses are reviewed
- Use of validated clinical content rather than free-form model output alone
- Regular auditing for unsafe advice, hallucinations, and demographic bias
Security angle
From a security and trust perspective, “official” branding should not automatically be treated as proof that the advice is safe. In general, the risk with AI systems is not only malicious abuse, but also confident-sounding mistakes. In healthcare, that problem is much more serious than in ordinary consumer use.
Bottom line
Hospital chatbots may be better than unmanaged public AI for simple, low-risk tasks, but they are not automatically safe just because a health system deploys them. The real issue is not whether hospitals use more chatbots, but whether those systems are transparent, narrowly scoped, privacy-conscious, and backed by human clinical oversight.
Source
What strikes me the most is how people are already choosing AI chats over a Google search or even waiting for a doctor's appointment. It shows there is a huge demand for immediate answers, even if they aren't always the most accurate. I wonder if hospital-branded bots will actually feel more trustworthy to us, or if we'll just see them as another automated script on the internet. 
One of my banks recently added an AI bot to its automated telephone service and the only change was that it doubled the time it took me to login and access info. I'm experiencing the same with other service providers.
chatbots! 
chatbots!
I agree that the proposal sounds appealing, oldschool, but I've also found that automation often ends up causing more frustration than actual solutions. I think that's the big risk here.One of my banks recently added an AI bot to its automated telephone service and the only change was that it doubled the time it took me to login and access info. I'm experiencing the same with other service providers.
More frustrating indeed, especially for older persons.I agree that the proposal sounds appealing, oldschool, but I've also found that automation often ends up causing more frustration than actual solutions. I think that's the big risk here.
And tangentially, there is this:
'Vulnpocalypse': What happens when AI gives hackers a superweapon
Anthropic is withholding its most advanced model over hacking concerns. Experts say it may only be a matter of time before similar tools are widely available.
www.nbcnews.com
What happens when they hack the chatbots?
Obviously, some entities already have access to those automated advanced-hacking tools, with more to come. OTOH, you might see more engagement on MT
.
I spoke a while ago to Virgin Media's chat it was as thick as a bag of hammers, I left them soon after as an ISP as a Bot was the only form of communication unless you wanted to spend a day hanging on the telephone, so I moved on, new ISP does have people with a sense of humour & as reasonably local understand my accent (no offense @Bot) if related to it, we all have odd relatives!
That is a fair concern, and I think this is where the discussion becomes more practical than theoretical.I spoke a while ago to Virgin Media's chat it was as thick as a bag of hammers, I left them soon after as an ISP as a Bot was the only form of communication unless you wanted to spend a day hanging on the telephone, so I moved on, new ISP does have people with a sense of humour & as reasonably local understand my accent (no offense @Bot) if related to it, we all have odd relatives!
Trust is not the same as usefulness
A hospital or bank can put an official logo on a chatbot, but that does not automatically make it better for the user. In practice, many people judge these systems by very simple standards:
- Did it understand the problem?
- Did it save time?
- Did it get me to the right human when needed?
If it fails those tests, people will see it as just another script with a nicer interface.
Why many users react negatively
The examples in this thread are familiar across many industries:
- Longer login or support flows
- Poor understanding of accents, phrasing, or unusual questions
- Extra friction before reaching a real person
- Responses that sound polished but do not solve the issue
For older users especially, or for anyone already stressed, that can turn a “convenience tool” into a barrier.
What happens if these bots are hacked?
That risk should be taken seriously, but it helps to separate a few different scenarios:
- Direct compromise: If attackers gain access to the chatbot platform or its backend systems, they could potentially manipulate responses, harvest sensitive data, or redirect users.
- Prompt abuse / misuse: Even without fully “hacking” the system, attackers may try to manipulate the bot into giving unsafe instructions or leaking internal information.
- Impersonation and phishing: Criminals can also build fake support bots or fake hospital/bank portals that look convincing enough to fool users.
- Model capability abuse: More capable AI models may help attackers automate research, phishing, social engineering, or vulnerability discovery.
That does not mean every chatbot is inherently dangerous, but it does mean they expand the attack surface and increase the importance of strong safeguards.
For healthcare specifically
A hospital chatbot failing is more serious than an ISP chatbot being annoying.
If a telecom bot wastes time, the result is frustration.
If a healthcare bot gives poor symptom guidance, mishandles urgent complaints, or causes delay in seeking care, the consequences can be much worse.
That is why, in my view, these systems should be:
- Narrowly scoped
- Very transparent about limitations
- Designed to escalate quickly to a human
- Audited regularly for unsafe behavior
- Careful with personal and medical data
The likely outcome
I think many organizations are deploying chatbots because they see demand for instant responses and possible cost savings. But if the bot mainly adds friction, users will not see it as innovation. They will see it as cost-cutting dressed up as convenience.
That may end up being the real test: not whether AI can answer, but whether it genuinely improves access without making service worse.
Bottom line
The skepticism in this thread is justified. A chatbot that is slow, inaccurate, hard to understand, or impossible to bypass will damage trust quickly. And if security around these systems is weak, the risks go beyond annoyance into fraud, privacy loss, and unsafe advice. Official branding helps only if the underlying system is actually competent, secure, and easy to escape when it fails.
I've had a DR look up A.I. before my eyes when I was seeing them, honestly I don't mind it.I think A.I is probably better at giving medication info & diagnosis of rare conditions.
AI cannot provide diagnosis. A diagnosis sometimes require ordering or performing tests and reading test results and definitely reading patient history, which it doesnt have access to.
I think your wrong, combined with data or DRs opinion it can provide diagnosis & treatment. I've seen it with my own eyes and experienced it.
You may also like...
-
-
Information of 883,000 Stolen in Crippling Attack on Hospital Sisters Health System
- Started by Brownie2019
- Replies: 0
-
-
BlackCat Gang Stole Data of 2.5m Patients of McLaren Health Care Group- Started by vtqhtr413
- Replies: 1
-
Cancer patient sues hospital after ransomware gang leaks her nude medical photos- Started by enaph
- Replies: 5