Ask Lemmy

1. If your options are having a doctor that uses AI or having no doctor at all. Some doctor is better than none.

2. I would ask more information about what AI they are using, where the data is processed (locally or online), where and how the AI collected data is stored (locally or in the cloud), who can access your data and whether it could be used for some AI training.

Yes I would but only if I can be sure the LLM wasn't listening. Collection of personal information requires consent in Canada and I wouldn't be giving consent.

I don't believe for one second the conversations aren't uploaded to a datacenter nor that those transcripts will be deleted.

I have all sorts of anxiety surrounding AI. Most of the anxiety comes from the misuse, copyright issues and departure from critical and creative thinking. However, one of the fields I actually think it could be very useful and of great benefit is medicine.

That being said, I'd be a no as well. The way this is worded and he track record we've seen with privacy doesn't fill me with much confidence. Feels like another instance off loading of thinking rather as a tool for better diagnosis.

It sounds like America from the process. The confluence of commercialization of healthcare and tools that can make it look like time and attention has been used leads to some bad places. I'd be very sceptical about any advice medical or otherwise I recieved.

The unfortunate truth is that without these tools the cost of care will be higher for health companies not using the tools. Which means bespoke human led care will be a luxury in America in the near future. I don't think it's a reality you are going to be able to avoid.

I would push back at every opportunity, double check all of the information you are getting, ask pointed "why this" questions, make doctors clearly communicate that they are the ones giving the recommendation. At the end of the day a good doctor with AI tools is likely to do a better job.

For note taking only, id be fine IF it was all run locally with no ability to be trained on.

Id want assurances from the Dr that they also carefully review the notes immediately after or that I get to see the notes before leaving due to the risk of hallucinations that could cause future care problems.

They could have it visible on a screen while youre in the room with you to help you be sure its accurate.

Edit: id care less about it being local if it wasn't medical/legal in nature.

i wonder if they hallucinate notes post-appointment, i notice that there have been complaints against certain providers that the "doctors" did other examinations that they dint do in-person and it appeared on their records.

nope

AI summaries often make up details, omit what is important, and get stuff wrong. Every error may follow you forever complicating diagnosis and treatment and ultimately can harm or kill you.

Imagine charging $300 then outsourcing your note taking to a machine that barely knows shit and has nothing to lose

One of our doctors started using AI transcription and summary. I find it lacking substance after AI is done summarizing. You can see her thought process when she type her notes, it's thorough but concise. The AI summary is definitely short, but it's not about shortening, it's about handing your note to another doctor and that doctor is able to follow through with the plan.

Yes, because there aren't really going to be any more doctors that aren't using AI. AI is widespread and unavoidable. It's definitely unfortunate but it's true.

I have literally seen walk-in doctors have ChatGPT on their screen, they are literally using it to look up symptoms too.

My dentist started using this. Not the same service, but automatic AI-based visit recording, transcription & record-keeping. She's experienced with a difficult issue I have, so I'm staying with her for now but this is definitely a minus.

Hello, It us absolutely justified to be worried, tell your doctor you concerns, and ask your doctor questions about the use of AI. If you want some help putting together questions for your doctor lmk.

I'm involved with the development / integration of AI. From the specific text of the AI agreement, it looks like these are the AI tools you're consenting to:

• Transcription tool: This is a speech-to-text tool. It can differentiate between speakers.

• Transcript -> clinical documentation tool. This takes the text of the transcript, interprets it, and generates clinical documentation based on it.

It does not seem like, as part of the agreement, it covers taking the clinical documentation and attempting to suggest diagnosis or care steps.

I am actually concerned by the "recording and transcript are automatically deleted" line. If your doctor reviews the generated clinical documentation vs the transcript, and misses something for whatever reason, if they are unsure about something in the future they can't go back and reference the original audio / generated transcript to verify accuracy?

There are also concerns about how they are following HIPAA laws:

What model / service are they using?

Did they do their due diligence in deciding what service to use?

Have they looked at other cases where data companies have said they don't persist/ sell your data and then they sold it / there was a breach of data that shouldn't have persisted in the first place?

Do they anonymize personal information before they send it to whatever service they are using? -Note that this is not possible for transcription models, as they cannot know what text to anonymize/censor until the model generates the text. That doesn't mean there are not HIPAA-compliant text transcription models, text transcription models can even be run locally on maybe consumer-grade devices, meaning the audio doesn't have to be sent to a 3rd party.

AI and the people pushing it are not trustworthy. They do not have your data security nor your wellbeing at heart, even if your doctor does. LLMs are inherently bad at data security and there is no way these companies can, in good faith, promise HIPPA compliance. Likely, the AI use will be on the part of the insurance company to find ways of denying your claims.

Your provider then reviews the content of that note to ensure its accuracy and completeness.

You know they're not gonna do that, in practice.

Show him the EULA for copilot (where it's for entertaining purposes only), and tell him you'll be going elsewhere and leaving an appropriate review.

I feel very strongly about this and I would change doctors. But of course it won't be long before they all do this and we'll have no alternative. The two biggest problems I see are

1. I saw a news story where a doctor who uses this said it saves her time because before seeing the patient she gets an AI summary of their chart, so she doesn't have to "go through several tabs" to read the actual information. Oh great, let the statistical probability text generator hallucinate up some shit about what's in a person's chart, to save 10 seconds of tab-clicking to read the ACTUAL patient records! If they want a summary there's no reason a traditional report or summary screen couldn't be programmed to pull data out of the most important fields and arranging them in the desired format.

2. THEN the doctor uses her damn phone to record your visit, everything you say, and that gets run through the AI which generates a visit summary and puts that into your medical records. So, god only knows what 3rd party private corporate vulture has access to your doctor/patient conversations and what they'll do with them, and again, what hallucinated shit will get put into your medical records!

So your doctor never reads your chart and never writes your chart! [Readacted] me now! Also what happens after a few iterations of an AI summarizing records that an AI wrote?

It depends on many things. The hard line for me would be is this running locally, on a server with the same IT management as my actual data, or on a third party servers. If the doctor either don't know this, or can't give adequate proof that it isn't running on some third party servers, then all the "prioritize your privacy" aren't worth shit.

But that's only the point where I give a hard no. The way it is used would also matter a lot. Is it used as a clutch for reference searching, or a full self driving decision making process that will write me a prescription in the end? This part is the same whether it's for medical advice or for anything else: if the user is skilled enough to be able to evaluate/validate the output of the process faster than it would have taken them to do it manually, then there might be some value. Some usages fits into this. Some don't. Summarizing large documents you did not read does not work as a safe thing, because, you'd have to read the document to check the summary. Getting the summary of a drug/sickness/whatever that you know about but need a reminder of, could be ok.

tl;dr: it have to run in a privacy-enabled context (no third parties), it have to be used as a clutch (no skipping work), and the user have to keep is brain en mental activity alive enough to steer the system instead of being dragged by it. As things stands right now, I doubt there's a lot of doctors that would fit all three points, but in the future, maybe.

If my doctor needs AI to treat me then they ain't no doctor, they're middlemen.

I know this might go against the flow here, but realistically if they're using the tools in the way they say they are (which you should 100% check with your doctor to let him know about possible hallucinations) it's not that bad. Speech-to-text is not prone to hallucinate, it can fail and detect wrong things but shouldn't outright hallucinate. After that, LLMs are good at summarizing things, yes they are prone to hallucinations which is why having the doctor review the notes immediately after the session is important (and they said they do), so I don't see this as such a big issue from the usability point of view.

You might still have issues from a privacy point of view and that's a much more complex discussion with them about what kind of contract they have with the LLM company to ensure no HIPAA violations (as from the LLM point of view it's just making a summary of a text it might store it, and then the whole stack is suable). They need to understand that just because they haven't kept a copy around doesn't mean the other party hasn't, and because they shared it out without your agreement (you're only agreeing to AI note taking which can be done locally so them sharing information with third parties is entirely up to them) they would be liable. I'm not a lawyer, so you might want to double check that, but I would be very surprised if that's not the way it works, otherwise Drs could get away with a bunch of HIPAA violations by having you sign something that says they use a computer to store data and then storing things in shared Google drive.

No. Absolutely not. I csnnot trust any current AI model with HIPPA compliance.

Find another doctor. I just had to fire my therapist because when I went in for this week's appointment they were playing some jesus worship service and song. I told her that it was our last session because I no longer had trust in their offices and added that I had no faith any progress would ever be made after I was triggered waiting to see my therapist. It could have been the receptionists choice in music or someone else from their office but since they do not advertise as a faith based therapy group they should have left that shit at home or should expect more of the same from people like me.

I would nope the fuck out and change doctors. A regurgitation machine prone to hallucinations has no place in medical care.

My medical provider started doing that when I last had a video conference with them, and I declined to allow the use of AI. They took no issue with that -- didn't even bring it up. It's very unlikely that your provider will care that you declined either. I recommend saving your energy for other problems and dealing with this later in the unlikely event that they do actually make an issue of it.

No, but.

One of my doctors has an assistant nurse (or whatever they're called in the hierarchy) take notes just so the conversation can be more fluid. She always asks my permission for if that's okay with me.

My other doctor types and reads out her notes with me towards the end of my visit to make sure she hasn't missed anything, and she makes me feel heard and involved.

No, I wouldn't consent. Sending my PHI to a third party is unnecessary, and AI data centers are a net negative on the planet. I also wouldn't trust that the Ai service provider isn't helping themselves to your data + doctor's feedback to use for further training anyway. Thank god healthcare providers are required to ask before shunting your info off to some third party.

But, if presented with this, I'd talk to my doctor about the extent that third-party AI-services are already being used in my own healthcare. If I can fully opt out, I'd stay. If I didn't have a real choice to opt out, and if it were easy to find a new doctor that didn't use Ai-services, ~l'd fuck off so fast, like bye felicia, I ain't dealing with this palantir-esque bullshit just for getting a rx refill~

Definitely ask for how they are using it. I know a number of physicians that are just using it as a dictation software to quickly make a first draft for their paperwork, helps lighten a big load.

OP. I'm a bit of an unfortunate expert in U.S. Healthcare.

The fact that you have a psychiatrist who you trust that has you on the right meds and have been with for 3 years is invaluable. You calling yourself stable is a huge thing. You wouldn't be saying that if you weren't on solid ground.

It would be completely crazy to give up a psychiatrist who is on your insurance over some AI garbage that is just transcribing notes for your doctor.

At bare minimum get a new psychiatrist who is on your insurance before switching. That should take about 6 months if you're very lucky.

Play it through: do you want to lose a quality prescriber and talk therapist? Also, maybe you should just tell them you're extremely concerned and see what they say or do.

The end result can't be worse than you giving up on your mental health. You already know how hard it is to find quality psych care.