Ask Lemmy

I would nope the fuck out and change doctors. A regurgitation machine prone to hallucinations has no place in medical care.

Definitely ask for how they are using it. I know a number of physicians that are just using it as a dictation software to quickly make a first draft for their paperwork, helps lighten a big load.

OP. I'm a bit of an unfortunate expert in U.S. Healthcare.

The fact that you have a psychiatrist who you trust that has you on the right meds and have been with for 3 years is invaluable. You calling yourself stable is a huge thing. You wouldn't be saying that if you weren't on solid ground.

It would be completely crazy to give up a psychiatrist who is on your insurance over some AI garbage that is just transcribing notes for your doctor.

At bare minimum get a new psychiatrist who is on your insurance before switching. That should take about 6 months if you're very lucky.

Play it through: do you want to lose a quality prescriber and talk therapist? Also, maybe you should just tell them you're extremely concerned and see what they say or do.

The end result can't be worse than you giving up on your mental health. You already know how hard it is to find quality psych care.

No. Absolutely not. I csnnot trust any current AI model with HIPPA compliance.

Find another doctor. I just had to fire my therapist because when I went in for this week's appointment they were playing some jesus worship service and song. I told her that it was our last session because I no longer had trust in their offices and added that I had no faith any progress would ever be made after I was triggered waiting to see my therapist. It could have been the receptionists choice in music or someone else from their office but since they do not advertise as a faith based therapy group they should have left that shit at home or should expect more of the same from people like me.

An AI tool does NOT prioritize privacy. It's literally the opposite.

Can you ask how AI is used in the app?

I feel very strongly about this and I would change doctors. But of course it won't be long before they all do this and we'll have no alternative. The two biggest problems I see are

1. I saw a news story where a doctor who uses this said it saves her time because before seeing the patient she gets an AI summary of their chart, so she doesn't have to "go through several tabs" to read the actual information. Oh great, let the statistical probability text generator hallucinate up some shit about what's in a person's chart, to save 10 seconds of tab-clicking to read the ACTUAL patient records! If they want a summary there's no reason a traditional report or summary screen couldn't be programmed to pull data out of the most important fields and arranging them in the desired format.

2. THEN the doctor uses her damn phone to record your visit, everything you say, and that gets run through the AI which generates a visit summary and puts that into your medical records. So, god only knows what 3rd party private corporate vulture has access to your doctor/patient conversations and what they'll do with them, and again, what hallucinated shit will get put into your medical records!

So your doctor never reads your chart and never writes your chart! [Readacted] me now! Also what happens after a few iterations of an AI summarizing records that an AI wrote?

AI is an overloaded marketing term. Definitely ask which kind of AI, how it is used, how and which of your data is going to be used.

I'm a therapist and I use SimplePractice for my practice. They recently added an AI note taker that is HIPAA compliant, and the consent form they suggest giving to clients sounds okay, but I read the actual privacy policy and the language used is way too vague for me to trust, so I don't use it.

In your position, I would:

1. Ask if you have to sign that, or if you can opt out. Your specific provider may be open to just not enabling the AI note taker for your profile, and they may be able to remove that form from the app for you on their end. This may not be in their control, but if they're a good person who cares about you, they'll make an effort to get it done anyway.

2. If not, ask for a link to the actual privacy policy and see if it sounds acceptable to you. Not the practice's Privacy Practices, not the Patient Portal privacy policy, but the actual privacy policy for the AI note taker (whoever you ask might have to do some digging to actually find it)

The privacy statements are fucking lies.

I will not share my innermost mental issues with some group of 20-something "move fast and break things" sociopaths in Silicon Valley.

It would be an absolute deal breaker for me. There has never yet been a commercially available AI that doesn't hallucinate, and there's no element of my healthcare where I'm comfortable having facts be unreliable.

My medical provider started doing that when I last had a video conference with them, and I declined to allow the use of AI. They took no issue with that -- didn't even bring it up. It's very unlikely that your provider will care that you declined either. I recommend saving your energy for other problems and dealing with this later in the unlikely event that they do actually make an issue of it.

Fuck no. I wouldn't even install the app. That's already completely unnecessary.

I would only be ok with an AI note taking app if the model is running on hardware the doctor physically has in their office because otherwise any privacy assurances don't mean that much.

I left. Similar thing happened to me, except the doctor said he was going to use an AI tool on his computer (or maybe it was an iPad/Android tablet? It was a couple years ago) but needed my permission. I said no. He said it meant he'd have to write stuff down. I asked if there was any way he could do it without feeding AI my data and training the AI. He said no. I said no to it. They would not schedule me again. I actually still need to change GPs.

I would be out of that office faster than the speed of light.

Given how captured our data is by the lack of regulation even in the medical space in the US. I simply do not want my personal data to be used in anything but in-house signal-to-noise improvement for diagnosis.

Anything else, which is most of it, is unacceptable and I do not consent.

Personally, I would straight up refuse and hunt for another practice that does not utilize LLM services. As anything that techbros call "AI" is an absolute shitshow. These techbros have a plan to absorb a lot of personal information from others in order to 'train' their models. Given that LLMs can never think, feel, make shit up confidently, and are geared towards being sycophantic...

They should never be used for the purposes that techbros and deluded CEOs are trying to make fetch. I would rightfully be suspicious of any practice that is willingly using this tech. I would ask for clarification from the practice in question, get shit in writing, and hunt for another practice if the meeting is unproductive.

I think your provider would probably be willing to exempt you from their AI processing shit if you talked to them, given the good relationship you seem to have built. Raising your concerns like that could benefit other patients in their care as well, depending on how well you articulate them and how receptive they are

id request further information. is the LLM used for secondary analysis? is it the primary and the doctor evaluates the results manually?

'ai' is just a tool in the right hands that can be beneficial. that said, it absolutely can be used by lazy assholes to pretend to do their job..

so, if you ave the resources to demand 'no ai'.. go for it, but im too poor to demand much, and id be more focused on the use of the ai.

Nope nope nope

AI and the people pushing it are not trustworthy. They do not have your data security nor your wellbeing at heart, even if your doctor does. LLMs are inherently bad at data security and there is no way these companies can, in good faith, promise HIPPA compliance. Likely, the AI use will be on the part of the insurance company to find ways of denying your claims.

Hello, It us absolutely justified to be worried, tell your doctor you concerns, and ask your doctor questions about the use of AI. If you want some help putting together questions for your doctor lmk.

I'm involved with the development / integration of AI. From the specific text of the AI agreement, it looks like these are the AI tools you're consenting to:

• Transcription tool: This is a speech-to-text tool. It can differentiate between speakers.

• Transcript -> clinical documentation tool. This takes the text of the transcript, interprets it, and generates clinical documentation based on it.

It does not seem like, as part of the agreement, it covers taking the clinical documentation and attempting to suggest diagnosis or care steps.

I am actually concerned by the "recording and transcript are automatically deleted" line. If your doctor reviews the generated clinical documentation vs the transcript, and misses something for whatever reason, if they are unsure about something in the future they can't go back and reference the original audio / generated transcript to verify accuracy?

There are also concerns about how they are following HIPAA laws:

What model / service are they using?

Did they do their due diligence in deciding what service to use?

Have they looked at other cases where data companies have said they don't persist/ sell your data and then they sold it / there was a breach of data that shouldn't have persisted in the first place?

Do they anonymize personal information before they send it to whatever service they are using? -Note that this is not possible for transcription models, as they cannot know what text to anonymize/censor until the model generates the text. That doesn't mean there are not HIPAA-compliant text transcription models, text transcription models can even be run locally on maybe consumer-grade devices, meaning the audio doesn't have to be sent to a 3rd party.

Hell no.

Shit, I left one doctor that would look up symptoms told to him on WebMD. AI is even worse. I want a doctor who graduated with an A or higher; not one who barely squeaked by with a D.

I know this might go against the flow here, but realistically if they're using the tools in the way they say they are (which you should 100% check with your doctor to let him know about possible hallucinations) it's not that bad. Speech-to-text is not prone to hallucinate, it can fail and detect wrong things but shouldn't outright hallucinate. After that, LLMs are good at summarizing things, yes they are prone to hallucinations which is why having the doctor review the notes immediately after the session is important (and they said they do), so I don't see this as such a big issue from the usability point of view.

You might still have issues from a privacy point of view and that's a much more complex discussion with them about what kind of contract they have with the LLM company to ensure no HIPAA violations (as from the LLM point of view it's just making a summary of a text it might store it, and then the whole stack is suable). They need to understand that just because they haven't kept a copy around doesn't mean the other party hasn't, and because they shared it out without your agreement (you're only agreeing to AI note taking which can be done locally so them sharing information with third parties is entirely up to them) they would be liable. I'm not a lawyer, so you might want to double check that, but I would be very surprised if that's not the way it works, otherwise Drs could get away with a bunch of HIPAA violations by having you sign something that says they use a computer to store data and then storing things in shared Google drive.

I would seek a new practitioner and cite this as the reason. Given the track record of these tools, I personally question the judgement of someone using them in clinical documents and even creating “temporary” recordings. Therefore, I would question their judgement and their ability to manage their caseloads if they feel this is needed. I can see this being disappointing after going through all the paperwork. Sorry you’re having to deal with that.

No. Flat out.

It depends on what model they're using.
If its just a front end for GPT or Claude, no.
If it's a local LLM sure, but I'd like a copy to review also.