this post was submitted on 18 Dec 2023
99 points (74.1% liked)
Technology
58143 readers
5397 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
100% accuracy is troublesome. Literally statistics 101 stuff, they tell you in no uncertain terms, never, never trust 100% accuracy.
You can be certain to some value of p. That number is never 0. .001 is suspicious as fuck, but doable. .05 is great if you have a decent sample size.
They had fewer than 1000 participants.
I just don't trust it. Neither should they. Neither should you. Not at least until someone else recreates the experiments and also finds this AI to be 100% accurate.
What they're saying, as far as I can tell, is that after training the model on 85% of the dataset, the model predicted whether a participant had an ASD diagnosis (as a binary choice) 100% correctly for the remaining 15%. I don't think this is unheard of, but I'll agree that a replication would be nice to eliminate systemic errors. If the images from the ASD and TD sets were taken with different cameras, for instance, that could introduce an invisible difference in the datasets that an AI could converge on. I would expect them to control for stuff like that, though.
What was the problem with that male vs female deep-learning test a few years ago?
That all the males were earlier in the day, so the sun angle in the background was a certain direction, while all the females were later in the day, so the sun was in a different angle? And so it turned out that the deep-learning AI was just trained on the window in the background?
100% accuracy almost certainly means this kind of effect happened. No one gets perfect, all good tests should be at least a "little bit" shoddy.
Definitely possible, but we'll have to wait for some sort of replication (or lack of) to see, I guess.