Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[R] Automatic hallucination detection experiments using SelfCheckGPT NLI on WikiBio (alien.top)

submitted 11 months ago by Separate-Still3770@alien.top to c/machinelearning@academy.garden

2 comments fedilink hide all child comments

Hi everyone,

We have recently written an article on HF’s blog on automatic hallucination detection using inconsistency scoring. The main idea is that hallucinations happen because the task asked at inference is not seen in the training set, which implies low confidence in the next token, therefore, inconsistent samples from the same prompt (https://arxiv.org/abs/2309.13638).

We look at the use of SelfCheckGPT NLI (https://arxiv.org/abs/2303.08896), an example of inconsistency scoring, on WikiBio and found that such a metric has high precision (aka flagged hallucinations indeed are ones) and calibrated recall (high scores = high chance of flagging hallucinations).

https://preview.redd.it/dtptqylv3y2c1.png?width=1189&format=png&auto=webp&s=db623d58e6b24f2f7eee57ff41115f544ee957be

This is quite promising as it could open the way to having AI systems that are more reliable, aka when the task is easy, we let the AI do it. When we detect it’s too hard and the model is hallucinating, we put a human in the loop.

https://i.redd.it/w7p9ciow3y2c1.gif

We have provided:

An article on HF Blog: https://huggingface.co/blog/dhuynh95/automatic-hallucination-detection
A Gradio demo to see the metric in action: https://huggingface.co/spaces/mithril-security/hallucination_detector
A Colab notebook to reproduce our results: https://colab.research.google.com/drive/1Qhq2FO4FFX_MKN5IEgia_PrBEttxCQG4?usp=sharing

We conducted these tests as part of our mission to build Confidential and Trustworthy Conversational AI. You can check out our core project, BlindChat, an open-source and Confidential Conversational AI (aka any data sent to our AI remains private, and not even our admins can see your prompts) at https://github.com/mithril-security/blind_chat/

you are viewing a single comment's thread
view the rest of the comments

[–] Visible-Positive6879@alien.top 1 points 11 months ago

My confusion with this paper is that nowhere in the paper or the many media posts about this paper do the authors show the confounding of this stochasticity on fact-checking -- how do we know that the inconsistency came from hallucination and not stochasticity? Without this causal analysis, I am not sure if their result has any technical footing whatsoever -- curious to know if anyone here has read this paper or has expertise/experience in the area of hallucination checks and can shed light on my confusion here