LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Handling Generic Queries in RAG Applications (alien.top)

submitted 1 year ago by tinyburger1@alien.top to c/localllama@poweruser.forum

4 comments fedilink hide all child comments

I'm trying to build an application using RAGs. I know how RAGs help ground the responses and all, but how do I handle generic queries from users which have nothing to do with what's stored in my vector database? For example, queries such as: "How many gold medals did China win during Tokyo Olympics?" vs "Parapharse this email for me: ... ". I would assume LLMs without RAGs would do a much better job answering the second question.

How do people usually handle these scenarios? Are there any tools that I can look at? Any help would be greatly appreciated. Thank you.

you are viewing a single comment's thread
view the rest of the comments

[–] tinyburger1@alien.top 1 points 1 year ago (1 children)

But won't it increase the inference time quite a bit? Or are there any GitHub projects to get started with this?

[–] artelligence_consult@alien.top 1 points 1 year ago

THAT is the cost side and that is a NASTY one. It is not only the financial - but it is, as you point out - the response time. And it is NOT just inference, you also have all the lookup that must happen.

But yes. This is where the price is paid that shows that we are still a factor if 10 or 20 away from fast interactive complex data AI.

But - do not worry, we get there ;)

No github I am aware of - people are very happy with their naive little innovation and never see the real problems in their simplistic tests. It is an 80/20 or higher order problem - MOST things work simple, SOME - ah - well ;) YOu also get into the "smalltalk" - you do not want to run a full research cycle when the user input is "Thank you, that was helpful" ;)

That said, really, if AI gets 10x faster (and it looks like hard+software is on the way for more than that) it is easily doable from the time side.