A loosely moderated place to ask open-ended questions
Search asklemmy π
If your post meets the following criteria, it's welcome here!
Looking for support?
Looking for a community?
~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~
Retrieval-Augmented Generation (RAG) is probably the tech youβd want. It basically involves a knowledge library being built from the documents you upload, which is then indexed when you ask questions.
NotebookLM by Google is an off the shelf tool that is specialized in this, but you can upload documents to ChatGPT, Copilot, Claude, etc., and get the same benefit.
If you self hosted, Open WebUI with Ollama supports this, but far from the only one.
Thanks, I got NotebookLM working pretty quickly. I think RAG is what I'm after. I'll continue to look.
Dunno why this is downvoted because RAG is the correct answer. Fine tuning/training is not the tool for this job. RAG is.
OP can also use an embedding model and work with vectorial databases for the RAG.
I use Milvus (vector DB engine; open source, can be self hosted) and OpenAI's text-embedding-small-3 for the embedding (extreeeemely cheap). There's also some very good open weights embed modelsln HuggingFace.
I understand conceptually how these work, but I have a hard time of how to get started . I have the model, I know embeddings exist and what they are, and rags, and vector dbs, and then I have my SQL DB. I just don't know what the steps are.
Do you have any guides you recommend?
Milvus documentation has a nice example: link. After this, you just need to use a persistent Milvus DB, instead of the ephimeral one in the documentation.
Let me know if you have further questions.
That's a great start! A lot of it depends on OpenAI, is there any guide you know of that lets me run completely locally? I use TabbyAPI for most of my inference, and happy to run anything else for training
It would work the same way, you would just need to connect with your local model. For example, change the code to find the embeddings with your local model, and store that in Milvus. After that, do the inference calling your local model.
I've not used inference with local API, can't help with that, but for embeddings, I used this model and it worked quite fast, plus was a top2 model in Hugging Face. Leaderboard. Model.
I didn't do any training, just simple embed+interference.
Ah okay, I think that makes sense. Thanks for your input! I'll give it a whirl
All the models will have token limits, especially if you're not paying for API access. You would have to tune a model based on the blog posts, but that's expensive, degrades model quality, and isn't easy to do.
Another thing you could do is have a model index the posts and then retrieve data based on search. The easiest way to do this would be download all the blog posts into a folder, then install cursor.com and open it on the folder. Cursor is for coding, but it will index your folder and then you can ask the model questions. You should be able to get this far with the free trial, but if you have a huge number of blog posts, it still won't work
Deepseek doesnβt seem to have a limit in my experience. In my experience itβs also way smarter! You can also just download it and run it
It will start hallucinating if you overfeed it
And don't feed it after midnight!