LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

What is the difference between GGUF(new format) vs GGML models ? (alien.top)

submitted 1 year ago by TalhaZubair147@alien.top to c/localllama@poweruser.forum

3 comments fedilink hide all child comments

I'm using llama models for local inference with Langchain , so i get so much hallucinations with GGML models i used both LLM and chat of ( 7B, !3 B) beacuse i have 16GB of RAM.
So Now i'm exploring new models and want to get a good model , should i try GGUF format ??
Kindly give me suggestions if someone using Local models with langchain at production level .

you are viewing a single comment's thread
view the rest of the comments

[–] tortistic_turtle@alien.top 1 points 1 year ago

GGUF won't change the level of hallucination, but you are right that most newer language models are quantized to GGUF, so it makes sense to use one.