gpt872323

joined 11 months ago
sorted by: new top controversial old
1
What is the major difference between different frameworks with regards to performance, hardware requirements vs model support? Llama.cpp vs koboldcpp vs local ai vs gpt4all vs Oobabooga (alien.top)
 

Is there a comparison of performance between different frameworks? What are the strengths of each?

I was not able to find anything similar. If someone with more knowledge can help put. I hope this would be a good resource for newbies as well as trying to pick a tool. I know there is a lot of overlap between them, but some may perform better natively.

CPU Only Performance GPU + CPU Performance Overall
Lllamacpp
Local AI
Koboldcpp
Oobabooga Text Generation UI
LLM Studio
vLLM
GPT4ALL
RWKV
text-generation-inference by HF n/a

​

3060 Performance with 13b Model in c/localllama@poweruser.forum
1
3060 Performance with 13b Model (alien.top)
 

Is the 3060 enough to be used in a small office for document questions and answers ? What is the best model to use? The cpu and RAM will be 4790k 32 gb.

Cheapest way to run local LLMs? in c/localllama@poweruser.forum
[–] gpt872323@alien.top 1 points 11 months ago

have this same question. I am thinking of a mini pc more powerful than both and price relatively ok. Mini pc not with nuc rather amd or intel mobile series.