512DuncanL

joined 10 months ago
sorted by: new top controversial old
Is it worth using a bunch of old GTX 10 series cards ( like 1060 1070 1080 ) for running local LLM? in c/localllama@poweruser.forum
[–] 512DuncanL@alien.top 1 points 10 months ago

Modded cards are quite easy to obtain in china

Is it worth using a bunch of old GTX 10 series cards ( like 1060 1070 1080 ) for running local LLM? in c/localllama@poweruser.forum
[–] 512DuncanL@alien.top 1 points 10 months ago (2 children)

You might as well use the cards if you have them already. I'm currently getting around 5-6 tokens per second when running nous-capybara 34b q4_k_m on a 2080ti 22gb and a p102 10gb (basically a semi lobotomized 1080ti). The p102 does bottleneck the 2080ti, but hey, at least it runs at a near usable speed! If I try running on CPU (I have a r9 3900) I get something closer to 1 token per second.

Is there a technical reason that distributed LLMs don't exist? in c/localllama@poweruser.forum