Rutabaga-Agitated

joined 10 months ago
sorted by: new top controversial old
How to upgrade to the next VRAM breakpoints, and is it worth it? in c/localllama@poweruser.forum
[–] Rutabaga-Agitated@alien.top 1 points 9 months ago

We created a 4x 4090 RTX setup through a mining rig That is 96gb VRAM for round about 10k... does not get cheaper than that. Best compute per cost rn I think.

https://preview.redd.it/nfq4olntq54c1.png?width=1812&format=pjpg&auto=webp&s=a5308bb5eec778072f8d6a394b5243ca33c7fd87

.*

Should corporations and smaller businesses be training, refining, and running local LLMs to avoid the future cost of the new cloud services M$ and Amazon are rolling out? in c/localllama@poweruser.forum
[–] Rutabaga-Agitated@alien.top 1 points 9 months ago

You are right. But if you have a chinese customer for example, there might come up different problems like with NVIDIA and GPUs. Independency is key for a lot of players.

Should corporations and smaller businesses be training, refining, and running local LLMs to avoid the future cost of the new cloud services M$ and Amazon are rolling out? in c/localllama@poweruser.forum
[–] Rutabaga-Agitated@alien.top 1 points 9 months ago (2 children)

Yeah might fit in the US, but not in Europe. Dependencies can lead to problems. Especially when there might be a conflict. I would not want to run important infrastructure that is dependent on US services only.

How to achieve more than 4k context? in c/localllama@poweruser.forum
[–] Rutabaga-Agitated@alien.top 1 points 10 months ago

Does anyone have some hints how to use exllamav2 and extended context length by using GPTQ weights?