LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Realistically, how far i can push my current PC? (alien.top)

submitted 11 months ago by constanzabestest@alien.top to c/localllama@poweruser.forum

14 comments fedilink hide all child comments

Title essentially. I'm currently running RTX 3060 with 12GB of VRAM, 32GB RAM and an i5-9600k. Been running 7B and 13B models effortlessly via KoboldCPP(i tend to offload all 35 layers to GPU for 7Bs, and 40 for 13Bs) + SillyTavern for role playing purposes, but slowdown becomes noticeable at higher context with 13Bs(Not too bad so i deal with it). Is this setup capable of running bigger models like 20B or potentially even 34B?

you are viewing a single comment's thread
view the rest of the comments

[–] sebo3d@alien.top 1 points 11 months ago

Don't quite know about 34B and beyond as i never tested it on myself, but you can more or less easily run a 20B model with these specs. I also have a 3060 with 32gigs of RAM and i get around 3Tokens/ second while generating using u-amethyst20B(I believe this is the best, or at least the most popular 20B model at the moment) Q4KM after offloading 50 layers to GPU.