LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

submitted 1 year ago by abandonedexplorer@alien.top to c/localllama@poweruser.forum

13 comments fedilink hide all child comments

I am talking about this particular model:

I specifically use: goliath-120b.Q4_K_M.gguf

I can run it on runpod.io on this A100 instance with "humane" speed, but it is way too slow for creating long form text.

These are my settings in text-generation-webui:

Any advice? Thanks

you are viewing a single comment's thread
view the rest of the comments

[–] nero10578@alien.top 1 points 11 months ago

Wait what? I am getting 2-3t/s on 3x P40 running Goliath GGUF Q4KS.