LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

submitted 1 year ago by abandonedexplorer@alien.top to c/localllama@poweruser.forum

13 comments fedilink hide all child comments

I am talking about this particular model:

I specifically use: goliath-120b.Q4_K_M.gguf

I can run it on runpod.io on this A100 instance with "humane" speed, but it is way too slow for creating long form text.

These are my settings in text-generation-webui:

Any advice? Thanks

you are viewing a single comment's thread
view the rest of the comments

[–] abandonedexplorer@alien.top 0 points 1 year ago (1 children)

Cant believe that worked lol! Thank you so much. The speed increased significantly!

[–] MINIMAN10001@alien.top 1 points 1 year ago

I mean it makes sense The value is chosen we're simply chosen for being a reasonable window at the time.

There was nothing hard coded about them they were simply a range of values that they had set for the UI.

It certainly is interesting though.