After taking note of Goliath-120b, I suddenly got strangely curious about Horde. Surprisingly, searching for Horde doesn't show many posts, so hopefully someone can answer a few questions:
- What I understood is that I could host something like 13b or 20b, or SD/SDXL, which I can run just fine and fast, and rack up credits overnight for running 70b or 120b LLMs without queue and fast-ish at any moment later. Right?
- If so, how long do prompts on those big models take, more or less, when you have credits to skip the queue? Is it usable? (i.e., how many seconds would it show on SillyTavern?)
- Seeing as I only ever used Oobabooga and SillyTavern, I'm assuming Kobold is more or less a drop-in replacement for Oobabooga, just a backend to the model but everything translates well? If no, what can I expect to lose/get from Kobold as opposed to Ooba?
- Is there a "Horde for r*tards" guide somewhere?
- What do people get from hosting Goliath-120b for others? Don't get me wrong, I appreciate the deep pocket generosity, but is this like a data gathering operation from their point of view?
Thanks for reading this far. There's a good doggo being very comfy hidden in the following period.
Every time I went on Horde, it's typically models people could run on an RTX 4090 or 3090. The problem is I too own an RTX 4090, so I don't see why someone should gain credits when there's no 70B models to spend their credits on.
They seem to have 70b and Goliath (the 120b monstrosity) on there. Currently I only see one 70b on https://lite.koboldai.net/'s list, but the other day I saw a couple Goliaths. I have no idea why would anyone host the 120b other than maybe "crowd-sourcing" a dataset (probably against TOS or something, but why would anyone do it?).