this post was submitted on 17 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
TabbyAPI released! A pure LLM API for exllama v2. (github.com)
submitted 10 months ago by panchovix@alien.top to c/localllama@poweruser.forum
6 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] panchovix@alien.top 1 points 10 months ago

By the hard work of kingbri, Splice86 and turboderp, we have a new API loader for LLMs using the exllamav2 loader! This is on a very alpha state, so if you want to test it may be subject to change and such.

TabbyAPI also works with SillyTavern! Doing some special configurations, it can work as well.

As a reminder, exllamav2 added mirostat, tfs and min-p recently, so if you used those on exllama_hf/exllamav2_hf on ooba, these loaders are not needed anymore.

Enjoy!