this post was submitted on 17 Nov 2023
1 points (100.0% liked)

LocalLLaMA

4 readers
4 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago
MODERATORS
communick@poweruser.forum
1
TabbyAPI released! A pure LLM API for exllama v2. (github.com)
submitted 2 years ago by panchovix@alien.top to c/localllama@poweruser.forum
6 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] Right-Structure-1619@alien.top 1 points 2 years ago

Does anyone know if they expose all the good stuff that Guidance uses for their guided generation and speedup? This plus guidance (kv cache, grammar control, etc) would be fast fast!