LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Is anyone using vLLM for inference? Are there any faster inference framework for LLama based models? (alien.top)

submitted 1 year ago by kitkatmafia@alien.top to c/localllama@poweruser.forum

0 comments fedilink hide all child comments

Given you have a V100 gpu at your disposal - just curious what different folks here will use for inference Llama based 7b and 13b models. Also would you use fastchat along with vLLM for conversation template?

no comments (yet)

sorted by: hot top controversial new old

there doesn't seem to be anything here