this post was submitted on 13 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
NVidia H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLM (github.com)
submitted 10 months ago by rihard7854@alien.top to c/localllama@poweruser.forum
23 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[โ€“] aliencaocao@alien.top 1 points 10 months ago (1 children)

Batchsize 1024 though...not for personal use case

[โ€“] Herr_Drosselmeyer@alien.top 1 points 10 months ago

Obviously. There aren't many people in the world with 50k burning a hole in their pockets and of those, even fewer are nerdy enough to want to set up their own AI server in their basement just for themselves to tinker with.