lengyue233

joined 10 months ago

NVidia H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLM in c/localllama@poweruser.forum

[–] lengyue233@alien.top 1 points 10 months ago (3 children)

Yes, batch size is intended for multiple sessions (1024 parallel sessions in the case).

NVidia H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLM in c/localllama@poweruser.forum

[–] lengyue233@alien.top 1 points 10 months ago

Are u going to talk with Yuki at 1024 batch size?