Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[D] Why choose an H100 over an A100 for LLM inference? (alien.top)

submitted 1 year ago by faschu@alien.top to c/machinelearning@academy.garden

14 comments fedilink hide all child comments

What are the benefits of using an H100 over an A100 (both at 80 GB and both using FP16) for LLM inference?

Seeing the datasheet for both GPUS, the H100 has twice the max flops, but they have almost the same memory bandwidth (2000 GB/sec). As memory latency dominates inference, I wonder what benefits the H100 has. One benefit could, of course, be the ability to use FP8 (which is extremely useful), but I'm interested in the difference in the hardware specs in this question.

you are viewing a single comment's thread
view the rest of the comments

[–] 3DHydroPrints@alien.top 1 points 1 year ago (1 children)

H100 was additionally specialized to have higher performance for transformer models. I think it is about 8x faster than a A100 for transformers, but don't nail me down on it

[–] Gurrako@alien.top 1 points 1 year ago (2 children)

At first I thought that number was almost unbelievably high. It appears it can be 8x faster when using FlashAttention and a multi-GPU setup. Without multi-gpu and flash attention, it is a bit more than 2x faster.

Source: https://lambdalabs.com/blog/flashattention-2-lambda-cloud-h100-vs-a100

[–] 3DHydroPrints@alien.top 1 points 1 year ago

Thanks for clarifying:)

[–] TwistedBrother@alien.top 1 points 1 year ago

Sure but isn’t it the case that the H100 is what can sustain such a high throughput system whereas A100s are generally independent?