tornado28

joined 11 months ago
sorted by: new top controversial old
[P][D] A100 is much slower than expected at low batch size for text generation in c/machinelearning@academy.garden
[–] tornado28@alien.top 1 points 11 months ago

A lot of people use CPU for inference. I'm guessing it's because of the batching issue that you describe.