LocalLLaMA

This post is an automated archive from a submission made on /r/LocalLLaMA, powered by Fediverser software running on alien.top. Responses to this submission will not be seen by the original author until they claim ownership of their alien.top account. Please consider reaching out to them let them know about this post and help them migrate to Lemmy.

Lemmy users: you are still very much encouraged to participate in the discussion. There are still many other subscribers on !localllama@poweruser.forum that can benefit from your contribution and join in the conversation.

Reddit users: you can also join the fediverse right away by getting by visiting https://portal.alien.top. If you are looking for a Reddit alternative made for and by an independent community, check out Fediverser.

Totally feasible to run LLMs at useful speeds. I'm running a 64gb 10/32 M1 Max. With LM Studio, I typically get

• 3-4 T/s using q5_k_m quants of ~70B models
• 6-9 T/s from q5_* and q6_k quants of ~30G models
• 25-30 T/s from q6_k and q8 quants of 7B models
• around 20 T/s from unquantized fp16 7B models

And this is my daily work and play machine, so I usually have all sorts of browser tabs and applications open simultaneously while running the models. From a fresh boot, it's cool to be able to load an entire model into memory and still be able to do "normal" work without having to use any swap space at all.

Yes, that M1 Max should running LLMs really well including 70B with decent context. A M2 won't be much better. A M3, other than the 400GB/s model, won't be as good. Since everything but the 400GB/s has had the memory bandwidth cut from the M1/M2 models.

Are you seeing that $2400 at B&H? It was $200 cheaper there a couple of weeks ago. It might be worth it to see if the price goes back down.