LocalLLaMA

4 readers

4 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago

MODERATORS

communick@poweruser.forum

Any M2 ultra reviews? (alien.top)

submitted 2 years ago by EvokerTCG@alien.top to c/localllama@poweruser.forum

6 comments fedilink hide all child comments

So I'm considering getting a good LLM rig, and the M2 Ultra seems to be a good option for large memory, with much lower power usage/heat than 2 to 8 3090s or 4090s, albeit with lower speeds.

I want to know if anyone is using one, and what it's like. I've read that it is less supported by software which could be an issue. Also, is it good for Stable Diffusion?

Another question is about memory and context length. Does a big memory let you increase the context length with smaller models where the parameters don't fill the memory? I feel a big context would be useful for writing books and things.

Is there anything else to consider? Thanks.

you are viewing a single comment's thread
view the rest of the comments

[–] SomeOddCodeGuy@alien.top 1 points 2 years ago (4 children)

M2 Ultra user here. I threw some numbers up for token counts: https://www.reddit.com/r/LocalLLaMA/comments/183bqei/comment/kaqf2j0/?context=3

Does a big memory let you increase the context length with smaller models where the parameters don't fill the memory?

With the 147GB of VRAM I have available, I'm pretty sure I could use all 200k tokens available in a Yi 34b model, but I'd be waiting half an hour for a result. I've done up to 50k in CodeLlama, and it took a solid 10 minutes to get a response.

The M2 Ultra's big draw is its big RAM; its not worth it unless you get the 128GB model or higher. You have to understand that the speed of the M2 ultra doesn't remotely compare to something like a 4090; CUDA cards are gonna leave us in the dust.

Another thing to consider is that we can only use ggufs via Llamacpp; there's no support for anything else. In that regard, I've seen people put together 3x or more Tesla P40 builds that have the exact same limitation (can only use Llamacpp) but cost half the price or less.

I chose the M2 Ultra because it was easy. Big VRAM, and it took me less than 30 minutes from the moment I got the box to be chatting to a 70b q8 on it. But if speed or price are a major consideration, moreso than level of effort to set up? In that case the M2 ultra would not be the answer.

[–] aikitoria@alien.top 1 points 2 years ago (2 children)

Is it not possible to port ExLlamaV2 to metal? At least on a 4090, it's much (much) faster at processing the input than llama.cpp

[–] SomeOddCodeGuy@alien.top 1 points 2 years ago (1 children)

I imagine there's a lot of work to do so, but I can't imagine it's impossible. Probably just not something folks are working on.

I don't particularly mind too much, because the quality difference between exl2 and gguf is hard for me to work past. Just last night I was trying to run this NeuralChat 7b everyone is talking about on my windows machine in 8bpw exl2, and it was SUPER fast, but the model was so easily confused; before giving up on it, I grabbed the q8 gguf and swapped to it (with no other changes) and suddenly saw why everyone was saying that model is so good.

I don't mind speed loss if I get quality, but I can't handle quality loss to get speed. So for now, I really don't mind only using gguf, because it's perfect for me.

[–] aikitoria@alien.top 1 points 2 years ago

Hmm, I didn't notice a major quality loss when I swapped from mistral-7b-openorca.Q8_0.gguf (running in koboldcpp) to Mistral-7B-OpenOrca-8.0bpw-h6-exl2 (running in text-gen-webui). Maybe I should try again. Sure you were using comparable sampling settings for both? I noticed for example SillyTavern has entirely different presets per backend.

Still need to try the new NeuralChat myself also, I was just going to go for the exl2, so this could be a good tip!

load more comments (1 replies)