k_michael

joined 2 years ago
sorted by: new top controversial old
What exactly is pulling 100-110W when running local LLM? in c/localllama@poweruser.forum
[โ€“] k_michael@alien.top 1 points 2 years ago

I think they all read from the same source, powermetrics, but interpret it in slightly different ways.

What exactly is pulling 100-110W when running local LLM? in c/localllama@poweruser.forum
[โ€“] k_michael@alien.top 1 points 2 years ago

Oh wow, that would be much more than I expected. Could make a lot of sense as the LLM is probably hitting the RAM very hard. Thank you ๐Ÿ™Œ

What exactly is pulling 100-110W when running local LLM? in c/localllama@poweruser.forum
[โ€“] k_michael@alien.top 1 points 2 years ago

I'm not training, just running inference for fun every now and then. This question was mostly just for curiosity. But thank you, you are right!

What exactly is pulling 100-110W when running local LLM? in c/localllama@poweruser.forum
[โ€“] k_michael@alien.top 1 points 2 years ago

That's a good point actually! Thanks

What exactly is pulling 100-110W when running local LLM? in c/localllama@poweruser.forum
[โ€“] k_michael@alien.top 0 points 2 years ago (2 children)

Hm, you are right, I do also remember the anandtech article on m1 max power draw. Maybe the tool really isn't reading reading the draw correctly ๐Ÿค” It's still interesting though, if I run a 3D game on my MBP i draw maybe 65-70W under full load. The LLM must be using some component that the 3D game isn't ๐Ÿคทโ€โ™‚๏ธ

1
What exactly is pulling 100-110W when running local LLM? (alien.top)
 

This is purely out of curiosity, but if anybody has some insights I'd love to hear it.

I am running 70B Q4 models on my M1 Max Macbook Pro (10 CPU, 32 GPU, 64 GB RAM). The lid is closed because I have an external monitor 4K attached via USB-C, so the display won't draw any power.

I am using both llama.cpp and LM Studio, and in both cases I run the LLMs with Metal acceleration.

Now, when running the LLM, I notice that according to iStat Menus my macbook is drawing between 95 and 110W ๐Ÿ˜ฎ

(The fans get loud quickly, just like the good old intel days. But it seems to be able to sustain this)

But how is that possible?

Where is that power draw coming from? The GPU alone is max 45W, and the CPU is something around ~30W max (I forgot the exact value), but it's not even used much. In the screenshot it pulls a meager ~12W. So That's a total of ~57W for CPU+GPU combined. Where do the other 50W+ go?

Where is the additional power draw coming from? I know there are lots of other components here: RAM (probably single digit power draw?), fans, memory controller, etc etc. But we are talking about a large chunk of power.

Does anybody know? :)

https://preview.redd.it/6xxet64ash0c1.png?width=2869&format=png&auto=webp&s=ca3a1f416b9f2764e7143d262a5540fb2d02fa44