Yes, that M1 Max should running LLMs really well including 70B with decent context. A M2 won't be much better. A M3, other than the 400GB/s model, won't be as good. Since everything but the 400GB/s has had the memory bandwidth cut from the M1/M2 models.
Are you seeing that $2400 at B&H? It was $200 cheaper there a couple of weeks ago. It might be worth it to see if the price goes back down.
There are quite a few Intel projects in AI. There's also the optimized DirectML they made with Microsoft. So anything that supports DirectML should also be well optimized on Intel hardware. Both CPUs and GPUs.
The easiest thing to do is to get a Mac Studio. It also happens to be the best value. 3x4090s at $1600 each is $4800. That's just for the cards. Adding a machine to put those cards into will cost another few hundred dollars. Just the cost of 3x4090s put you into Mac Ultra 128GB range. Adding the machine to put those cards into puts you in Mac Ultra 192GB range. With those 3x4090s you only have 72GB of RAM. Both those Mac options give you much more RAM.
There's no point to it. Since if it's too big to fit in RAM, it would be disk i/o that would be the limiter. Then it wouldn't matter if you had 400GB/s of memory bandwidth or 40GB/s. Since the disk i/o would be the bottleneck.
Because it wouldn't be any faster than doing CPU inference. Since both CPUs and GPUs are already waiting around for data to process. It's that i/o that's the limiter. This changes none of that.
That's where context shifting comes into play. You don't re-evaluate the entire context. You just process the additions.
"Previously, we had to re-evaluate the context when it becomes full and this could take a lot of time, especially on the CPU. Now, this is avoided by correctly updating the KV cache on-the-fly:"
So I guess the default is: 49152
It is. To be more clear, llama.cpp tells you want the recommendedMaxWorkingSetSize is. Which should match that number.
I'm really interested in having a 51B model. I would love something between 34B and 65/70B.
Absolutely. That is a much better way to do it. But that was a recent development. More recent than this thread. That post at github about doing it that way happened 3 hours after I posted this thread.
As per the latest developments in that discussion, "iogpu.wired_limit_mb" only works on Sonoma. So if you are on an older version of Mac OS, try "debug.iogpu.wired_limit" instead.
I guess you've only used a 7B model. IMO, the magic doesn't really start happening until 30B.
That's why Intel is pitching OneAPI. They want it to be the single API to bring everything together. That's why it also supports nvidia GPUs, AMD GPUs, CPUs and even FPGA.