DavidSJ

joined 1 year ago
sorted by: new top controversial old
Macs with 32GB of memory can run 70B models with the GPU. in c/localllama@poweruser.forum
[–] DavidSJ@alien.top 1 points 11 months ago

There will hopefully be more optimizations to speed this up.

Speculative, Jacobi, or lookahead decoding could speed things up quite a bit.