This seems like a very brilliant and almost obvious idea, is there a reason why this method wasn't a thing before? Besides the PCIe bandwidth and storage speed requirements.
this post was submitted on 30 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
Because it wouldn't be any faster than doing CPU inference. Since both CPUs and GPUs are already waiting around for data to process. It's that i/o that's the limiter. This changes none of that.
Is here a better way to use bigger models than can fit in RAM\VRAM? I'd want to try 70b or maybe even 120b but I only have 32\8gb.
70b? Q4, llama.cpp, some layers on gpu.
Might need to run Linux to get the system ram usage low enough
If it works it's a miracle.
Given my M1 Max's 400GB/s memory bandwidth, what would be the bottleneck for this on Apple Silicon? Disk speed? Is it possible to get this running on Metal?
There's no point to it. Since if it's too big to fit in RAM, it would be disk i/o that would be the limiter. Then it wouldn't matter if you had 400GB/s of memory bandwidth or 40GB/s. Since the disk i/o would be the bottleneck.
This technique is actually really useful for batch processing.
I.e. if you run 100 generations and reuse the layer while it is loaded that will go much faster than the total serial time.
If we can fit 1 layer at a time, can we do 3 or 4 at a time? A bit bigger but a bit faster than 1 at a time. Or am I dreaming?
That doesn't make much of a difference. You still have to transfer the whole model to the GPU for ever single inference step. The GPU only saves you time if you can load the model (or parts of it) once and then do lots of inference steps.
Would you choose speed over quality? Personally I prefer quality. So this is a great project! Because in real life, quality usually takes time…
Hey there! I think this is doing offloading?
If so, it's not a new thing. Check out https://huggingface.co/docs/accelerate/usage_guides/big_modeling for a guide with code and videos about it
Was going to say the same XD
I think it becomes faster to run on CPU than that.
one of those cases where proving something can be done doesn't make it useful. This has to be one of the least efficient ways to do inferencing. Like the people who got Doom running on a HP printer. Great you did it but it's the worst possible version.
I feel the 70 B model itself won't fit my storage ( SSD ) since I have only 50 GB left.