LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Hardware question: combining a 3090 and a p40 (alien.top)

submitted 11 months ago by Noxusequal@alien.top to c/localllama@poweruser.forum

5 comments fedilink hide all child comments

As the title says when combining a p40 and a rtx 3090 a few use casese come to mind and i wanted to know if they could be done ? greatly appreciate your help:
first could you run larger modells where they are computed on the 3090 and the p40 is just used for vram offloading and would that be faster then system memory ?

Could you compute on both of them in a asymetric fashion like putting some layers on the RTX3090 and fewer on the p40 ?

Lastly and that one probably works you could run two different instances of LLms for example a bigger one on the 3090 and a smaller on the p40 i asume.

you are viewing a single comment's thread
view the rest of the comments

[–] Tiny_Arugula_5648@alien.top 1 points 11 months ago (2 children)

no absolutely not.. not how you described it. the issue isn't about RAM it's about the numbers of calculations that need to be done. With GPUs you need to load the data into VRAM and that is only going to be available for that GPUs calculations it's not a shared memory pool. So load data into the p40 it will only be able to use that for it's calculations.

Yes you can run the model on multiple GPUs. If one of those is very slow with lots of RAM then the layers you offload to that card will be processed slowly. No there is no way to speed up calculations. VRAM is only making the weights readily available so you're not constantly loading and unloading the model weights.

[–] Noxusequal@alien.top 1 points 11 months ago (1 children)

the question is is that still faster then system memory or not ?

[–] ReturningTarzan@alien.top 1 points 11 months ago

It's not. If the only thing you're using the P40 for is as swap space for the 3090, then you're better off just using system RAM, since you'll have to swap via system RAM anyway.

[–] Hoppss@alien.top 1 points 11 months ago

This is not true, I have split two separate LLM models partially across a 4090 and a 3080 and have had them both run inference at the same time.

This can be done in oobabooga's repo with just a little tinkering.