LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

Does Nvidia's new "System Memory Fallback for Stable Diffusion" also compatible with LLM in general? (alien.top)

submitted 10 months ago by NorthCryptographer39@alien.top to c/localllama@poweruser.forum

2 comments fedilink hide all child comments

Hi all, today Nvidia released a new driver version that appears to allow the GPU to use system memory instead of crashing when it runs out, seen here: https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion

I was wondering if this is compatible with LLM, and how I could enable that (or if it would just work by default).

top 2 comments

sorted by: hot top controversial new old

[–] a_beautiful_rhind@alien.top 1 points 10 months ago

This works by default and it's bad. The only way I'd accept it is if the overage is by less than a gig and it attempted to clear off the system ram as fast as possible. Otherwise you may as well not use the GPU at all and take the slow ride.

[–] dampflokfreund@alien.top 1 points 10 months ago

Yes, it's system wide. You can set your prefered way in Nvidia control panel->global settings-> cuda systemem fallback policity.

Driver default is prefer systemmem fallback, which means it's going to offload to RAM instead of crashing when VRAM is full.

No System Mem fallback is basically the old memory management, it crashes once your VRAM is full.