LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Does Nvidia's new "System Memory Fallback for Stable Diffusion" also compatible with LLM in general? (alien.top)

submitted 1 year ago by NorthCryptographer39@alien.top to c/localllama@poweruser.forum

2 comments fedilink hide all child comments

Hi all, today Nvidia released a new driver version that appears to allow the GPU to use system memory instead of crashing when it runs out, seen here: https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion

I was wondering if this is compatible with LLM, and how I could enable that (or if it would just work by default).

you are viewing a single comment's thread
view the rest of the comments

[–] a_beautiful_rhind@alien.top 1 points 1 year ago

This works by default and it's bad. The only way I'd accept it is if the overage is by less than a gig and it attempted to clear off the system ram as fast as possible. Otherwise you may as well not use the GPU at all and take the slow ride.