this post was submitted on 29 Nov 2023
1 points (100.0% liked)

LocalLLaMA

3 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago
MODERATORS
communick@poweruser.forum
1
How fast is 3090 for Codellama 70B 4/8bit? (alien.top)
submitted 11 months ago by Snoo-83094@alien.top to c/localllama@poweruser.forum
5 comments fedilink hide all child comments
 

I am considering purchasing a 3090 primarily for use with Code Llama. Is it a good investment? I haven't been able to find any relevant videos on YouTube and would like to understand more about its performance speeds.

you are viewing a single comment's thread
view the rest of the comments
[–] flossraptor@alien.top 1 points 11 months ago

With a dedicated 3090 (another card for OS) a 34b 5bpw just fits and runs very fast. Like 10-20t/s. The quality is good for my application, but I'm not coding.