this post was submitted on 29 Nov 2023
1 points (100.0% liked)

LocalLLaMA

11 readers
4 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago
MODERATORS
communick@poweruser.forum
1
How fast is 3090 for Codellama 70B 4/8bit? (alien.top)
submitted 2 years ago by Snoo-83094@alien.top to c/localllama@poweruser.forum
5 comments fedilink hide all child comments
 

I am considering purchasing a 3090 primarily for use with Code Llama. Is it a good investment? I haven't been able to find any relevant videos on YouTube and would like to understand more about its performance speeds.

you are viewing a single comment's thread
view the rest of the comments
[–] Herr_Drosselmeyer@alien.top 1 points 2 years ago

With a 3090 and sufficient system RAM, you can run 70b models but they'll be slow. About 1.5 tokens/second. Plus quite a bit of time for prompt ingestion. It's doable but not fun.