this post was submitted on 07 Feb 2024
194 points (95.3% liked)
Technology
59472 readers
3684 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I'm afraid to even ask for the minimum specs on this thing, open source models have gotten so big lately
I think I read somewhere that you'll basically need 130 GB of RAM to load this model. You could probably get some used server hardware for less than $600 to run this.
Unless you’re getting used datacenter grade hardware for next to free, I doubt this. You need 130 gb of VRAM on your GPUs
So can I run it on my Radeon RX 5700? I overclocked it some and am running it as a 5700 XT, if that helps.
To run this model locally at gpt4 writing speed you need at least 2 x 3090 or 2 x 7900xtx. VRAM is the limiting factor in 99% of cases for interference. You could try a smaller model like mistral-instruct or SOLAR with your hardware though.