Yeah I can run any open weight models which leave me enough vram to not crash. But its a bit of a gotcha because you also need enough system ram to load the model. I use it to heavily parallelize training tasks.. Honestly, I need to tinker with it more but I'm pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.

Someday TM

[–] dubs@lemmy.dbzer0.com 1 points 6 hours ago

I’m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.

Can you expound a little more on what you mean by this?

I like ollama, but I only really use it to load models and then hit the API. My current issue with them is that they don't seem to support non-text base interaction very well.

[–] tatterdemalion@programming.dev 3 points 14 hours ago (1 children)

I've been thinking about getting a real NAS machine instead of having my gaming PC always on. This looks pretty slick.

[–] TropicalDingdong@lemmy.world 4 points 13 hours ago (1 children)

Yeah I've got the 128 gb Asus one as a processing machine, and honestly, I wish I would have just got it as a server. For my purposes, I need this bad-ass GPU/ CPU much closer to the storage.

Honestly, just a basic 2:1 is fine for me considering I'm rarely if ever running my compute locally.

[–] tatterdemalion@programming.dev 3 points 13 hours ago (1 children)

I'm not sure I need a lot of compute on my NAS. I guess I might as well get as much as I can before it becomes cost prohibitive. Not planning on running any local LLMs on it though, so a GPU is probably overkill.

[–] TropicalDingdong@lemmy.world 4 points 13 hours ago

Yeah I do a ton of machine learning research and like, distance between the database and the place I'm doing the processing really matters. My NAS is on the other side of my house and it would be great to have maybe two NAS, a primary one for like, my movies, personal files, old projects etc, and then this one right next to it for active projects.

Right now I'm having to do it all on my primary machine, which means I often have to freeze updates for long periods of time once I get a particular compute configuration set up for a particular project.