LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Cheapest way to run local LLMs? (alien.top)

submitted 11 months ago by ClassroomGold6910@alien.top to c/localllama@poweruser.forum

13 comments fedilink hide all child comments

Not super knowledgeable about all the different specs of the different Orange PI and Rasberry PI models. I'm looking for something relatively cheap that can connect to WiFi and USB. I want to be able to run at least 13b models at a a decent tok / s.

Also open to other solutions. I have a Mac M1 (8gb RAM) and upgrading the computer itself would be cost prohibitive for me.

you are viewing a single comment's thread
view the rest of the comments

[–] ButlerFish@alien.top 1 points 11 months ago

You get charged while to pod is running, and the pod is running until you turn it off on the runpod control panel even if you aren't actually doing anything on there right now.

If you added a volume (cloud hard drive) when you created it then, even when it is turned off, you are paying 10 cents / gigabyte / month to rent that hard drive so your data is still there when you turn it on again.

For niche usecases where it needs to be available but isn't running stuff most of the time like that home assistant I mentioned, look at runpod serverless which is much more fiddly and hard to use but will let you pay essentially per prompt... for playing with LLMs and interacting it's much better to just rent a server and turn it off when you are done.