LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Cheapest way to run local LLMs? (alien.top)

submitted 1 year ago by ClassroomGold6910@alien.top to c/localllama@poweruser.forum

13 comments fedilink hide all child comments

Not super knowledgeable about all the different specs of the different Orange PI and Rasberry PI models. I'm looking for something relatively cheap that can connect to WiFi and USB. I want to be able to run at least 13b models at a a decent tok / s.

Also open to other solutions. I have a Mac M1 (8gb RAM) and upgrading the computer itself would be cost prohibitive for me.

you are viewing a single comment's thread
view the rest of the comments

[–] knownboyofno@alien.top 1 points 1 year ago (1 children)

What do you define as "decent" tokens per second? Do you have a budget yet? Do you want to run the 13B at full precision or a quantized precision?

[–] ClassroomGold6910@alien.top 1 points 1 year ago

20 tok/s seems like the minimum I would be sane with lol