LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

Is there a technical reason that distributed LLMs don't exist? (alien.top)

submitted 10 months ago by chinawcswing@alien.top to c/localllama@poweruser.forum

31 comments fedilink hide all child comments

Why is there no analog to napster/bittorent/bitcoin with LLMs?

Is there a technical reason that there is not some kind of open source LLM that we can all install on our local host which contributes computing power to answering prompts, and rewards those who contribute computing power by allowing them to enter more prompts?

Obviously, there must be a technical reason which prevents distributed LLMs or else it would have already been created by now.

you are viewing a single comment's thread
view the rest of the comments

[–] JackRumford@alien.top 1 points 10 months ago (2 children)

It’s terribly inefficient in many ways. Data centers with best GPUs are the most efficient hardware and energy wise. They are often built in places with access to cheap/green energy and subsidies. Also for research/development cash is cheap, so there’s little incentive to play with some decentralized stuff which adds a level of technical abstraction + needing a community. Opportunity cost wayyy outweighs running this in a data center for the vast majority of use cases.

[–] ColorlessCrowfeet@alien.top 1 points 10 months ago (1 children)

some niche community uses where the budget is none and people will just distribute the electricity/GPU cost

Aren't there a lot of people who don't run their GPUs 24/7? That would put the marginal cost of equipment at zero, and electricity costs what, something around $1/W-yr?

[–] TheTerrasque@alien.top 1 points 10 months ago (1 children)

Transferring the state over the internet so the next card can take over is sloooow. You'd want cards that can take a lot of layers to minimize that.

In other words, you want few and big gpu's in the network, not a bunch of small ones.

[–] ColorlessCrowfeet@alien.top 1 points 10 months ago

Yes, for actually dividing models across machines, which was the original idea. I'd shifted to a different (and less technically interesting) question of sharing GPUs without dividing the model.

For dividing training, though, see this paper:

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient

[–] Prudent-Artichoke-19@alien.top 1 points 10 months ago

Distributed inference IS indeed slower BUT its definitely not too slow for production use. I've used it and it's still faster than GPT4 with the proper cluster.