LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

Ok, tell me how stupid this concept is: Using a Blockchain-like system where the process of mining trains a decentralized model. (alien.top)

submitted 10 months ago by VRpornFTW@alien.top to c/localllama@poweruser.forum

4 comments fedilink hide all child comments

I'm not really talking about a cryptocurrency, more of an anonymous collaborative system where your local copy of the blockchain syncs up to the most recent stage of the model's training and dataset, and the process of mining is a pooled effort to find the next best loss reduction.

Training data would obviously be a problem, but I'm sure there are ways to append data to the dataset in the mining process.

Something decentralized, massive, and unstoppable.

Man I'm high...Time to go talk to Mistral.

you are viewing a single comment's thread
view the rest of the comments

[–] __SlimeQ__@alien.top 1 points 10 months ago

This isn't an original concept but the data transfer rates make distributed training like this pretty impractical. Unless you're doing some galaxy brained techniques you will need to distribute like 120gb every step (for a 70B model) and training on an old step is a waste of time. So parallelizing it on internet connected machines is a horrible option.

You don't really need this setup anyways as you can train a 4bit 13B Lora on a gaming pc. This would only be useful for big foundation models, maybe, in which case you're going to get way faster results by just renting some cloud gpus