this post was submitted on 13 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
We are Higgsfield AI. We have a large GPU cluster and want to finetune your dataset. (alien.top)
submitted 10 months ago by RiskApprehensive9770@alien.top to c/localllama@poweruser.forum
14 comments fedilink hide all child comments
 

Hey LocalLLaMA. It's Higgsfield AI, and we train huge foundational models.

We have a massive GPU cluster and developed our own infrastructure to manage the cluster and train massive models. We constantly lurked in this subreddit and learned a lot from this passionate community. Right now, we have spare GPUs, and we are excited to give back to this incredible community.

We built a simple web app where you can upload your datasets to finetune it. https://higgsfield.ai/

There's how it works:

  1. You upload the dataset with preconfigured format into HuggingFaсe [1].
  2. Choose your LLM (e.g. LLaMa 70B, Mistral 7B)
  3. Place your submission into the queue
  4. Wait for it to get trained.
  5. Then you get your trained model there on HuggingFace.

[1]: https://github.com/higgsfield-ai/higgsfield/tree/main/tutorials

top 14 comments
sorted by: hot top controversial new old
[–] fadenb@alien.top 1 points 10 months ago

What do you consider "large" cluster? How many MW GPU capacity do you operate?

[–] herozorro@alien.top 1 points 10 months ago (1 children)

please do something like this, or provide detailed example, on how an open source framework api can be added to a coder LLM.

how do we prepare the data with code sample, docs, so the coder LLM learns it can can do code completions and answer documentation?

[–] RiskApprehensive9770@alien.top 1 points 10 months ago (1 children)

You can train on any dataset as long as it follows our format.

Soon we'll publish a video tutorial.

[–] herozorro@alien.top 1 points 10 months ago

but what would be the proper formatting example for code? just paste in a bunch of files from a repo? or should be more a cheatsheet format?

[–] mcmoose1900@alien.top 1 points 10 months ago

Well, awesome. Thanks.

I'll be over here assembling some TV show transcripts for a fandom tune.

Out of curiosity, is it a full finetune or a LoRA? What context length?

[–] tortistic_turtle@alien.top 1 points 10 months ago

cool stuff

[–] fctvzm@alien.top 1 points 10 months ago

👏

[–] MaxSan@alien.top 1 points 10 months ago

Just curious why you don't plug it into bittensor? I mean, you get best of both worlds then?

[–] dahara111@alien.top 1 points 10 months ago

Registered. I am very interested and grateful to use it, but I haven't uploaded the dataset to huggingface, so I can't use it yet.

And I don't understand the new learning paradigm that is done just by registering the model and dataset.

What is it that is running behind the scenes?
A very simple snippet OR code would be helpful to understand.

For example

If I give you a model and a dataset, the code will run something like this, and under what conditions will the training be finished.

[–] nomusichere@alien.top 1 points 10 months ago

I don't want to be the party pooper. But this site doesn't seem legit. Have any of you got any info other than provided?

[–] Terrible-Mongoose-84@alien.top 1 points 10 months ago

Hi, is it possible to learn a RAW file? I have about 2gb of artistic text marked up with tags and titles, is it possible to train Mistral on them?

[–] kristaller486@alien.top 1 points 10 months ago

This is awesome! What training parameters are you using?

[–] Scary-Knowledgable@alien.top 1 points 10 months ago

If you collapse does that end the universe????

[–] Grammar-Warden@alien.top 1 points 10 months ago

Spidey-Senses tingling: Caution