Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[D] Do you obsessively watch your models train? (alien.top)

submitted 11 months ago by TehDing@alien.top to c/machinelearning@academy.garden

60 comments fedilink hide all child comments

I find myself watching tensorboard more than working- just wondering if others who have fallen into this pattern have words of advice wrt productivity

you are viewing a single comment's thread
view the rest of the comments

[–] PrimaCora@alien.top 1 points 11 months ago

For stable diffusion, not as much. Put the settings right and the training is done in 5 minutes. See the result, alter the settings and go again. Those settings are max possible batch size, previews off and saving checkpoint to off. Otherwise training takes 3 times longer or thrashes an SSD if used.

For voice cloning, extensively. As soon as the loss changes or the loss updates stop it has to be killed. Worse is for newer ones like Style TTS, they have a constant VRAM usage up until a random point where it grows infinitely.