Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 10 months ago

MODERATORS

communick@academy.garden

[D] Do you obsessively watch your models train? (alien.top)

submitted 9 months ago by TehDing@alien.top to c/machinelearning@academy.garden

60 comments fedilink hide all child comments

I find myself watching tensorboard more than working- just wondering if others who have fallen into this pattern have words of advice wrt productivity

you are viewing a single comment's thread
view the rest of the comments

[–] timo_kk@alien.top 1 points 9 months ago (6 children)

I mark the expected duration of my experiments both in a google calendar as well as a project journal. That way, I know when it's "time" to check in on the runs.

I also use Weights & Biases, which sends me a mail if something crashes so I can check up when I really need to.

Curve watching is just a waste of time, you should train yourself to get out of the habit even if it's difficult for you.

[–] deepneuralnetwork@alien.top 1 points 9 months ago

Oooh, the calendar idea is a great tip. I’m going to borrow this from you.

[–] Zemeniite@alien.top 1 points 9 months ago

Instead of relying on Weights & Biases sending me an email I’ve implemented alerts being sent to either a Slack or a Discord channel. They are always sent on error with the error message. I also receive a message after a predetermined interval of training with metrics.

Just another idea if someone doesn’t want to rely on W&B

[–] 0ctobogs@alien.top 1 points 9 months ago (1 children)

I use pushover.net to send my phone a push notification when something noteworthy happens. Just drop a little function in my trainer loop. Extremely useful and you can programmatically set the notification message so it'll tell me what my numbers are when I'm at lunch or whatever.

[–] timo_kk@alien.top 1 points 9 months ago

Dude that's slick af

[–] LoyalSol@alien.top 1 points 9 months ago (1 children)

Curve watching in the initial steps of training is important, but once you get stable behavior time to go to the kitchen, grab some coffee, and do something else with the rest of the day.

[–] lumin0va@alien.top 1 points 9 months ago

No it isn’t. Has anyone here ran more than one experiment at a time? Clearly not because you can’t curve watch hundreds of runs at the same time. Anything that can be achieved by curve watching can be easily automated.

[–] maleits_gavatxos@alien.top 1 points 9 months ago (2 children)

If people wrote documentation instead of watching the training progress chart get updated: *flying cars*

[–] LtFr0st@alien.top 1 points 9 months ago

lol

[–] Weird-Field6128@alien.top 1 points 9 months ago

I am stealing this, sorry

[–] blearx@alien.top 1 points 9 months ago (1 children)

I wondered about weights and biases. Does it go well with tensorflow? Are you liking it so far?

[–] timo_kk@alien.top 1 points 9 months ago

Yeah, I'm liking it. It has strengths and weaknesses like every tool. The biggest advantages for me are:

Dead easy to use.
Very good for collaborations: You can set up teams or share reports.
Good features for analyzing results: You can filter, group etc.

Disadvantages:

The interface becomes slow when you've uploaded lots of runs to the same project. That's why I usually split runs into multiple projects depending on the hypotheses that I want to verify.
Exporting data from the server is not trivial. Once you've figured it out though you can re-use your code.

The pros outweigh the cons for me.