this post was submitted on 27 Nov 2023
1 points (100.0% liked)
Machine Learning
1 readers
1 users here now
Community Rules:
- Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
- Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
- Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
- Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.
founded 10 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I mark the expected duration of my experiments both in a google calendar as well as a project journal. That way, I know when it's "time" to check in on the runs.
I also use Weights & Biases, which sends me a mail if something crashes so I can check up when I really need to.
Curve watching is just a waste of time, you should train yourself to get out of the habit even if it's difficult for you.
Instead of relying on Weights & Biases sending me an email I’ve implemented alerts being sent to either a Slack or a Discord channel. They are always sent on error with the error message. I also receive a message after a predetermined interval of training with metrics.
Just another idea if someone doesn’t want to rely on W&B