Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 10 months ago

MODERATORS

communick@academy.garden

[Discussion] What are best practices when building/training very small models? (alien.top)

submitted 10 months ago by Snagnar@alien.top to c/machinelearning@academy.garden

6 comments fedilink hide all child comments

I am currently trying to build small convolutional regression models with very tight constraints regarding model size (max. a few thousand parameters).

Are there any rules of thumb/gold standards/best practices to consider here? E.g. should I prefer depth of the model over width, do skip connections add anything in these small scales, are there any special training hacks that might boost performance, etc?

Any hints or pointers, where to look are greatly appreciated.

you are viewing a single comment's thread
view the rest of the comments

[–] semicausal@alien.top 1 points 10 months ago

In my experience, it honestly depends on what you're trying to have the models learn and the task at hand.

- Spend lots of time cleaning up your data and doing feature engineering. Regulated industries like insurance spend significantly more time in feature engineering than tuning fancy models, for example.

- I would recommend trying regression and random forest models first, or even xgboost