PaganPasta

joined 1 year ago
sorted by: new top controversial old
[D] Best method of knowledge distillation available? in c/machinelearning@academy.garden
[โ€“] PaganPasta@alien.top 1 points 1 year ago (1 children)

What are your losses/objectives? I'm inclined to say you need a huge amount of data for it as well.

[D] Best method of knowledge distillation available? in c/machinelearning@academy.garden
[โ€“] PaganPasta@alien.top 1 points 1 year ago (3 children)

I've used KD and it has always performed better than training from scratch for me.

Could be a data related issue(scarcity, quality)? Or maybe you need to find good hyper-params.

To which domain are you applying it ?