i think anyone can pull off smaller watches, the problem is that you can't pull them off wearing casual modern clothing, i think modern shoes (brightly coloured, thick soles) especially do not work with smaller watches, if you wear simple clothing, casual but smarter than average, you can easily pull a smaller watch off.
take a look at these photos, all of them are wearing small watches, but they don't look out of place;
you just need to wear clothes that are not obviously modern, and the same advice in reverse would apply if you were trying to pull a vintage suit or some other type of vintage clothing off: don't wear a massive modern steel watch, or anything too busy. it just wouldn't look right.
my collection philosophy is to try not to buy any more.
a significant amount of the watches i own are not ones i purchased myself, but watches that originally belonged to my father, my brother and i think one of them was my great grandfathers watch.
i don't want to buy any more because i don't need any more, i see something that i like, i buy it, and i end up not wearing it after a while, then i see something new that i like, and the cycle continues.