I believe it's refreshing to have a variety of sizes in your collection. Certain watches with their history and heritage in mind are meant to be worn big or small (I.e. 45mm+ flieger watches & classic 36mm datejust)
But it's all up to your personal preference, you like small or big sized watches regardless of wrist size - go for it!
With my 6.25 wrist -
Dress 34 to 38mm
Tool/Sports/Versatile 36 to 42mm