Dear DC,
In the first place, why don't you rotate wearing all your watches so that you don't feel guilty at all.
Of course each of us have very different personalities but if you continuously reach for a few pieces and put off wearing the others, it may be a sign that you should dispose those you always put off wearing, unless they are deemed to be precious to be exposed to damage and not because you have no more love for them.
Without knowing your preferences and whether you lead an active or passive lifestyle (work/pleasure), its hard to imagine what sort of watches would suit you, besides your personal taste.
I'm sure each of us would have an idea of what he would let go given our personal preferences.
I think you need to ask yourself whether you can be a 1 or 2 or 3 or 4 watch guy or you like to collect whatever watches that you are attracted to. Given tat you have 6 watches but yet feel that you are neglecting some of them, I'd guess that you can be a 2/3 or 4 watch kind of guy. If that is accurate then why not have one dress watch for formal functions, one for work, and another for outdoor/casual weekend.
Between the Speedy and IWC, I'd choose one for outdoor/casual use but I suspect the IWC would be hardier and you can always put in on a water resistant strap since you find the bracelet uncomfortable.
The Milgaus can be kept for those formal occasions or for work (if you are office based)
Sell the rest and get the GS..
Some people I know wear the same watch day in and out for years. For them, if they appreciate the fine things in life I'd recommend them a once and for all top drawer watch be it PP, AP, VC, ALS, etc.
Other people like me, get bored wearing the same watch more than a day or two and like rotating each of my watches daily. So for me having 20-30 watches is manageable and having just 3 -5 watches even top drawer would be boring.