Sorry to hear that man.
Well, at least now you can be less careful and truly enjoy the watch, eh? I'm only half-joking , but sounds like an experience almost every watch owners have gone through. Furthermore, the brush metal surface is a scratch magnet compared to polished surface.
Scratches and dings are part of watch owning experience. Try live with it a few more weeks and see if it still bothers you. Otherwise, just wait until you collect more scratches, dings, and further war-scars and have a full service and overhaul in the future or before your warranty expires. You'll get more value for money for the service cost, and the watch will be as good as new or even better.
Some Patek owners send their watch for service and specifically request NOT to polish out some notable scars, which adds personality to the watch.
If you have it changed or polished now, remember that the next scratch is just around the corner... it is inevitable.