Most men assume that everything women do is for the male gaze - because that's what they are taught/sold by advertising, media, porn, etc., - that women exist to please and excite them - but more and more (although it was always true to some extent) women dress for themselves first, and to impress other women second. What men think comes in third, and only usually a man that they are actually interested in. What random men think is largely irrelevant to all but the worst pick-me girls.

I encourage you to say something non-confrontational the next time a rando makes an unsolicited comment - something like, "I like it, that's why I chose it."