Man, that shirt is gay

Last night, I saw some gay guys. Nothing really unusual here, except that last night, for whatever reason, I had an urge to tell one of them, "Man, you look really gay."
I resisted the urge, of course. It just seemed so offensive. Sometimes I tell my friends they look gay, but I'd never say that to a stranger without expecting some kind of confrontation.
See, and we're delving deep into human nature here, straight guys don't like it when people think they're gay. Not even a little bit.
But if you're gay, why would you care? If you're an out-of-the-closet homosexual who is dressing in a manner designed to advertise your homosexuality, would it be offensive if someone merely recognized it?
Hey, I can tell by the way you're dressed that you're a homosexual.
That's a little weird, maybe. And maybe you're unnecessarily stating the obvious. And maybe it's a really bad conversation starter (unless perhaps you yourself are gay, in which case it might just be a really bad pickup line), but I'm not sure why it would be offensive, although I'm fairly sure it would be.
As I've thought about this further, I think this instance, as most social instances go, depends on the phrase and the source.
First is the word, "gay." It's an innocent word, really. Almost totally neutral. The use of the word itself is acceptable at all times. But straight people like me have traditionally used the word to insult other straight people (like me). Much of the time, when a straight guy says "gay" he isn't even referring to homosexuality. He's using it as a much more ambiguous insult. "You're gay," "You're stupid," "You're bad at Halo." It's all the same.
So I understand how using the word "gay" can be insulting. However, the only people who can possibly use "gay" as an insult are straight people, especially straight men. A gay man telling another gay man he looks gay is redundant, but not insulting.
This brings me to my next question: What if you substituted a synonym for "gay" in place of "gay."
I can tell by the way you're dressed that (You're a homosexual/you like men/you participate in an alternative lifestyle).
Would that be offensive? I don't know. Maybe somebody out there can help me answer this question, because I don't know any gay people I could ask and I don't want to try it out on a stranger.