I'm like you in that I never cared about what others thought of me. At first, in the instinctual teen-rebel way, then later on, with the "it's my life, I don't owe you anything" mentality.

There seems to be some kind of "magic timeframe" for this, where you can segue seamlessly from "I DON'T CARE I AM A REBEL and anyway YOU'RE ALL IDIOTS" to "No, really, do whatever; I don't really care" without having to go through an awkward, peer-pressured transitional phase. :D Looks like we were both lucky enough that our personal timelines fell into that sweet spot!

sometimes fandom can be so like other 'normal social groups with the backbiting

This... is probably true. I think I was also lucky that I fell into slash-fandom, and the more mature side of that, pretty early -- I have figured out, by now, that I really missed a LOT of wank-potential and juvenile interactions by heading straight to Harry/Snape once I found my first major fandom. (Well, the first where social interaction was a large part; TPM fandom was mostly mailing lists and feedback, not interpersonal.)

If I'd fallen into the clutches of the Harry/Draco fangirls, or spent a stint in Harry/Ginny normality? I might describe fandom very differently...