Sure you can. Introverts make up a large fraction of the population (just have a look at this poll[1]). If Agile is reduces the quality of their work, then the "pack" may well be better off without it.
What collaborative face-to-face environment doesn't reduce the quality of introverts' work?
Most work environments today are open, regardless of industry. Strong personalities and herd thinking exist outside of the tech world too.
Pointing the finger at agile might get eyeballs, but you have to keep in mind that open office plans have been popular long before Agile became a mainstream buzzword, probably because they're cheaper than putting up walls. I tend to think the Japanese management fad of the 80s probably had a little to do with it too.
And not every group practising agile is pair programming.
I am no fan of Agile, and as a former introvert, I can say that the only thing that stifled me in my introverted days was my own introversion.
[1]: http://news.ycombinator.com/item?id=943722