You know what I mean, right? The way some people just have this
look about them that, for no reason you understand or can help, irks you. I suppose there could be two slightly different things here: maybe some people just have an annoying expression (a haughty peer, a smug half-grin...) that bespeaks an underlying annoying personality - but perhaps some people who aren't really annoying when you get to know them simply have a set of features that, through no fault of their own, just tend to annoy people.
Exhibit A: the bird M&S have flogging their clothes at the moment, Lily Cole:
She's nearly as annoying-looking as Sophie Ellis-Bexter, who I am reliably informed is at least as annoying as she looks. Lily, on the other hand, might be perfectly bearable in real life (notwithstanding the fact that she's a model, of course).
Exhibit B: Jerry 'Fucking' Seinfeld!
In his case, the annoyingness of his face is merely the icing on the cake of an extremely annoying voice and persona.
So whose face rubs you up the wrong way? Over to you...