It's certainly a valid point. So how do you deal with it? I'm not sure. I guess you have to trust in your own sincerity--it's hard to know on what other terms you could proceed. Not being ashamed of your own past is an important part of that. Everyone struggles to come to an understanding of the world, and no one succeeds in a complete sense. So failing at this a part of your humanity.
The other thing is, at some point, you've done so many about turns it's hard to remember what direction you were originally facing, and it becomes less important.