Life is about choices, my man.  You don't get to choose family, for instance.  But you do choose how often you see them, and how you treat them (among other things).  You choose whether to be kind at Christmas, or the guy who complains that he didn't get the Xbox he really wanted.  

With the girl, same thing.  You're choosing to put yourself in a situation where there is no winner.  You can opt out and decide not to be alone with her again, which would be the smart thing.  But instead you're looking for excuses, saying "it's not my fault how I feel".  No, it isn't your fault how you feel.  What is your fault is the choices of how you process and deal with those feelings.