Prior to college, I only spoke 3 languages (Persian/Spanish/English) and didn't have this problem at all, since all three were learned when younger.  However, in college, I took Portuguese for Spanish native speakers and got conversationally fluent really fast.  But, as a consequence, it seems I've started stuttering in English, while knowing the word in Persian instead.  I wonder if this is related to my Portuguese...