This is normally only a problem when conversing with monolinguals. Almost anyone who has expended the immense energy required to learn a second language will understand that language ability and intelligence are completely separate.
That being said, it is a shitty feeling that I have experienced many times.