I think I get what you're getting at.

Being 'hard to read' and awkward can make people uncomfortable, especially if they've witnessed an outburst, retreat, or some seemingly inscrutable reaction. As an example. I catch myself putting people off with a 'neutral frown' in conventionally inappropriate situations, sometimes. I have a tendency to get over-excited and start talking over others or interrupting, a weird contrast to a shy/introverted demeanor.

I also have an issue where I perceive myself as unreliable/untrustworthy, and a tendency not to reciprocate when something personal is shared. That might be interpreted as deliberate when simply I missed the cue (or get distracted by it).

...What I'm getting at is you might be mixing up some self-esteem issues with what people actually perceive interacting with you. You might be mixing up being 'untrustworthy' with simpler misunderstandings. I'm speaking from personal experience, here.