I don't know what to tell you, that has been my experience, and the experience of many others I have discussed it with.

Hating matrix because lots of people are on the flagship server is like hating email because most people use Gmail.

Well for one, yes, I absolutely do hate that most people use Gmail. This i s simply not the case with many competing protocols. For another, no, it's not the same, because Gmail makes no illusions of being private.