I'm struggling to understand why anyone would join the Fediverse if this is a dealbreaker.

ActivityPub is like RSS and Sendmail had a baby that doesn't respect your privacy at all. Every time you post, comment, or even vote, it's blasted out to every server and user within the blast radius. If you delete something, there are dozens of databases and image proxy caches ready to ignore the delete request and host your regrettable post forever.

Every time you send a DM, you have to trust all the admins of both instances involved not to read the messages.

Way more is visible than it appears. With the right API calls people know what you upvoted and downvoted.