Matrix definitely is federated.
You ran into the trap of taking "fediverse" at face value. It neither invented nor monopolizes federation. E-Mail is federated and has nothing to do with the fediverse. Wikipedia's page on federation lists the very internet itself as the prime example.
Not implementing ActivityPub doesn't mean Matrix isn't federated.
It's not like we haven't had public downvoting in the fediverse yet. We already found out in the past.
Kbin had public downvotes and the main effect was that people put more thought into their downvotes. While I assume it wasn't without abuse, I've never seen it brought up as an active problem, just something that could theoretically happen.