The specification only seems to say that message content are encrypted, making no mention of encrypting any other data than message content.

Correct, what's not encrypted are things like typing notifications, read markers, recipients. Which was my whole point: this is inferred easily by the server anyway: it hosts your account and your contacts list already, it routes your messages to recipients and across the whole network. You can't really operate without this level of trust. Neither in XMPP nor in Signal.

Also, just because you self host it doesn’t make the unencrypted (meta)data any less dangerous.

You seem to hold a very naive take on all of this. This is the basis of federation. In a centralised system (Signal), everyone must trust that the one provider to act (and keeps acting) in good faith. Federation loosens this by having you trust a provider of your choice, and by giving you the ability to move on otherwise. Zero-trust is only theoretically achievable with peer-to-peer, but we have yet to come-up with a system that is performant and efficient enough at scale to be deemed usable. P2P networks often resort to edge gateways to do some caching or connection brokering, and at that point you are back to the same tradeoffs as with federation, only with more steps and worse results.

That just makes your server the point of failure.

Always was, always will be. Like I said, remove the server and you are onto something… Make it work well, and you will be the first, and do the world a great service.

By your logic, why encrypt at all?

E2EE is one mitigation against one type of threat. Not a silver bullet.