I believe it actually does matter. Every person using Discord brings their friends onto the platform. Users leaving will probably also lead to others in their bubble leaving/following to an alternative.

So lots of people leaving has a bigger effect than just that set of users being absent. Even if you didn't pay for Nitro, some of your friends might have and may stop now.