Here's the problem with your conclusion, who then decides who gets to vote? By what criteria? How much support would you need? How are you going to take the right away from people? Through violence? Even if there was popular support among some partition of people against everyone else voting, how big would these partitions have to be to be justified? If 20 percent thought the other 80 percent should be subject to their votes alone, is that cool?
Say, it wasnt that extreme... say 97 percent of people believe the other 3 shouldn't vote. 100 percent of those in the 97 percent outside of some some group X or above some minimum attribute Y believe that those in X or those who are lesser than Y should not get to vote. Maybe those people are those with low IQs or those in some cult or those who have been convicted of crimes more than once, etc. The bigger group of people peacefully and democratically narrow democracy and now they hold dominion over the nonvoter for the good of all...
What then stops them from doing the same again and again? You likely think that you will still have power, be a part of the ruling class. But what stops the rest of the voters from doing the same to an arbitrary group that you belong to? What if they say that only land owners should vote? Only those who trace their ancestry back to founding members of rhe nation? Only those who meet a certain level of capitalistic investment in the economy? Only those who wish to preserve the existing culture and way of life? Only those who believe in defending that way of life through force? Only those who belong to a specific religion and belive it has be the model for the governance of the nation? Only those that support their one party leader? Only those who literally worship this leader as a god and hold his dynasty as divine? Each step can be fully justified and will have the precedent set that all they need is the consensus of those who would usurp power from those who would be excluded by the new rules in order to strip them of their vote and any other rights they see fit. The ruling class gets smaller and smaller and more like minded, power hungry, violent, and unconcerned with the welfare of the people.
You see how your position is pretty problematic and opens a giant can of worms of abuse and autocracy?