Edit: I misremembered the names of things, see the reply from waggz@programming.dev

Actually removing him from office falls to the senate rather than to congress, and it requires a two-thirds supermajority of senate. That was what prevented him from being removed each time he was impeached - congress impeached him, senate did nothing