The microcode in integrated CPUs took care of routing your inputs and outputs to where they need to be, and triggering the various arithmetic operations as desired.

In the transition from plugboards to programmed sequence control the thing that took over the task of routing values between registers, through the ALU, and to/from IO ports was the control unit. Microcode being one way to implement functionality in the control unit.

One other approach was to use what was basically a finite state machine, implemented physically in-circuit. The output of that FSM was fed into a series of logic gates along with the current instruction value, with the output of that combination being connected to the control lines of the various CPU elements. Thus the desired switching/routing behavior occured.

Modern chips are really complicated hybrids of microcode and a ton of interacting finite state machines. Especially in x86 complex or less commonly used instructions will be implemented in microcode, whereas simple/common instructions will be implemented by being "hardwired", somewhat similar to the FSM technique described above (although probably more complicated).