You seem to have similar setup
Though for gaming I still use it mostly "directly"
Sunshine only when Im bored on work laptop
The VM still outputs to same display and has keyboard/mouse/etc. passed

I've been happy with the setup for the last year and half or so