I solved this by adding a layer for games.
Since I use Dvorak, games usually meant reconfiguring every game key layout, and with Factorio, that's every key, and key combination. So I made a QWERTY layer and bound a layer switch to pressing both shift keys at the same time. I just toggle between the layers between games and non games.