Gamescope is a compositor, a self-contained graphical session, meant to run a single application. When it's running in a nested context, i.e. as a window inside another graphical environment, it needs some kind of interface to be able to render the picture onto a window's surface, and to receive inputs from it. The --backend ... option defines which interface will be used to achieve that.

• The wayland backend uses some native Wayland protocols, but it's been broken for a while.
• The sdl backend uses SDL, which is a standalone rendering library often used by Linux-native games like Factorio and Stardew Valley. It simply captures Gamescope's output and renders it onto a window without any Wayland protocol fuckery.
• The drm backend is used when Gamescope runs in a rootful context, i.e. it is the only graphical environment, like big picture mode on the Steam Deck. It uses the Linux kernel's Direct Rendering Manager interface to render directly to the output device.
• The headless backend runs Gamescope without opening a window or showing its output. Everything happens in the background.
• The default is auto, which tries to guess the most optimal backend based on the runtime environment.