Honestly though CPU processing is going to add almost no latency at all.

Almost all latency is sending to the RAM itself, so if you can compress it in a CPU cache before sending it, its nearly free RAM space.

Also systems are fucked fast these days you could probably make something responsive in pure python - other than millisecond timing scenarios like medical and audio stuff

Depends on the workload. This probably would not be great for gaming or audio production, but it might be for video editing and local LLMs