That argument carries enough weight that EURONCAP safety rating requires physical buttons for key features:
"Under the Safe Driving section of Euro NCAP’s safety rating, the key driving controls – indicators, windscreen wipers and washer, hazard warning lights, horn and eCall emergency button – should be operated by a direct physical input rather than hidden within a menu on a touchscreen. "
The "it doesn't effect me" crowd is of the same kind that says, that controlling a phone during driving, doesn't effect them or that they can drive just as safe after non-excessive alcohol consumption.