Constantly, yeah. But the idea that the potential is there and can theoretically be used against you at any time without you being able to stop it is the concern. What if there's a backdoor, for example?
I'm not particularly paranoid, but to have an increasing number of devices & gadgets with this potential is naturally going to make a lot of people uncomfortable.