I know about the setting. Why are you saying that information is sent to Google's servers? As far as I have found, that information is only stored locally on your phone
Edit: If this is just about the fact it's on the phone locally, of course if they have your actual phone they can see it. Signal is end to end encrypted, but it isn't go to be encrypted on each end, otherwise you couldn't read messages. Them getting your actual phone is very different from them intercepting the communication without you knowing