It can look dumb, but I always had this question as a kid, what physical principles would prevent this?
The problem is that when you push an object, the push happens at the speed of sound in that object. It's very fast but not anywhere near the speed of light. If you tapped one end of the stick, you would hear it on the moon after the wave had traveled the distance.
For example, the speed of sound in wood is around 3,300 m/s so 384,400/3,300 ~= 32.36 hours to see the pole move on the moon after you tap it on earth.