I thought it was depending on the species. Like iirc Santa's reindeer are all female as the males shed their antlers in the winter, and the females keep theirs.

But I think you're right for regular deer indeed, antlers are male only.