Not a massive penalty, it was just that everybody else would be really smug about how uncivilized you were. It could get annoying when you had 5 people doing a performative mockery of a late 19th century aristocratic stereotype in chorus. Lots of "Oh dear" and "I say" and miming of the adjusting of a monocle.