wipe the smile/grin off someone’s face

[Carl Culture]

to stop someone who is annoying you from feeling so happy or pleased

More... (https://www.macmillandictionary.com/us/dictionary/british/wipe-the-smile-grin-off-someone-s-face)