I'm very atheist and yet I'm listening to O come all ye faithful and singing along with my whole chest when they go:

Oh sing choirs of angels, sing in exultation
Oh come, o come ye to Bethlehem
O come and behold Him, born the King of Angels

and it gives me all the joy.