That's not really how a social norm works.  But if you're terrified of the consequences of someone not wanting to go to lunch with you, then what you do is that you stop asking them.  And that's the end of it.