Argyle went ahead after just 15 minutes as Carl McHugh headed in Craig Tanner's cross from the right.
Luton almost levelled when Jack Marriott's cross-shot was turned away by Luke McCormick, who then saved Cameron McGeehan's strike but could not keep out Josh McQuoid's header.
However, Brunt's late close-range finish handed Argyle all three points.
Media playback is not supported on this device
Luton Town boss John Still told BBC Three Counties Radio:
"Everything went against us today, the three substitutions, player sent off. We came from a goal down, we've tried to win, I didn't want to try and draw!
"I have no complaints about the late goal, the players have done fantastic. In the other games I would have complaints because they were different situations, we had a proper, up and at them, let's try and win this game attitude."
"It's a penalty, we've seen it, with the greatest of respect the assessor said it was a penalty, no doubt about it, but there's nothing we can do about that. "
Ryan Brunt's injury time strike ensured Plymouth remained top of League Two after a hard-fought victory over Luton.