It must, in fact, have been distinctly annoying to the pair of schemers."
There were two schemers, and they must have been annoyed by something.
The tomb built for Humayun was the inspiration for the Taj Mahal.