It must, in fact, have been distinctly annoying to the pair of schemers."
There were two schemers, and they must have been annoyed by something.
Sir James felt pity briefly, as he looked into the girl's face.