Now that I'm thinking of it, that can get out of hand rather quickly. Hmmm, I'm thinking that since it says modifiers and not specifically ability modifiers to the stealth skill, I'm thinking it is referring more specifically to armor check penalties since it is something that even as a team will be an issue. Fortunately that is only a -1 from the looks of it, which is more than made up for by your other stealthy comrades each contributing their own roll.