The decision drops the German to third place, behind team-mate Lewis Hamilton and Red Bull's Max Verstappen and cuts his points lead over Hamilton to one.
Mercedes were found to have gone beyond the degree permitted to help a driver solve a technical problem.
They broke a rule saying: "The driver must drive the car alone and unaided."
Mercedes have lodged an intention to appeal against the decision.
The Formula 1 rules were changed this year to introduce restrictions on how much teams can help drivers while they are out on track.
Teams are allowed to intervene if the car has a "critical problem if a failure of a component or system is imminent and potentially terminal".
Race stewards found Mercedes were allowed to give Rosberg instructions on the settings he needed to resolve the problem.
But they should not have responded to his subsequent question on whether he should then change gear so the car went straight through seventh gear.
Rosberg's engineer Tony Ross replied to the German's information that he had a gearbox problem with the words: "Driver default 1-0-1, chassis default 0-1, chassis default 0-1. Avoid seventh gear, Nico, avoid seventh gear."
Rosberg then asked: "What does that mean, I have to shift through it?
Engineer: "Affirm Nico, you need to shift through it. Affirm, you need to shift through it."
Both Mercedes F1 team boss Toto Wolff and his opposite number at Red Bull, Christian Horner, questioned the need for the rules restricting driver assistance.
Horner said: "The rule is rubbish. It doesn't make a great deal of sense. But the rules are the rules.
"The cars are technically very complex and you can understand why Mercedes would want to give that message to keep their driver running.
"It's a team sport and the cars are a lot more complicated than they were even four years ago for the drivers to work out what they should and shouldn't be doing.
"The question going forward is are these rules right for F1? That's a different question."
Wolff, speaking before the decision to penalise Mercedes, drew parallels between this situation and one in last weekend's Austrian Grand Prix, when Force India did not tell their driver Sergio Perez that he had a critical brake problem and the Mexican crashed out on the final lap.
Wolff said: "We see that those rules maybe need a re-think between the FIA and the teams to maybe go more into detail on what is allowed or not because not communicating at all, you might as well unplug the radio and throw it out of the car. It is part of driving since a long time."
However, the rule is a pet project of FIA president Jean Todt and he has so far been deaf to questions about its suitability.
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.

She met the country's Princess Stephanie and Prime Minister Xavier Bettel at the capital city's museum of modern art, Mudamon.
The Duchess was given a guided tour of the exhibitions by British artists Sir Tony Cragg and Darren Almond.
The treaty, signed in 1867, confirmed Luxembourg's independence and neutrality.
On the trip Kate, who is a patron of the National Portrait Gallery, met people who worked in the arts and charitable sectors in Luxembourg.
She also spoke to Oscar-winning director Laurent Witz, who won an academy award in 2014 for the Best Animated Short Film, Mr Hublot.
Before leaving Luxembourg she was introduced to schoolchildren, aged five and six, from St George's International School.
The UK's deputy ambassador to Luxembourg said: "We have a good strong bilateral relationship and always have with the Luxembourgers, they still very much recognise the role we had in liberating them during the war and we have a very strong relationship between the two financial centres particularly, and that's something we are very anxious to continue and maintain.
"There are also close ties between the two royal families, with three generations of the Luxembourg royal family having been educated at our military academy at Sandhurst."
Nico Rosberg and Mercedes have been penalised 10 seconds for breaking radio transmission rules during the British Grand Prix.
[NEXT_CONCEPT]
The Duchess of Cambridge has visited Luxembourg on her second solo trip, to mark 150 years of the Treaty of London.