Nelson Jobim is the third minister to lose his job since Ms Rousseff took office in January - a turnover that has strained her governing coalition.
He is reported to have called one fellow minister a "weakling" and described others as "idiots".
Mr Jobim will be replaced by the former Foreign Minister, Celso Amorim.
Nelson Jobim is one of several ministers who also served under Ms Rousseff's predecessor, Luiz Inacio Lula da Silva.
He is said to have been angry that Ms Rousseff overruled him on a multi-billion dollar contract to buy fighter jets.
In recent weeks he been reported as making a series of critical remarks about fellow ministers.
First he said at an opposition event that he was surrounded by "idiots".
Then he said in a television interview that he had voted for Ms Rousseff's opponent Jose Serra in last year's presidential election.
The final straw appears to have been a magazine interview, parts of which have been leaked, in which he reportedly called another minister a "weakling".
Mr Jobim's resignation is the third to shake Ms Rousseff's government since she took office on 1 January.
Last month, her Transport Minister Alfredo Nascimento resigned over a corruption scandal in his department, though he denied any wrongdoing.
And in June her chief of staff, Antonio Palocci, stepped down in the face of questions about his rapid accumulation of personal wealth.
President Dilma Rousseff of Brazil has replaced her defence minister after he made disparaging remarks about other senior officials.