Firms that have not implemented a quota of female directors will have to leave some vacancies unoccupied.
Some leading German business figures have criticised the new legislation.
Similar measures have been introduced in other European countries including Norway, Italy and the Netherlands.
"This law is an important step for equality because it will initiate cultural change in the workplace," Chancellor Angela Merkel, who initially was against the reforms, told parliament on Wednesday.
"It has been decided and it is coming. We can't afford to do without the skills of women."
Women's Affairs Minister Manuela Schwesig told public radio on Wednesday that she did not expect many positions to go unfilled because of a shortage of female candidates.
"There are enough women who are qualified to do these very important jobs," she said.
Correspondents say that the new rules will affect about 100 listed companies with employee representatives on non-executive supervisory boards. Another 3,500 firms will in future have to publish gender-equality targets.
While quota supporters welcomed Wednesday's developments as groundbreaking, critics including German Industries Federation chief Ulrich Grillo said that a quota system was counter-productive.
He argued that voluntary schemes were far more effective in getting more women into top jobs.
More than 80% of German boardroom positions are occupied by men, Deutsche Welle reported, even though roughly 40% of the federal cabinet is female.
Of the the 160 most important publicly traded companies, women make up 17.4% of supervisory boards and only 6.1% of management boards, Spiegel Online reported.
It says however that Germany's laws on 'co-determination', which guarantee employees seats on supervisory boards, have made some progress in narrowing the gender gap.
Chancellor Merkel's cabinet is due to approve the quota bill on 11 December, and subsequent parliamentary approval is expected to be a formality.
Last year the EU Commission said that it was considering forcing companies to increase the number of women on their boards.
Germany's biggest companies have been ordered to ensure that 30% of supervisory board positions are held by women from 2016 under a law agreed late on Tuesday by the governing coalition.