Tomomi Inada, a close ally of Prime Minister Shinzo Abe, stepped down over an alleged cover-up of documents on a controversial peacekeeping deployment.
She was seen as an Abe protÃ©gÃ©, sharing his staunchly nationalist views.
Her resignation comes as the prime minister sees a sharp plunge in public support over alleged cronyism.
The 58-year-old Mrs Inada had already been widely expected to be replaced in a cabinet reshuffle next week that the prime minister hopes will help rebuild his ratings.
The row over the South Sudan peacekeeping mission erupted in December when the defence ministry said peacekeepers' logs for July, when the security situation there was worsening, had been discarded.
But two months later officials said they had found the data on a computer, leading to allegations that the logs had been withheld on purpose.
Peacekeeping missions are controversial in Japan, which has a pacifist constitution. Japanese peacekeepers are only deployed under strict conditions, including that a cease-fire must be in place.
Any change in the security situation in South Sudan could have affected whether the mission would continue.
Mrs Inada was appointed defence minister in 2016. But her months in the position have seen a series of gaffes.
She particularly came under fire for suggesting in a speech that the military wanted voters to back the ruling party.
Mr Abe apologised to the country on Friday immediately after Mrs Inada's resignation.
He also said that for the time being, Foreign Minister Fumio Kishida would add the defence portfolio to his duties.
In his years in power, Mr Abe has taken the controversial position that nation needs to change the pacifist constitution to formalise the existence of the military.
Japan is highly divided on the issue and claims the defence ministry has not been transparent over the South Sudan deployment will likely fuel public scepticism.
The prime minister is also facing plummeting support amid two patronage scandals. He denies wrongdoing, but his approval ratings have fallen below 30% according to recent polls.
The opposition, however, has so far failed to capitalise on his unpopularity. On Thursday opposition leader Renho, who goes by one name, stepped down in order to take responsibility for the opposition's weakness.

22 November 2015 Last updated at 14:58 GMT
Yusf Pirot said he wanted to show people that "not all Muslims are terrorists".
Another video of a blindfolded Muslim man asking for hugs in the aftermath of the Paris attacks has also gone viral.
The woman once seen as on track to becoming Japan's first female prime minister has resigned as defence minister.
[NEXT_CONCEPT]
A video of a blindfolded Muslim teenager in Nottingham giving out free hugs has attracted millions of views on social media.