A suicide attacker detonated a vehicle and a fierce gun battle followed. Officials say the attack is now over and the area has been cleared.
A Taliban spokesman said the group carried out the attack.
It comes a week after it said it was launching its "spring offensive", warning of large-scale attacks.
Tuesday's bombing happened during the morning rush hour in Pul-e-Mahmud, a busy neighbourhood where homes, mosques, schools and businesses nestle close to the Ministry of Defence, other ministries and military compounds.
Soldiers and security officers are reported to be among the casualties, but the majority are civilians.
The blast shattered windows up to 1.6km (one mile) away. The Associated Press quoted a police officer as saying it was one of the most powerful explosions he had ever heard, and he could not see or hear anything for 20 minutes after.
It appears the initial blast cleared the way for Taliban fighters to enter the area - a commonly used tactic.
"One of the suicide attackers blew up an explosives-laden truck in a public parking lot next to a government building," Kabul police chief Abdul Rahman Rahimi told AFP news agency.
"The second attacker engaged security forces in a gun battle before being gunned down," he added.
Hours after the assault was declared over, another blast rocked Kabul. Police told a BBC reporter in Kabul it was caused by an improvised explosive device (IED).
"It was as huge as in the morning," said Afghan MP Elay Ershad. "So everyone [is] scared of this situation."
Casualties are high because the assault happened in a densely packed neighbourhood during rush hour.
Usually the Taliban does not say it is behind such attacks, which cause large numbers of civilian deaths. But today they did because the target was high-profile and it appears that, for them, hitting the target was worth the civilians killed and wounded.
What we hear less of are similar planned attacks that are foiled by the Afghan security forces, and there are plenty. But preventing such attacks altogether is next to impossible.
The country has been at war for three decades, explosives are easily available and bomb-making skills are common.
The Taliban's "spring offensive" is part of a propaganda war. The group fights all year round and so do the Afghan security forces. It is the story of the past 37 years.
In a statement on its Pashto-language website, the Taliban said several militants successfully breached the defences of the National Directorate of Security, Afghanistan's main spy agency which protects high-ranking government officials
It is unclear precisely how many attackers were involved and the Taliban have been known to exaggerate details.
The presidential palace - only a few hundred metres from the blast - condemned the attack "in the strongest possible terms".
"Such cowardly terrorist attacks will not weaken the will and determination of Afghan security forces to fight against terrorism."
A tweet from President Ghani's office's official account suggested the attack "clearly shows the enemy's defeat in face-to-face battle with ANSDF" - the Afghan National Security and Defense Forces.
But correspondents say that in fact the Taliban has gained in strength since the bulk of British, American and other Nato forces left in 2014.
It has also been buoyed by a flood of foreign fighters joining its ranks, and now controls sizeable parts of Afghanistan.

Brighton and Hove City Council has set up a cross-party panel to look into the issue of the "party houses".
There are about 300 in the city, with concerns about the problems they cause most evident in the Queen's Park area.
Chair of the panel Geoffrey Bowden said: "We don't want to be killjoys... but there are responsibilities."
The councillor said the properties were houses "which sit cheek by jowl with families which live there all the time".
"There are some really superb operators, and they do certainly take care
"We need to bottom-out whether these are businesses or not, because if they are businesses then they should be paying business rates, and they should be paying for commercial waste to be taken away," Mr Bowden said.
The panel plans to gather evidence from as many people as possible, including the owners of "party houses", before reporting back to the full council.
Mr Bowden said:  "People having a decadent weekend is one thing, but if you live next door and have to get up for work in the morning you might take a more hardline approach."
At least 28 people have been killed and 329 injured in a huge explosion in the centre of the Afghan capital Kabul, police and officials say.
[NEXT_CONCEPT]
Houses let out to groups for stag and hen weekends are to be the subject of a council investigation amid complaints about noise and anti-social behaviour.