Officials said they would instead encourage political dialogue between Burundi's opposing sides.
Burundi's President Pierre Nkurunziza had fiercely opposed the AU plan's to send peacekeepers.
His decision last April to seek a third term in office has led to ongoing violence and fears that Burundi is sliding into ethnic conflict.
At least 439 people have died and 240,000 have fled abroad since last April, the UN says.
The AU could have deployed troops without Burundi's consent - a clause in its charter allows it to intervene in a member state because of grave circumstances, which include war crimes, genocide and crimes against humanity - but it would have been the first time it had done so.
Top AU diplomat Ibrahima Fall said such a move would have been "unimaginable".
Burundi on the brink
Tit-for-tat killings spread fear
Burundi's football-playing president
Presidents who cling to power
10.4m population
50 years - life expectancy for a man
2nd poorest country in the world
85% are Hutu, 14% Tutsi
300,000 died in civil war
AU Peace and Security Council chief Smail Chergui said, after the bloc's meeting in Ethiopia: "We want dialogue with the government, and the summit decided to dispatch a high-level delegation."
Earlier this week, human rights group Amnesty International published satellite images it said were believed to be five mass graves near Burundi's capital, where security forces were accused of killing scores of people in December.
A fact-finding mission by the AU has reported arbitrary killings, torture and the "closure of some civil society organisations and the media".
Mr Nkurunziza is the former leader of a Hutu rebel group, who has been in power since a 2005 peace deal. Both the government and the opposition are ethnically mixed.
Ethnic conflict between Hutus and Tutsis in the 1990s claimed an estimated 300,000 lives.
Find out more about Burundi
The African Union has abandoned its plan to send 5,000 peacekeepers to help restore stability to troubled Burundi.