Marshall Leonard is being held by police after throwing a bomb into one of the department stores in Tupelo, Mississippi, early on Sunday.
Mr Leonard is known in the city for the 4ft (1.2m) flag he flies over his car.
Nobody was hurt or injured by the improvised device that produced only a loud bang but no damage.
"An employee was sitting in the vestibule taking a break. He told the employee to run - that he was going to blow the place up," said Police Chief Bart Aguirre.
The suspect then threw the newspaper-wrapped package into the store - around 0130 local time (0630 GMT).
Walmart, along with several other retailers, pulled the battle flag and other memorabilia from their shelves following the killing in June of nine black worshippers at a church in Charleston, South Carolina.
The man charged with the killings was pictured holding the flag.
Explosives experts said that the package contained enough explosive material to damage the store, but its poor assembly rendered it ineffective.
He was arrested by police about a half-hour later when his silver Mazda adorned with stickers of battle flag and the Mississippi state flag - which itself features an inset Confederate battle flag - was seen running a red traffic light.
"He's a strong supporter of keeping that flag flying," the police chief said. "This is his way of bring attention to that".
Mr Leonard, who lives alone and is unemployed, does not currently have a lawyer, but will be appointed one by a judge if it is deemed he cannot afford one.
The Confederate battle flag became a potent symbol for the southern states fighting the Civil War as they sought to break away from the union.
It is seen by some as an icon of slavery and racism while others say the banner symbolises their heritage and history.

The move was initially criticised by Colombia, which complained that it had not been consulted.
But Colombian Defence Minister Luis Carlos Villegas has now published a review hailing the results.
The number of murders in the border area has dropped by 10%, he said.
More than 150,000 gallons of petrol and 400 cattle have been seized by Colombian forces since Venezuela's border security operation was launched on 19 August, added Mr Villegas.
"We have reached a new era of co-operation with Venezuela's defence minister," he said.
The report was presented to President Juan Manuel Santos and his cabinet during a meeting in the border department of Arauca.
Mr Villegas praised the results of the operation launched unilaterally by Venezuela, but said Colombia was already successfully tackling criminal activities in the area.
"I can affirm today that the border with Venezuela on the Colombian side is safer now than a year ago," he said.
"And I can undoubtedly affirm that the border is safer today than 60 days ago."
Venezuela surprised Colombia on 21 August when it deployed more than 1,000 troops and closed the main crossings along the 2,200km (1,400-mile) border.
The move followed a shootout in which several Venezuelan soldiers were wounded by smugglers.
Other troops were taken to the area in the following days to tackle crime and smuggling.
Venezuelan President Nicolas Maduro said that for many years Colombia had overlooked the problem, which was hurting the Venezuelan economy.
Items subsidised by Venezuela's socialist government, including cheap petrol, are smuggled and sold at huge profit on the Colombian side of the border.
Mr Santos recalled his ambassador to Caracas shortly after the operation was launched, complaining about the treatment of Colombians who lived illegally in Venezuela.
More than 1,500 of them were expelled and thousands more fled in the following days.
"I agree that criminal organisations working in the border area are a big problem, but the best way to deal with it is by working together," said Mr Santos.
The two presidents met in Ecuador on 17 September and agreed to "a progressive normalisation" of their common border.
A man apparently angered by Walmart's decision to stop selling Confederate battle flags has been accused of bombing one of the retailer's stores.
[NEXT_CONCEPT]
The Colombian government has acknowledged that crime and smuggling have gone down since Venezuela deployed thousands of troops along their common border two months ago.