Both locals and immigrants joined the protest with placards saying "Africa Unite" and "Welcome foreigners".
An anti-xenophobic protest also took place in the coastal city of Port Elizabeth.
South Africa's army was deployed to flashpoints earlier this week to prevent further violence.
At least seven people have been killed, 5,000 left homeless and many foreign-owned shops looted since the attacks started about three weeks ago.
"We will defeat xenophobia like we defeated apartheid," the premier of South Africa's Gauteng province, David Makhura, told the crowd in Johannesburg.
Gauteng is the economic heartland of South Africa and includes Johannesburg.
Many unemployed South Africans accuse foreigners of taking their jobs.
South Africa has an official unemployment rate of around 25%.
The BBC's Pumza Fihlani, who was at the scene, said the crowd marched through Johannesburg's Hillbrow suburb, which some describe as Lagos because of its huge Nigerian population.
Protesters sang a sorrowful song, Senzenina, or "What have we done?" It was popular at funerals of anti-apartheid activists during white-minority rule.
"Mandela must be turning in his grave. This is not the South Africa he fought for," Johannesburg resident Vusi Hlongwane told the BBC.
South Africa's freedom was won through such solidarity. "Africa for all" was the message shared by the tens of thousands who marched through the streets of Johannesburg.
A sea of people from different parts of the continent carried banners in support of the day's event. They are hoping the same spirit that led this country to freedom will help bring unity here.
The events of the past few weeks have brought shame to many South Africans and the country.
But many speakers sought to remind South Africans that in the days of apartheid it was the rest of Africa that became a home for this country's leaders, reminding them that its liberation was won with the support of the entire continent.
"They were welcomed with open arms and never treated how some of you are treating them now," one leader said.
As they danced and sang together, calling for an end to the violence that has brought fear to some communities here, a group shouted: "This is for Mandela!" Another chanted "Mayibuye iAfrika" - an old slogan calling for the revival of Africa. For those hours, it seemed that Africa stood united.
In Port Elizabeth, a silent march was held, with some protesters waving a Zimbabwean flag and and carrying placards which said: "Diversity is our strength".
South Africa has faced a backlash across Africa over the attacks.
Official data suggests there are about about two million foreign nationals in South Africa, about 4% of the total population. But some estimates put the number of immigrants at five million.
Most of them are Africans and Asians who came to South Africa when apartheid ended in 1994, with the election of Nelson Mandela as South Africa's first black leader.

The biscuits were taken from Burton's Food Ltd on Ty Coch Way in Cwmbran, Torfaen, on 17 June.
The man, from Liverpool, has been charged with theft and allowing himself to be carried in a motor vehicle which had been taken without the owner's consent.
He appeared at Newport Magistrates' Court on Thursday.
The empty trailer was found in Warrington and vehicles used in the theft have since been seized by police.

The blaze broke out at Denbeath Bowling Club in Buckhaven, Fife, at 12:35.
Police are warning people to keep their windows closed due to the amount of smoke in the air. Surrounding roads are closed and police are diverting drivers away from the area.
The bowling club is next to Denbeath Primary School in the town, however there have been no injuries.
About 30,000 people have taken part in a march through South Africa's main city, Johannesburg, to demand an end to a recent wave of xenophobic attacks.
[NEXT_CONCEPT]
A 24-year-old man has appeared in court charged with the theft of a trailer containing £20,000 worth of biscuits.
[NEXT_CONCEPT]
Firefighters are tackling a huge blaze at a bowling club in Fife.