He was elected to the House of Commons in 1964 and served as MP for Swansea West for 46 years, including when Swansea achieved city status in 1969.
Mr Williams stood down at the 2010 election. At the time he was the MP with the longest continuous service in the House of Commons, earning him the title of Father of the House.
Ed Miliband has led tributes to a "great Parliamentarian".
The Labour leader said Mr Williams had been a "dedicated servant of the people of Swansea".
"Throughout his career - culminating in becoming Father of the House of Commons - he commanded respect from across the political spectrum," Mr Miliband added.
"He will be missed by his many friends in the Labour Party. My thoughts today are with his family."
Fellow Labour MP Geraint Davies, who succeeded Mr Williams in Swansea West, said he died on Sunday night.
He said the former MP would be "dearly missed" by friends, family and the "countless people he helped over many years in Swansea".
"Alan Williams was a good friend for 18 years - a wise counsel and strong champion for Swansea for nearly half a century," Mr Davies said.
"He was a great Parliamentarian who knew where all the skeletons lay and was a great cross-examiner. He was particularly ferocious in select committees but also a kindly and private man."
David Cornock, BBC Wales' parliamentary correspondent
Alan Williams was the sort of politician who enjoyed asking awkward questions.
The recipients were usually those in authority - and sometimes in his own party.
After Tony Blair's landslide election victory in 1997, he was one of the few Labour MPs to speak openly against the Labour government's devolution plans.
Four years later, he was elected chair of the liaison committee, a group of senior MPs whose job it is to question the prime minister directly.
Before then, he had made his name with tough questioning on the watchdog public accounts committee. Whether it was the Royal Household or the Welsh Development Agency spending public cash, the Swansea West MP was forensic in trying to get to the bottom of the story.
He had been a minister under both Harold Wilson and Jim Callaghan in the 1960s and 1970s - in industrial, consumer protection and economic roles.
In his role as Father of the House of Commons he presided over the election of the current Speaker, John Bercow in 2009.
He is also unusual among MPs in having an international footballer for a daughter. Sian Williams played for both Wales and England, before going on to manage the Welsh women's team.
Mr Williams was the longest serving Welsh MP after former prime minister David Lloyd George - with a continuous 45 years, five months, 29 days before he stepped down in 2010.
When he became Father of the House in 2005 he said: "It sounds twee but every day I walk into the House of Commons, I get a kick out of it and I hope all new MPs feel the same."
Swansea East AM and former leader of Swansea council, Mike Hedges, also paid tribute to Mr Williams, a married father of three.
He said: "Alan Williams was an excellent and very hard working constituency MP who always put Swansea first.
"He will be sadly missed by the people of the city".
Neath MP Peter Hain said: "Alan was a supreme House of Commons man, diligent, questioning, challenging, always on top of the issues.
"He was highly respected across all political parties and I was indebted to him for his wise counsel even when we did not agree - for instance he was a critic of devolution where I was a big advocate."
The son of a coal miner, Mr Williams was born in Caerphilly in 1930 and attended Cardiff High School for Boys before later studying at the Cardiff College of Technology and Commerce and University College, Oxford.
He was a lecturer and journalist before entering politics, where he held numerous roles, including shadow Welsh secretary.
The former Swansea Labour MP Alan Williams has died at the age of 84.