The Milton Keynes-based 32-year-old was a 2012 bid ambassador and Games participant, but questioned its legacy.
"The country was so united in 2012 and everyone was going sport crazy, but it seems strange that no one's really capitalised on it," he told BBC Sport.
"Not only table tennis, but I'd like to see the other sports you don't see."
Figures released by Sport England show that while football, cycling, cricket and athletics have seen increased participation, badminton, bowls and basketball have experienced declining numbers.
Baggaley is England's most successful male table tennis player of all time at the Commonwealth Games, and believes the legacy of London 2012 has resulted in mixed fortunes for his sport.
"I think recreationally the 2012 legacy has been successful," he said. "My brother is a table tennis coach and he has masses of people coming to his sessions now.
"It's been incredible, but at the top level table tennis isn't seen. It needs big events on TV which will push it forward as a big sport."
Baggaley won the World Championship of Ping Pong earlier in 2015, and is ready to embark on the 2015-16 season with the Rio 2016 Olympics less than a year away.
"I've just come back from training in China. I feel sharp and I'm raring to go with the confidence of winning the Ping Pong tournament," he said.
"I want to give the 2016 Olympics a shot. I'll have to play the best table tennis of my life to see if I can get there.
"Age isn't really a concern as long as you're fit and healthy. I'm sharp, I feel fresh and I've been working physically hard."

The firm, one of the world's biggest fashion retailers, reported net profits of 3.6bn Swedish kronor ($423m; Â£284m), up 36% from a year earlier.
The company said its collections had been "well-received" in the December-to-February period.
H&M plans to open about 400 new stores in the 2014-15 financial year.
"We have made a very good start to 2015 - in terms of both sales and profits," said H&M chief executive Karl-Johan Persson.
"Our attractive customer offering and strong expansion both through stores and online, as well as our work on continuous improvement, are among the reasons for increased market share gains and good profits," he added.
However, the company warned that it could be hit by currency movements, noting that "the strong US dollar will affect our sourcing costs going forward".
H&M makes most purchases in US dollars, and has a large share of sales in euros, so some analysts are concerned that margins will be affected by the dollar's appreciation against the euro.
In the first quarter, sales increased by 15% in local currencies, and by 25% in Swedish kronor, to 40.2bn kronor.
The company said it planned to open new stores in Taiwan, Peru and Macau in the first half of 2015, and in South Africa and India in the second half.
H&M also said it planned to open online markets in Portugal, Poland, the Czech Republic, Romania, Slovakia, Hungary, Bulgaria and Belgium in the spring, and Switzerland in the autumn.
London 2012 Olympic table tennis player Andrew Baggaley says it is "strange" that minority sports have not capitalised on the Games.
[NEXT_CONCEPT]
Swedish fashion retailer Hennes & Mauritz (H&M) has said "strong" store and online expansion led to improved sales and profits in the first quarter.