It also wants company directors to be held responsible for unwanted phone calls from their businesses
The taskforce has recommended that the threshold at which regulators are able to act on complaints should be lowered.
The Nuisance Calls Task Force said cold-callers should not cause "severe distress" to consumers.
Payment Protection Insurance (PPI) cold-callers could face fines of up to 20% of their annual turnover under the proposals.
And Ofcom and the Information Commissioner's Office (ICO) will be able to share information on rogue companies.
Up to one billion unwanted phone calls are received by members of the public in the UK each year, the taskforce said.
Which? executive director and taskforce chairman Richard Lloyd said many British firms were "breaking the law".
He said companies that were contacting people despite their having "opted out" of receiving direct marketing calls were acting illegally.
He told BBC Radio 4's Today programme the way in which the law was enforced had not been good enough.
Mr Lloyd said the trade in personal data, responsible for the vast majority of nuisance calls, was "out of control", adding that the "market in personal data needs properly investigating and sorting out". He also called on regulators to clamp down on the abuse of existing legislation.
"We want to see business with good reputations that aren't keeping a close enough eye on this making sure that there is someone senior on their board who will be held to account if those nuisance calls are being generated by that businesses activity," Mr Lloyd said.
Between April and June this year, 40,000 people complained to the Information Commissioner about unwanted live or automated calls to their phones.
Most focused on accident or PPI claims, as well as some debt consolidation company calls.
The Nuisance Call Task Force said many consumers did not know that they had unwittingly given their consent to be contacted by these companies.
Ed Vaizey, the Minister for Culture and the Digital Economy, said: "For too long, nuisance calls have plagued consumers, often at very inconvenient times of the day and in some cases, leaving vulnerable people like the elderly too scared to answer the phone.
"That's why we're determined to tackle this scourge through the first-ever nuisance calls action plan.
"We've already made progress, including making it easier for Ofcom to share information with the ICO about companies breaking the rules, and we're currently looking at lowering or removing the legal threshold before firms could be hit with fines of up to £500,000."
Companies should be fined £500,000 in order to combat "out of control" nuisance calls, a government taskforce has recommended.