The UK's prison population should be cut by 12,000, a report recommends.
The public back more non-custodial sentences, says the IPPR.
The Institute for Public Policy Research says 5,000 inmates should be in mental health units while another 5,000 should be in drug rehab centres.
A further 2,000 female prisoners serving less than six months should have been given community sentences.
The left-of-centre think-tank says the new Ministry of Justice should make lowering the prison population one of its priorities.
'Expensive and ineffective'
In its report, the group says the majority of British people do not believe prison is the appropriate place for vulnerable offenders.
It says the public would support the use of tougher, more visible community service sentences instead of custodial ones.
IPPR director Nick Pearce said: "Prison is an expensive and ineffective way of warehousing social problems.
"Prison should be used far less in Britain but to greater effect."
He added: "If more drug and mental health treatment was provided outside prisons and women sentenced to less than six months were given community sentences, we could stabilise our prison population to 10% lower than it is today."
The IPPR says it is no coincidence that Britain tops the incarceration league in Western Europe when 67% of British prisoners are caught re-offending within two years of their release.
The prison population in England and Wales currently stands at about 80,000 inmates.
The think-tank also argues that the money earmarked for expanding prison places should be moved into residential drug treatment centres.
And it says that community service orders should be high-visibility "payback" sentences with objectives set by local communities to establish greater public trust in non-custodial punishments.
Paul Cavadino, chief executive of Nacro, the crime reduction charity, backed the report.
He said: "Prison often worsens mental health problems, especially depression and disorders with a depressive element.
"All health authorities should be required to fund psychiatric assessment schemes at police stations and courts to divert mentally disordered offenders from prison into health and social care."
The IPPR's proposals come from a collection of essays that will be published in May to mark the 10th anniversary of Labour's 1997 election victory.