Statistics gained under the Freedom of Information Act show nine councils have reduced how much they plan to spend.
It has led to concern that vulnerable groups are denied support some carers call "light at the end of the tunnel".
The Welsh Government said it was up to councils to ensure they had necessary respite placements.
According to figures provided by 21 of the 22 local authorities to BBC Wales, cuts have been made to defined respite budgets in Blaenau Gwent, Ceredigion, Neath Port Talbot, Newport, Swansea and Torfaen.
In Carmarthenshire, Conwy and Monmouthshire, the overall residential care budget, which includes provision for respite, has been cut.
In several other counties the budget for respite in 2011-12 was up on last year, but in some cases represented a cut on previous years.
No figures were provided by Anglesey council.
The figures show Swansea Council has cut its internal provision respite budget for older people and younger adults by over 80% in the last two years.
In 2009-10 it spent over £680,000, but this year the budget is less that £108,000.
Director of social services Chris Maggs admits the council is operating in a tough financial climate, but insists the changes will improve choice.
"We're faced with some incredible challenges as local authorities, and we obviously need to look at how we use all of our resources." said Mr Maggs.
The council has decided to transfer money from respite care into its own residential homes, who will provide short breaks in the future.
"What we're doing is re-using our money," he said.
"Whilst it is true that we're reducing the total amount of money, we're also improving the range of services we have by using our existing services in a different way, so it maximises what we've actually got."
But the potential changes have angered some families who depend on respite care.
Suzanne Hayes, from Rhossili, near Swansea, is a carer for her 27-year-old stepson John, who has cerebral palsy.
She described the six weeks of respite allocation they are given every year as "light at the end of a tunnel", but she feels strongly that the family need more support.
"John needs to have breaks because he can mix with different people," she said.
"We live in a beautiful area, but it is very isolating for John."
The short respite breaks also give the family time to recover while John is away.
"It's not the big things, it's going for a coffee. It's having your own knife and fork in your hands, because we have to feed John."
According to the charity Carers Wales, respite provision is already in short supply, but the cuts being made by local authorities will limit support even further.
"For individual families if a respite service which they've relied on gets cut, that's a real problem," said director Roz Williamson.
"But for most families, the problem is that they can't get access to anything - so the qualification for getting any help from a local authority keeps going up and up and up - so most families get nothing."
A consultation on the future of respite care in Wales has recently closed, following an independent review which said there was "a strong case for funding more outcome focussed respite services in Wales".
The Welsh Government said it recognised "the important role that respite care has in supporting individuals and families at what can be an extremely difficult time".
A spokesperson said: "We are protecting the social services budget through the local government settlement. Funding for social services will increase in cash terms by £35m by 2013-14.
"It is a matter for local authorities to ensure they have necessary respite placements."
Cuts have been made to support for families needing respite care in parts of Wales, figures obtained by the BBC have shown.