James Lynch, a spokesman for the UN refugee agency (UNHCR), told the BBC that passengers saw smugglers being paid after the boat was intercepted.
Australian Prime Minister Tony Abbott on Friday admitted using "creative" strategies to stop migrant boats but refused to go into detail.
The country's immigration and foreign ministers denied payments were made.
"The boat that was rescued by the Indonesian navy on 31 May - we have interviewed the 65 passengers and they have said that the crew received a payment," said Mr Lynch.
He said the passengers - 54 from Sri Lanka, 10 from Bangladesh, and one from Myanmar - were transferred to a customs boat for four days "before being put on two boats and sent back to Indonesia".
He added: "Mr Abbott says the reason is to save lives because people risk their lives on the boats, and I agree, but I'm not sure putting them back on boats and sending them back is the answer."
Mr Lynch said Australia had a responsibility under UN treaties, which it had signed, to see if any of the migrants were in need of international protection.
He accused Australia of "sending the wrong signal" to other countries in the region - including Myanmar, Indonesia, Malaysia and Thailand - which the UN is trying to persuade to allow migrants to disembark.
Indonesia's foreign ministry said it was "very concerned" about the alleged payment.
The Indonesian navy said it intercepted the boats on their return and arrested the crew, who said they had each been paid A$5,000 ($3,900; Â£2,500) to turn back.
Local police chief Hidayat told AFP news agency: "I saw the money with my own eyes."
Speaking to Radio 3AW on Friday morning, Mr Abbott refused to deny that a payment had been made, saying simply that "creative strategies" had been developed to stop the migrant boats.
"We have stopped the trade and we will do what we have to do to ensure that it stays stopped," he said.
Under Australia's controversial policies, no migrants and asylum seekers are allowed to reach its territories by boat. They are instead intercepted at sea and turned back or taken to detention facilities on the island nation of Nauru and on Manus Island in Papua New Guinea.
Australia asylum: Why is it controversial?

The prosecutors, from Spain's top criminal court, allege Neymar and his father hid the true value of the player's multi-million-euro transfer from Santos to Barcelona.
A Brazilian investment fund that owned 40% of Neymar's sports rights argues it was deprived of its full share.
In previous interviews, Neymar, 24, has denied any wrongdoing.
Barcelona said it paid €57m (£43m) for Neymar in 2013, with the player's parents, Neymar da Silva Santos and Nadine Goncalves da Silva Santos, receiving €40m and his former club, Santos, €17m.
But investigators say the fee was closer to €83m and Barcelona concealed part of the deal. The club also denies any wrongdoing.
The Brazilian third-party investment fund, DIS, alleges it was financially harmed by the transfer when it received €6.8m of Santos's €17m fee and argues it was deprived of its full share.
Neymar and his parents, along with Barcelona's ex-President Sandro Rosell and President Josep Maria Bartomeu, had all given statements at an earlier court hearing in February.
Speaking to O Globo TV in Brazil in February, Neymar said: "Before saying nonsense - that we hid this, or hid that - they should prove it.
"My father is doing everything to ensure that I can just focus on playing football. He handles the books. But in the moment when you see someone you love suffering, it starts to hurt."
Neymar and his father also face accusations in a separate case of defrauding the Spanish tax office, while the player faces a tax evasion investigation in Brazil. He denies wrongdoing.
Neymar's move to Barcelona FC has been a big success on the pitch. The striker became part of a goal-scoring trio with Lionel Messi and Luis Suarez.
In 2015, Neymar helped the club win five trophies and came third in the Fifa Ballon d'Or competition. He also scored in the Uefa Champions League final against Juventus.
He won the Spanish league and cup double with Barcelona in 2016.
Migrants on a boat headed for Australia have told the UN that the crew was paid by the Australian navy to turn back.
[NEXT_CONCEPT]
Spanish prosecutors have called for Brazil and Barcelona football star Neymar to be tried for fraud.