The US-based campaign group says Kigali's reputation for being tidy comes at a price.
HRW says it has documented the illegal detention of thousands, including street children and sex workers.
Rwanda's Justice Minister Johnston Busingye denied there were unofficial detention centres in the country.
Mr Busingye added HRW "seeks to spread falsehood and speculation" but that the allegations will be investigated.
He said the centre where people were being taken was for short stays only before long term rehabilitation.
HRW says in its report, based on research carried out over four years, that people are transported from the capital's streets to a vast former garage which was used to interrogate genocide suspects after 1994.
The organisation says the rounding up of street vendors, beggars, homeless people, street children, sex workers and petty criminals reflects an unofficial policy of keeping those considered undesirable away from the public eye.
HRW says detainees are held there in deplorable conditions for periods ranging from a few days to several months.
Former detainees told HRW of routine beatings for actions as trivial as talking too loudly or not standing in line to use the toilet.
Carina Tertsakian, who put together the report, says the Rwandan authorities should immediately close the detention centre and release all the detainees.
"If some of these detainees are suspected of committing a criminal offense they should be brought to justice according to the law," she added.
Rwanda's justice minister told HRW in 2014, and said again on Thursday, that the place where people were taken provided rehabilitation and avoided "unnecessary incarceration".
"While HRW insists that Rwanda should charge drug addicts and other criminals with serious crimes that carry jail terms, the country has instead chosen to focus on rehabilitating and reintegrating them to offer the chance for a better life," he said.

Although Hibs lost on penalties after a 1-1 aggregate score, the experience was one Laidlaw had not dared dream of.
He last played in a 5-4 loss to Annan Athletic in April 2015 while on loan at Elgin City from Raith Rovers.
"It's a big difference playing at Borough Briggs compared to playing away to Brondby," said the 24-year-old.
Having returned to Raith, where he failed to dislodge first-choice keeper Kevin Cuthbert, he suffered a torn kidney in an innocuous training ground incident last December.
"There was a concern that, with the severity of the injury, I might never get back to play football," acknowledged Laidlaw.
"It took six months to get back and I'm just delighted that I'm fit again.
"My main aim was to get back to training but obviously I'm delighted I got the chance to play and I think I did myself justice."
Laidlaw's inclusion came at the expense of Otso Virtanen, whose first-minute mistake in the first leg led to the costly loss of an away goal to Brondby.
"I feel for Otso," added Laidlaw. "I've been there myself many a time when I've made a mistake and dropped out the team. But I know he's a really good goalkeeper.
"I see him in training every day and I really rate him. He's not in the Finnish under-21 team for nothing so he'll bounce back from it, definitely."
Laidlaw produced two outstanding saves against Brondby to help his team take the Danes to penalties.
And though that should be enough to ensure he keeps the number one jersey in the short term at least, it's not something he's had a chance to reflect upon.
"I'm not really looking into that," he stressed. "I'm just glad I played and did really well.
"I'm not looking at the next few games or the start of the [league] season. I'm just concentrating on doing well in training and taking it from there."
Rwandan police are arbitrarily detaining vulnerable people picked up on the streets of the capital, according to Human Rights Watch (HRW).
[NEXT_CONCEPT]
Ross Laidlaw admits he had to pinch himself as he made his Hibs debut in their Europa League second qualifying round second leg win over Brondby.