Gordon McRorie's penalty and a DTH van der Merwe try put Canada 8-0 in front at half-time.
The lead became 15-0 as Jeff Hassler bounced his way out of some tame tackling to touch down.
But number eight Mihai Macovei went over for two late tries and Florin Vlaicu's penalty won it for Romania.
It is the first time Canada have lost all four of their group matches at a World Cup, while Romania can still finish third in Pool D - securing automatic qualification for the 2019 World Cup - if they beat Italy on Sunday.
A Canada victory seemed on the cards after a one-sided first half. Van der Merwe became the first player from a tier-two nation to score a try in four successive World Cup matches, after scores against Ireland, France and Italy.
The 29-year-old showed the one moment of clinical finishing in an opening period that saw numerous promising situations.
Van der Merwe is set to join Scarlets from Glasgow at the end of this tournament and the Welsh side will no doubt have been impressed by his powerful running and ability.
When Hassler took full advantage of some weak defending to touch down for a converted try after the interval, Canada were in full control at 15-0 up.
Romania's predictable battering-ram approach up front was making little impact against a resolute wall of red.
Countless times the Canadians saw off several phases of play and the lack of progress and defensive pressure resulted in an inevitable Romania handling error as they tried to play the ball out wide in slippery conditions.
Finally Romania seemed to twig, keeping the ball within yards of their huge pack and driving forward from close quarters, with their captain Macovei the inspiration.
He went over to set Canada nerves jangling and, after Jebb Sinclair was sin-binned for bringing down a rolling maul, Macovei's second close-range score made it 15-12.
Vlaicu added the extras and slotted a nerveless late penalty in a game that saw both sides miss several kicks at goal.
Romania coach Lynn Howells: "They've earned the right for a beer but it won't be to a great extent.
"It'll be in the team room and we'll have control over them."
Canada captain Jamie Cudmore: "What do you think it feels like? It's horrible. The guys are extremely disappointed after working so hard."
Canada manager Gareth Rees: "To echo Jamie's comments, it is totally unacceptable to finish a campaign with that performance, for 30 minutes, and to finish a game like that.
"I would like to apologise on behalf of the players that we let those people down."
Canada: Jones, Hassler, Hearn, Blevins, van der Merwe, Hirayama, McRorie, Buydens, Barkwill, Wooldridge, Beukeboom, Cudmore, Sinclair, Moonlight, Carpenter.
Replacements: Pritchard for Jones (74), Trainor for Blevins (49), Mack for McRorie (49), Sears-Duru for Buydens (45), Ilnicki for Wooldridge (67), Dala for Moonlight (63).
Not Used: Piffero, Gilmour.
Sin Bin: Sinclair (73).
Romania: Fercu, Lemnaru, Kinikinilau, Vlaicu, Botezatu, Wiringi, Surugiu, Lazar, Turashvili, Ion, Popirlan, van Heerden, V. Ursache, Lucaci, Macovei.
Replacements: Gal for Wiringi (51), Calafeteanu for Surugiu (22), Tarus for Lazar (74), Radoi for Turashvili (65), A. Ursache for Ion (45), Carpo for Lucaci (77).
Not Used: Burcea, Apostol.
Sin Bin: Fercu (20).
Att: 27,153
Ref: Wayne Barnes (RFU).
Romania scored 17 points in the final 28 minutes as their powerful pack overcame Canada in the biggest comeback in World Cup history.