Giants, who have failed to win any of their past eight league matches, were undone by Luke Waterworth's 76th-minute try in a topsy-turvy match.
Shaun Robinson, Rhodri Lloyd, Connor Dwyer and Mike Butt also crossed for the part-time Lions, who have only won two league matches in 2017.
Aaron Murphy and Sam Wood were among those to cross for Giants.
Darnell McIntosh and Ollie Roberts also went over for the hosts, but the final whistle was met by boos from the home fans.
Huddersfield were the second top-flight club to lose to Championship opponents in round five, following Hull KR's win over Leigh on Saturday.
Elsewhere in the Challenge Cup on Sunday, Super League side Salford overcame Toronto Wolfpack 29-22.
Dewsbury, Featherstone, Halifax and Barrow also won to book their places in Tuesday's sixth-round draw, which will be broadcast live on the BBC News Channel (18:30 BST).
Swinton head coach John Duffy revealed after the game that his side only had 15 fit players on Saturday, before giving debuts to on-loan Wigan pair Josh Woods and Caine Barnes and out-of-contract Liam Carberry.
"It was a massive achievement for the club but everything goes on the players and the medical staff today - nothing to do with the coaching staff," said Duffy.
"I'd like to thank our medical staff for doing a great job over the last couple of weeks. They have turned it around for us and the boys really dug deep."
Giants had gone into the break 18-16 ahead, after Murphy went over down the left, Wood cut inside to score and McIntosh finished a grubber kick.
However, Rick Stone's side, without around 10 first-teamers, have only won twice all season in the top flight and cracks began to show as Dwyer's score helped put the visitors ahead for the second time in the match, having led 10-6 at one point during the first half.
Giants were back in front when the ball was spread right to Roberts and he found a gap to score, before Waterworth crossed over to send the second-tier side through to the sixth round.
Stone said: "The Championship is full of hardened pros who have been around the block and know know to win games, and we saw a bit of that today.
"There's no denying that is a massive blow for us and everyone is very disappointed - you can feel it.
"There were a few senior players out today and a few long-term injuries and those with niggles, but we had a good enough team out to win but we didn't."
Huddersfield: McIntosh; Simpson, S Wood, Mellor, Murphy; Gaskell, Farrell; Rapira, Leeming, Mason, Smith, Roberts, Dickinson.
Replacements: Wakeman, Ikahihifo, English, O'Brien.
Swinton: White; Robinson, Hankinson, Lloyd, Butt; Woods, Atkin; Bracek, Waterworth, Lever, Dwyer, Jones, Austin.
Replacements: Gore, Bate, Barnes, Carberry.
Referee: Scott Mikalauskas

Denis McAinsh, of Glasgow, will stand trial alongside Alexander Chaplin, 51; Barry Jeffrey, 32; and Michael Chaplin, 50, all of Dundee; and James McGing, 50, also of Glasgow.
All deny charges of supplying cannabis over a period of more than a year.
A warrant was issued for a sixth man - Dean Holmes, 46, of Dundee - after he failed to appear in court.
All six men face an allegation that they were concerned in the supply of cannabis between August 2013 and September 2014 at addresses in Dundee, Berwick railway station, the M90 near Glenfarg in Perthshire, and other locations unknown.
Alexander Chaplin and Dean Holmes face a further charge of producing cannabis at addresses in Dundee, while Barry Jeffrey is also accused of dealing cocaine in the city.
Sheriff Lorna Drummond QC continued the case to a further pre-trial hearing next week.
Championship side Swinton knocked Super League strugglers Huddersfield Giants out of the Challenge Cup fifth round.
[NEXT_CONCEPT]
A 75-year-old is one of six men set to stand trial in Dundee accused of dealing drugs in Scotland and England.