The midfielder slid a 30-yarder narrowly wide in the fourth minute but otherwise his team-mates produced little to trouble Tavarelli early on.
However, Jomo Sono's men started the second half more brightly, with Sibusiso Zuma forcing a save from Tavarelli and shooting over in the opening minutes.
And they refused to buckle when Paraguay doubled their lead with another piece of brilliance from Arce.
The Brazil-based defender curled a superb free-kick in off the underside of Arendse's crossbar 10 minutes into the half.
But South Africa were soon back in the game.
Mounting pressure culminated in Teboho Mokoena's shot being deflected into his own net by Struway.
That left Paraguay sweating and their decision to defend their lead in depth cost them dear.
South Africa went close through a Radebe header before Tavarelli made his rash challenge to concede the crucial penalty.
Manchester United's Fortune had never scored an international goal but confidently thumped home the spot-kick to spark South African celebrations.
Paraguay: Tavarelli, Arce, Gamarra, Ayala, Caceres, Caniza, Alvarenga (Gavilan 66), Acuna, Campos (Morinigo 72), Struway (Franco 86), Santa Cruz.
Subs Not Used: Baez, Bonet, Cardozo, Chilavert, Cuevas, Paredes, Sanabria, Sarabia, Villar.
Booked: Caceres, Caniza, Tavarelli, Franco.
Goals: Santa Cruz 39, Arce 55.
South Africa: Arendse, Nzama, Carnell, Aaron Mokoena, Radebe, Fortune, Sibaya, Tebeho Mokoena, Zuma, Issa (Mukasi 27), McCarthy (Koumantarakis 78).
Subs Not Used: Vonk, Lekgetho, Mngomeni, Mnguni, Pule, Nomvete, Buckley, Marlin, Pienaar, Molefe.
Booked: Aaron Mokoena, Issa, Radebe, Zuma.
Goals: Struway 64 og, Fortune 90 pen.
Ref: L Michel (Slovakia).