Scotland defender Jennifer Beattie lashed in the game's only goal late on after Arsenal had the better chances.
Defeat saw four-time winners Arsenal fail to reach the final for the first time in the cup's six-year history.
City will play at their own Academy Stadium in the final against Birmingham City on Sunday, 2 October, after their 4-0 win at London Bees on Saturday.
After City keeper Marie Hourihan had denied Asisat Oshoala on multiple occasions, the ball deflected straight to Beattie in the Arsenal six-yard box from a Toni Duggan free-kick with 11 minutes remaining and she fired in to win the game.
The winner came moments after Danielle Carter's powerful effort for the Gunners had struck the crossbar, and England captain Steph Houghton cleared the ball off the line late on to deny Arsenal again.
The first half had seen City go close to opening the scoring, with a Jane Ross shot deflected wide, before Arsenal forward Danielle van de Donk had to be withdrawn because of a concerning head injury, caused by a clash of heads.
City - who currently lead the Women's Super League One table and are aiming for a double - previously lifted the Continental Cup in 2014 when they beat Arsenal in the final.
Manchester City and England captain Steph Houghton: "When a chance falls to Jen Beattie, I've got every faith she'll stick it in the back of the net.
"This is special for us because the Continental Cup final was the first trophy we won, it's the only trophy we've ever won and that makes it extra special.
"It was good for me to get back on the line and do my bit at the end, it proved vital because Kelly Smith is always going to create a chance but it was just a relief I got back there to block it.
"We had to be professional and that emotion and adrenaline got us through to the final and we're thankful for that."
Arsenal and England right-back Alex Scott: "On reflection we had most of the chances and we should have been putting them away.
"We should have come away with the win but they came away with a set-piece but apart from that we weren't really threatened.
"I was proud of the girls today, that's as solid as we've been all season but a ricochet on a set-piece falls right to Jen and that's that, it's in the back of the net."
Manchester City Women: Hourihan, Stokes, Beattie, Houghton (C), Bronze, Walsh, Scott, Christiansen, Duggan, Ross, Parris.
Substitutes: Bardsley, Stanway, Asllani, Corboz, Middag.
Arsenal Ladies: Byrne, Scott, Stoney, Henning, Mitchell, Nobbs, Janssen, Van de Donk, Oshoala, Carter, Smith.
Substitutes: Van Veenendaal, Rose, Corredera, McCabe, Devlin, Losada, Natlia.
Manchester City Women reached the Continental Cup final with a narrow home win over holders Arsenal Ladies.