The Most Reverend Justin Welby told the BBC's Songs Of Praise the killings had put a "chink in his armour".
He said his reaction to the attacks had been "first shock and horror and then a profound sadness", heightened because he and his wife once lived in Paris.
The gun and suicide bomb attacks on 13 November, carried out by so-called Islamic State, left 130 people dead.
Prime Minister David Cameron will meet French President Francois Hollande in Paris on Monday to discuss the fight against terror.
They will discuss how to co-operate on counter-terrorism and in the fight against IS in Syria and Iraq, UK officials said.
The archbishop said: "Saturday morning, I was out and as I was walking, I was praying and saying: 'God, why - why is this happening? Where are you in all this?'"
"He said 'in the middle of it' and also in answer from Psalm 56 - 'he stores up our tears in a bottle, none of our sufferings are lost,'" he added.
The archbishop said his shock had been made worse because he and his wife had lived in Paris for five years.
"It was one of the happiest places we have lived and to think of a place of such celebration of life seeing such suffering is utterly heart-breaking."
A bombing campaign against IS, particularly by French air forces, was launched in Syria shortly after the attack, but the archbishop warned against a potentially damaging instant reaction.
"Two injustices do not make justice. If we start randomly killing those who have not done wrong, that is not going to provide solutions. So governments have to be the means of justice," he said.
Archbishop Welby also said the way IS militants had perverted their faith in order to believe their acts glorified their God, was "one of the most desperate aspects of our world today".
Cardinal Vincent Nichols, leader of the Roman Catholic Church in England and Wales, told the programme it had taken him "quite a while" to come to terms with what had happened in Paris.
He said terrorists wanted to make people live in fear, breed hatred and strike divisions in society - but people had to try to resist this.
The terrorists' actions, he said, were a blasphemy against God, but had shown "the goodness of people" in Paris, quoting a newspaper which said: "Terrorists came to shed blood; Parisians stood in line to give blood."
"We really should be solid in our commitment to each other - to stand in the face of this evil," the cardinal said.
Strong action was needed but political and military judgement was required to decide how best to stop them, he added.
Songs of Praise will be shown on BBC One at 16:15 GMT on 22 November, and on BBC iPlayer.
The Archbishop of Canterbury has said the terror attacks in Paris made him "doubt" the presence of God.