Is it permissible to ask a non-Muslim to swear by what he venerates?
Question

There is a man who is married to a Christian woman.
There was a problem between them, and she swore that she did not do a certain thing.
But he did not believe her, and he said to her: Swear by Jesus if you are telling the truth.
He did not intend to venerate Sayyiduna ‘Isa more than he venerates Allah – Allah forbid.
Did he become a disbeliever by saying that?
Praise be to Allah.Firstly:
It is not permissible to swear by anything or anyone except by Allah, may He be exalted, or one of His attributes, because the Prophet (blessings and peace of Allah be upon him) said: “Whoever swears by something other than Allah has committed an act of shirk.” Narrated by Abu Dawud (3251) and by at-Tirmidhi (1535), who said: This is a hasan hadith. It was classed as authentic by al-Albani in Irwa’ al-Ghalil (8/189).
Ibn al-Qayyim (may Allah have mercy on him) said:
Another major sin is swearing by anything or anyone other than Allah, as the Messenger of Allah (blessings and peace of Allah be upon him) said: “Whoever swears by something other than Allah has committed an act of shirk.” The one who said that this is disliked (makruh) has made a serious mistake, because the Lawgiver stated that it is shirk, therefore it is more serious than other major sins."(I‘lam al-Muwaqqi‘in  6/571-572).
This hadith is general in meaning and applies to both Muslims and disbelievers, because the minor issues of sharia – meaning its rulings – are also addressed to the disbeliever. One of the rulings of sharia is that it is not permissible to swear by anything or anyone other than Allah.
Wali ad-Din al-‘Iraqi (may Allah have mercy on him) said:
The correct view on which the majority of scholars are agreed is that the minor issues of sharia are addressed to the disbelievers, so silk is prohibited for them as it is for the Muslims.(Tarh at-Tathrib  3/227).
Based on that, it is not permissible for anyone to ask another person to swear by someone other than Allah, even if the one who is being asked to swear the oath is not Muslim. Therefore it is not permissible to ask him to swear by his god that he believes in and venerates.
The scholars stated that if a disbeliever is required to swear an oath, he should only swear by Allah, such as the Jews, Christians and Zoroastrians, and even the atheists who do not believe in the existence of Allah. They should not be asked to swear by anyone except Allah.
Khalil ibn Ishaq al-Maliki (may Allah have mercy on him) said in his Mukhtasar (p. 228):
An oath to establish rights should be sworn by Allah besides Whom there is no god worthy of worship, even if he [the person being asked to swear the oath] is a Jew or a Christian. End quote.
Al-Kharashi (may Allah have mercy on him) said in his Sharh (7/237):
The well-known view is that the Jew or Christian should say this phrase, just like a Muslim, when swearing an oath.
As for a Zoroastrian, he should swear by Allah only. End quote.
Ibn Qudaamah (may Allah have mercy on him) said:
For both Muslims and disbelievers, oaths should be sworn by Allah, may He be exalted, because no one should swear by anyone except Him, because He, may He be exalted, says (interpretation of the meaning): {and let them both swear by Allah} [al-Ma’idah 5:106]. And because the Prophet (blessings and peace of Allah be upon him) said: “Whoever wants to swear an oath, let him swear by Allah or remain silent.”"(Al-Mughni  14/223).
Al-Haskafi al-Hanafi (may Allah have mercy on him said in al-Durr al-Mukhtar (11/617):
The idol worshipper should swear by Allah, may He be exalted, because he acknowledges Him, even if he worships someone else.
Ibn al-Kamal stated that the Dahris do not believe in Allah, may He be exalted.
I say: Based on that, then how could they swear an oath? End quote.
This question was answered by ‘Ala’ ad-Din in Takmilah Haashiyah Ibn ‘Abidin, where he said:
I say, they should swear by Allah, may He be exalted, because of what was narrated from al-Mabsut in Mi‘raj ad-Dirayah: Free and slave, men and women, evildoers and righteous, disbelievers and Muslims are all the same with regard to swearing oaths, because the purpose here is to pass a judgement, and they all believe that swearing a false oath is prohibited."(Takmilah Hashiyah Ibn ‘Abidin  11/618).
Al-Qadi Shihab ad-Din ibn Abi’d-Dam ash-Shafa‘i said in his book Adab al-Qada’ (1/553, 554):
If the one who is swearing an oath is a Dahri who does not believe in any Creator or God, make him swear by Allah, the Creator, the Provider.
If it is said that that will not deter him from telling lies, so what benefit is there in it?
Our response is that there are two benefits in that:
Firstly, we will be able to pass our judgement concerning them. Allah says (interpretation of the meaning): {And judge, [O Muhammad], between them by what Allah has revealed and do not follow their inclinations} [al-Ma’idah 5:49].
Secondly, he will increase in sin by this oath and will be punished because of the sin, so he may hasten his own punishment by way of (divine) vengeance. End quote.
Thirdly:
If a Muslim asks a non-Muslim to swear by his own god, not intending thereby to venerate that god, then he has done something prohibited, but it does not constitute major disbelief (kufr akbar).
That is because swearing by something other than Allah constitutes minor disbelief (kufr asghar), not major disbelief.
But it may constitute major disbelief if he intended to venerate that god as he venerates Allah, or more.
Usually the Muslim who does that is either unaware of the prohibition on doing that, or he justifies it on the basis of his thinking that it is permissible in the case of a disbeliever, so that he will not tell lies.
In these two cases, the Muslim is not sinning, but after he comes to know the Islamic ruling, he must refrain from doing that, and should not ask anyone to swear an oath by anything or anyone except by Allah.
And Allah knows best.