Sujood Ul-Sahw if in doubt while praying
Question

The question is regarding the unsureness of the number of rakaats ones prayed. How would we go about correcting it. Some say at the end of prayer you say salam once and go into sujdah and others say you finish your prayer and than do the sujdah.
This issue is very confusing please explain.
Praise be to Allah.The prostration of forgetfulness (sujood ul-sahw) is required whenever one inadvertently adds extraneous parts or misses parts or is in doubt regarding parts of the arkaan (essential pillars) of the prayer and/or its other requirements. The question posed regards what to do when in doubt regarding the number of raka’aat prayed, and the answer is covered in the following points:
First, definition of doubtfulness: when two possibilities are equally probable without knowing or feeling one is more likely true than the other.
Second, when one doubts after performing salaam: One should disregard such doubt. For example, take the case of one who has completed praying dhuhr then questions after finishing the prayer, "Did I really pray four raka’aat or only three?" Such doubt should be disregarded completely unless it is substantiated by clear and certain indications; otherwise, it opens the door to waswasah (the whispering of Satan) and unwarranted extraneous additions to the prayer.
Third, when one doubts during the prayer; this must fall into one of the following two cases:

One can discern that one of the two possibilities about which one is in doubt is more likely or more certain by virtue of one’s prevalent feeling or most likely inclination: In this case, one should act according to his best assumption and perform the prostration of forgetfulness after the salaam. The evidence for this opinion is what was related by Ibn Mas’ood, may Allah be pleased with him, that the Prophet (peace be upon him) prayed either an extraneous addition or missed an essential part of the prayer (one of the narrators of the hadeeth was in doubt which). After saying ‘salaam,’ it was said to him, "Oh Prophet of Allah, has there been a new change in [the way of performing] the prayer?" He replied, "Why [would you say that]?" They responded, "You prayed in such and such manner." So he bent his legs and faced towards the qiblah and performed two prostrations then performed tasleem (saying ‘salaam’ to the right then the left). After facing towards us, he said, "If something new had been introduced regarding the prayers, I would have informed you about it. To the contrary, I am but a human like you I forget as you forget, so if I forget, remind and inform me. If any of you doubts during his prayer then he should try to discern the more certain and correct case, then complete the prayer based upon it, then perform tasleem, followed by two prostrations." (Al-Bukhari, Fath Al-Baari #401).
One cannot discern nor is there any prevalence apparent to indicate which of the two cases is more certain: In this case, one should assume the least of the two and continue based on this assumption, then perform the two prostrations of doubt before tasleem. An example of this would be one who while praying the dhuhr prayer is completely uncertain whether he has completed three or four raka’aat and cannot determine which is more true. Thus, he would assume the lesser of the two, i.e. three, and complete another raka’ah then sit for the tashahhud, then prostrate two prostrations before saying "salaam." The basis for this ruling is the hadeeth related by Abu Sa’eed Al-Khudri, may Allah be pleased with him, in which he said: the Prophet (peace be upon him) said, "If any of you doubts during his prayer and he does not know how many [raka’aat] he has prayed, whether it is three or four, then he should discard and cast away his doubt. He is to continue upon what one is sure of [i.e. the lesser] then perform two prostrations before making tasleem. If he ends up [in reality] praying five [raka’aat], then his prayer will be an intercession for him; and if he ends up completely the [requisite] four [raka’aat] then it is targheeman for shaytaan." (targheeman: i.e., a way of vexing Satan and humiliating and debasing him, as well as rejecting him as a result of his failure to achieve his desire of disturbing the worshipper.) (Al-Nawawi in explaining the hadeeth from Sahih Muslim, 5/60).