Why didn't the entire village know Naruto was Minato's son? Besides the out of universe reason that his identity was hidden as a plot device, is there any in universe reason people don't know Naruto is the 4th Hokage's son?

After all, Minato was married to Uzumaki Kushina. She must have been pregnant for a while before her and Minato's death. Being the wife of the 4th Hokage, she must have been, at least to some degree, a celebrity figure. Everybody would have known she was pregnant. She must have been sporting a good baby bump for at least a couple months before her death. Shortly after the time of her death, there's suddenly a newborn baby named Uzumaki Naruto wandering around (that happens to have a striking resemblance to the 4th, which becomes more and more obvious as he ages), and nobody made the connection they were related?


  Suddenly orphaned, Naruto was left to grow up knowing nothing of his parents, receiving only his mother's last name because the Third Hokage thought that it was best that nobody knew that he was related to the Fourth Hokage.


I know that the Third hid his relation to the 4th, but simply using the 4th's wife's last name doesn't seem like it would be very effective. Was that really all it took to fool just about EVERYBODY? Is there some other explaination to Naruto's anonymity?
While I don't like posting speculative answers, upon plenty of time and thought I think this is the best I can think of. Expanding on my comment on the question, I'll try to logically conclude why many didn't relate the boy Naruto with the son of Hokage.


We see the scale of invasion pretty early in the episode and see it
was a sudden war worse than the attack by the Sand/Sound villages.
The strategy of saving the villagers probably resulted from the
Nine-Tailed Fox Attack. The knowledge of Jinchuriki is SECRET
information and wasn't revealed to anyone but the Hokage and top-tier Anbu.
Hence, while plenty of people had seen Kushina pregnant,
no one knew the significance of the birth (even Minato didn't know).
The sudden death of the Fourth and his wife naturally led people to
conclude the worst. We also know that some Anbu and the Third's wife died
as well. When so many people die together, how can they assume a new
born baby survive?   
We know of at least two orphans from the Nine-Tailed attack. Iruka and
    Naruto. (I believe Kurenai's parents died too, but not sure if it was 
    filler). When the Third Hokage retook the position, the village was 
    turmoil, and plenty of rehab efforts were made, including for orphan
    children whose parents had died. Uzumaki Village is extinct, but
    there may have been several wandering clans in the Land of Fire (not the
    village), who may have visited the village. And Naruto may've been
    thought to be the son of guests who unfortunately died in the attack.
    The nature of the Third's kindness is well known and no one would question
    his interest in the kid of a guest of the village who died because
    of the failure.
The Hokage's family are treated as special in village. Tsunade (the First's
granddaughter), Asuma (ran away from home to escape the shadow of
Hokage's son), Konohamaru (Honorable Grandson + Special Tutor) and
even Boruto (Konohamaru would naturally take him in). Everyone likes
them. Treats them with respect. In Naruto's case only the shadow of
fear lingered, and it didn't change. If he were "special" why didn't
anybody intervene? This created the vicious circle of some people
staying away because of the fear that being the Fourth's son makes him
a target (Kakashi), some jounin let him be (Shikaku), some actively
shunned him. This shunning gradually increased and seeped into the
kids as well. No one expects a village oddball to be that of Hokage
heritage.
The most important point is hiding in plain sight. Naruto isn't
the first nor the last to use this technique for hiding the true
nature of main character. Superman is a prime example. People don't
expect "special" people to live among the common! What they don't
expect they just don't see it. This is true in real life as well.