In Chapter 17 of 'Foundation's Edge', one of the Gaians ("<PERSON>") tells a fable to <PERSON> and <PERSON>. Here's an abridged version:
We have a tale about that — a fable, perhaps. I cannot vouch for its authenticity. In fact, on the face of it, it sounds like fiction. [...] I was about to tell our guests the story of Eternity. [...] the fable states that there were those who could step out of time and examine the endless strands of potential reality. These people were called the Eternals and when they were out of time they were said to be in Eternity. It was their task to choose a Reality that would be most suitable to humanity. They modified endlessly—and the story goes into great detail, for I must tell you that it has been written in the form of an epic of inordinate length. Eventually they found (so it is said) a Universe in which Earth was the only planet in the entire Galaxy on which could be found a complex ecological system, together with the development of an intelligent species capable of working out a high technology. 489
That, they decided, was the situation in which humanity could be most secure. They froze that strand of events as Reality and then ceased operations.
So, <PERSON> himself implied that the Eternals from 'The End of Eternity' had "created" the universe which contained his Galactic Empire and the subsequent Foundations.
Later (in the real world), in 'Forward the Foundation', <PERSON> was doing some investigations about mental powers, and he tells <PERSON>:
There is a curious story, about twenty thousand years old and therefore back to the misty origins of hyperspatial travel. It's about a young woman, not much more than <PERSON>'s age, who could communicate with an entire planet that circled a sun called Nemesis.
Again, <PERSON> himself implied that the events of 'Nemesis' happened in the same universe as the Empire and the Foundations.
That's why so many people think they're connected.
However... there are contradictions between those other two works and the Robots / Empire / Foundation series.
* It's explicitly stated in 'The End of Eternity' that the extinction of humanity in the far-distant future is caused by their discouragement when they finally reach the stars and discover that the stars are already populated by the many other star-faring species who achieved interstellar travel thousands of centuries before humanity. (The reason for the delay in humanity reaching the stars is the existence of Eternity, which chooses the safe path for humanity - which does not lead to interstellar travel.) However, even after Eternity is destroyed, allowing humans to develop interstellar travel earlier, the other species would still exist. Even if humans get to space thousands of centuries earlier, they should still discover at least some of those other intelligent species - even if they're still planet-bound. However, the Galactic Empire and the subsequent Foundations have never discovered any non-human intelligent species. Ever.
* When <PERSON> and Pelorat finally reach Earth's solar system in 'Foundation and Earth', they do not detect a companion star for Sol, such as described in 'Nemesis'.
* There are no robots in 'Nemesis'. There were robots when humans had not left Earth, and robots accompanied the first Earth colonists (Spacers) to the first interstellar colonies. The expedition to the Nemesis system is supposedly the first time humans have left the solar system, which places it firmly between the time of Earth-bound humans and the time of 50 Spacer worlds... yet there are no robots in 'Nemesis'. 44