How did people even know that the One Piece was in the Grand Line? The people say it's in the Grand Line, but I can't find anything on how they knew. In the opening with his last words, Gold Roger only says he left it in one place, not that he left it in the Grand Line. Did I miss something?
The one piece in itself is a guess. If you remember, Luffy even yelled at Ussop in  Shabody for asking such a question, and for once Luffy was actually trying not to think about the possibility of the one piece being nonexistent. However, we know that:


   There are 4 Red Poneglyphs, and each one points to a certain location. Once they are laid out, these four locations will have Raftel in the middle. 


However, we do not have actual evidence that Raftel is in Grand Line. Now that you mention it, probably, just maybe, it is somewhere else. (The irony; people were looking for it for so long in Grand Line and if it's not there then really, what a disappointment.)

As far as we know, it could be anywhere, even on top of the Red Line, we don't know. Note, at times Rayleigh's facial expressions really did suggest wild thoughts in readers, like the one piece does not exist or that the Void Century is such a simple puzzle after all. 

In Conclusion:
The people assume it is in Grand Line because Gol D. conquered it, and short after that, he got executed.