Can I use my Macbook Pro Retina as a monitor for PS3? I want to connect my PlayStation3 to my Retina Macbook Pro so that I can play my games using the MBP's display, is it possible to do it with HDMI port or something else?
No, the video ports (HDMI and Thunderbolt) are output only. The MacBooks aren't designed to function as monitors.

You could try getting a video capture device that supports live viewing, but the performance and quality are likely to be far from optimal—such solutions are designed for recording video, not live playback or gaming.