What is the biggest disk that can be used in place of the optical drive in an early 2011 15 inch MacBook Pro? The limitation in a 15inch early 2011 mbp is that the hdd should have a height of 9.5mm and that the SATA port only works reliably with drives that connect at SATA 2 speeds (3G bits/sec)

To use a hdd in the optical drive you have to use an adapter, of course. The height is easy to verify.

I'm looking for the drive with the most storage space that will work reliably in this case.

There are reports that the 1.5TB hdd by HGST works in this case. Are there any other drives that work?
I added an SSD to my Macbook back in 2012, and did a write-up about it here: http://blog.omgmog.net/post/upgrading-my-macbook-pro/

Most 2.5" drives should do, the dimensions of these drives is pretty much standardised.

If you're putting a HDD fitted using an adapter in the old optical drive bay, it won't have the shock detection that the lone HDD bay has. Choosing SSD in the optical bay is better from a shock management aspect.