Question: It took 5-6 months to hear from the referee in both of my accepted papers. In my one rejection it was only a couple of weeks. 

I do not think it could possibly take 5-6 months to read and understand my papers. 

Should I conclude that my referee's wait a few months to even begin reading my papers? Are they trying to answer questions I posed, to piggy-back before others see the publication? I just am having a hard time understanding the delay.

Answer: 
Reading your paper is not the only obligation your referee has. They are active researchers who, besides their own research, have teaching, administration, thesis defenses, other papers to review etc. They are also normal people, who have to make their personal life function.

5-6 months is a long time, but depending on the field and the length of your paper, it is not unusual. 

Don't be so worried about other people stealing your ideas. Most likely the referees has more ideas of their own than they have available time.
It’s unlikely they’re all sitting on the paper, although as has been mentioned it’s very field-dependent. A six-month delay in my field without any feedback would be approaching a four- to five-sigma event, but in math or some humanities fields, for instance, it might be more common. 

What is possible is that the editor has *some* reviews in hand, but not enough to render her decision. If somebody is straggling, then a new review has to be solicited, leading to more delays.