Question: It took 5-6 months to hear from the referee in both of my accepted papers. In my one rejection it was only a couple of weeks. 

I do not think it could possibly take 5-6 months to read and understand my papers. 

Should I conclude that my referee's wait a few months to even begin reading my papers? Are they trying to answer questions I posed, to piggy-back before others see the publication? I just am having a hard time understanding the delay.

Answer: 
Reading your paper is not the only obligation your referee has. They are active researchers who, besides their own research, have teaching, administration, thesis defenses, other papers to review etc. They are also normal people, who have to make their personal life function.

5-6 months is a long time, but depending on the field and the length of your paper, it is not unusual. 

Don't be so worried about other people stealing your ideas. Most likely the referees has more ideas of their own than they have available time.
A personal experience as editor (which might help understand why this happens):

An interesting paper finds its way to me via the editorial office. I usually try to use one of the suggested reviewers and another one - so those are invited and have about 3 weeks to reply. Both reviewers fail to reply within 21 days and the next on the list are invited ... After about 12 weeks, I have two reviews: "Reject" and "Minor Revision". In that case, I prefer to get a third opinion (which is also policy) - so add another 6 to 8 weeks. In total it took 24 weeks from submission to a 'Major revision' decision - and resulted in a very unhappy author. 

Unfortunately there is no easy solution: it is difficult to find (good) reviewers, and they tend to be inundated with review requests as they are known to be reliable. 

Complain to the editor-in-chief (or publisher) - in a friendly way of course - perhaps they can reduce the time given to reviewers or find other ways.