Question: It took 5-6 months to hear from the referee in both of my accepted papers. In my one rejection it was only a couple of weeks. 

I do not think it could possibly take 5-6 months to read and understand my papers. 

Should I conclude that my referee's wait a few months to even begin reading my papers? Are they trying to answer questions I posed, to piggy-back before others see the publication? I just am having a hard time understanding the delay.

Answer: 
It’s unlikely they’re all sitting on the paper, although as has been mentioned it’s very field-dependent. A six-month delay in my field without any feedback would be approaching a four- to five-sigma event, but in math or some humanities fields, for instance, it might be more common. 

What is possible is that the editor has *some* reviews in hand, but not enough to render her decision. If somebody is straggling, then a new review has to be solicited, leading to more delays.
A personal experience as editor (which might help understand why this happens):

An interesting paper finds its way to me via the editorial office. I usually try to use one of the suggested reviewers and another one - so those are invited and have about 3 weeks to reply. Both reviewers fail to reply within 21 days and the next on the list are invited ... After about 12 weeks, I have two reviews: "Reject" and "Minor Revision". In that case, I prefer to get a third opinion (which is also policy) - so add another 6 to 8 weeks. In total it took 24 weeks from submission to a 'Major revision' decision - and resulted in a very unhappy author. 

Unfortunately there is no easy solution: it is difficult to find (good) reviewers, and they tend to be inundated with review requests as they are known to be reliable. 

Complain to the editor-in-chief (or publisher) - in a friendly way of course - perhaps they can reduce the time given to reviewers or find other ways.