Question: It took 5-6 months to hear from the referee in both of my accepted papers. In my one rejection it was only a couple of weeks. 

I do not think it could possibly take 5-6 months to read and understand my papers. 

Should I conclude that my referee's wait a few months to even begin reading my papers? Are they trying to answer questions I posed, to piggy-back before others see the publication? I just am having a hard time understanding the delay.

Answer: 
Reviewing a paper takes me 4 hours upto 2 days. I only accept review invitations when I have time to read those papers. I have not yet declined any serious invitation. I try to submit my (naturally constructive) review within two weeks after being invited by an editor, no matter the time being given. 

In contrast, I had to wait seven months once to receive a rejection. Other peer-reviewed journals accepted within two - three months, including revision. Therefore, I have the same question as you have. I think it strongly depends on the journal and the reviewers it relies on.

I have a theory but it is just a theory. Some journals have procedures about how often a reviewer can be approached. What if you just have to wait until a reviewer in your field becomes available?

Further I notice that reviewers easily exploit the maximum time given by an editor. An editor probably waits for all reviews to return. Then there are reviewers that need reminders or just never respond after having accepted an invitation, forcing an editor to start all over again. I even consider the possibility that a small number of reviewers intentionally slows down the process for competitive reasons. 

I feel the whole review and decision process would benefit from more transparency and behavioral professionalism. Being busy is no excuse. Nobody is enforced to accept an invitation. We all, without exception, have 24 hours in a day. 

You could consider sharing your review experience at SciRev.org
A personal experience as editor (which might help understand why this happens):

An interesting paper finds its way to me via the editorial office. I usually try to use one of the suggested reviewers and another one - so those are invited and have about 3 weeks to reply. Both reviewers fail to reply within 21 days and the next on the list are invited ... After about 12 weeks, I have two reviews: "Reject" and "Minor Revision". In that case, I prefer to get a third opinion (which is also policy) - so add another 6 to 8 weeks. In total it took 24 weeks from submission to a 'Major revision' decision - and resulted in a very unhappy author. 

Unfortunately there is no easy solution: it is difficult to find (good) reviewers, and they tend to be inundated with review requests as they are known to be reliable. 

Complain to the editor-in-chief (or publisher) - in a friendly way of course - perhaps they can reduce the time given to reviewers or find other ways.