How many people have worn the straw hat, if only for a second? How many people have worn Luffy's straw hat? Canon, confirmed, and on the head (I'm counting Robin even if she wore it atop of her other cowboy hat).
Here you can read who had worn the straw hat and what it has gone through during the series. I will summarize the list here and add proof from the manga. For the complete panel the thumbnails are cut from, click the thumbnails. Until chapter 801,  it seems 7 people were seen wearing the hat.




Chapter 2 − Monkey D. Luffy


  Luffy is the current holder of the straw hat.






Chapter 1 − Shanks


  He gave it to Luffy as a young boy after he said the same words that Gold Roger, his former captain, had once said.






Chapter 603 − Gol D. Roger


  Much later in the series, it was revealed that Gold Roger, the man who would become the pirate king, had owned that hat.






Chapter 81 − Nami


  She is the member of Luffy's crew that he more often gives it over to for save keeping.






Chapter 114 − Robin
Not mentioned on the list, but as explained by OP, Robin had taken Luffy's hat from him during their first encounter.





Chapter 314 − Usopp


  Usopp briefly wore the straw hat during the Davy Back Fight against the Foxy Pirates.






Chapter 515 − Sweet Pea


  Luffy was in a panic when he couldn't find his hat, but it saw one of the women in the crown watching him wearing hit.