Media playback is not supported on this device
After much deliberation, McIlroy last month opted out of the Games, saying he was concerned about the Zika virus.
Speaking ahead of this week's Open, McIlroy said he was "very comfortable" with his decision to pull out of the Olympics, adding he may not bother watching the Rio golf action on TV.
"I'm very happy with the decision I've made, I have no regrets," said McIlroy.
"I'll probably watch the Olympics, but I'm not sure golf will be one of the events I'll watch."
Asked which events he'd watch, McIlroy replied: "Probably the events that…track and field, swimming, diving, the stuff that matters."
McIlroy rejected suggestions that the decision of several top players to withdraw from the Olympics was "letting the game down".
The sport is making its return to Olympic competition after an 112-year absence.
"I don't feel like I've let the game down at all. I didn't get into golf to try to grow the game, I tried to get into golf to win championships," he told a pre-Open Championship news conference at Royal Troon.
"All of a sudden you get to this point and there is a responsibility on you to grow  the game, and I get that.
"But at the same time that's not the reason that I got into golf. I got into golf to win. I didn't get into golf to get other people into the game."
Also in his news conference, McIlroy said that golf had to improve is drug testing policy because it is "some way behind the other sports".
The World Anti Doping Agency's (Wada) most recent report from 2014, revealed 507 tests were carried out on golfers with eight testing positive.
"On average, we get tested four or five times a year. It's very little compared to other Olympic sports," said McIlroy.
Northern Ireland's world number four added: "I have been tested once for drugs this year, the day before the US Open. It was only a urine test, not a blood one.
"But I don't think there are drugs that can make you better across the board.
"There are drugs that can make you stronger and make you concentrate more but not that can make you a better all-round golfer, as far as I am aware.
"I could take HGH (Human Growth Hormone) and get away with it."

So far, more than 3,500 have been assessed and as a result more homes removed from the list and others added.
Historian John McCormack said planners must be "flexible" with owners with regards what warranted listing.
Last year he labelled the planning backlog for protected or listed buildings as "outrageous".
Mr McCormack said: "I'm whole heartedly behind the listing idea but that doesn't mean everything inside is worthy of listing.
"You've got to do it with humanity, courtesy and flexibility and you have to say to people why you want their home to be listed and get their interest going.
"They will be the best preservers of what is inside, they after all live there and it's their homes, and they want them to be the best as possible."
In 2012 Guernsey's Environment Department held a questionnaire asking how important it was to protect Guernsey's historic buildings.
The results showed a majority felt it was important but views were divided on the buildings that should qualify.

Gwent Police said the man was arrested in Hamburg on 12 August and will be extradited to the UK.
The man is believed to be Thomas Doran, who is wanted as part of Gwent Police's Operation Imperial - an investigation into serious offences against vulnerable adults on the Gwent Levels.
The extradition process can take six weeks, police added.
Rory McIlroy has suggested that golf will not be one of the sports that "matter" at next month's Rio Olympics.
[NEXT_CONCEPT]
The States of Guernsey planning department has said it is making "steady progress" on visiting and classifying listed buildings.
[NEXT_CONCEPT]
A 60-year-old man from Cardiff has been arrested in Germany on suspicion of conspiracy to kidnap.