Media playback is not supported on this device
McIlroy, 27, is one of 20 players to have withdrawn from next month's Games, citing fears about the Zika virus.
He said he would not watch the Olympic golf on television.
Speaking after Thursday's first round at The Open, McIlroy said: "I decided to stop dancing around the issue and tell everyone what I thought."
Media playback is not supported on this device
The four-time major winner added: "I've thought about Olympics and golf for the last seven years and this is just how I feel.
"Some people don't like it, I get that; but it's my opinion.
"I don't think anyone can blame me for being too honest. It was seven years of trying to give the politically correct answer and finally I just cracked."
On Tuesday, the Northern Irishman said: "I'll probably watch the Olympics, but I'm not sure golf will be one of the events I'll watch."
Asked which events he would watch, McIlroy replied: "Probably track and field, swimming, diving, the stuff that matters."
Media playback is not supported on this device
Those comments received criticism from English squash player Laura Massaro, whose sport is not included in the Olympics.
She called McIlroy's comments "unacceptable" and said he had shown a "lack of appreciation for how the Olympics can transcend an individual sport".
However, McIlroy, who is six shots off the lead after the first day of the Open, has tried to elaborate more on his comment that he "didn't get into golf to try to grow the game".
Find out how to get into golf with our special guide.
"I feel I've done my bit to grow the game," added the world number four. "It's not like I'm uninterested and golf is a great vehicle to instil values in kids.
"I'm an ambassador for the PGA Junior League and I've used my success in golf in a very positive way."
We've launched a new BBC Sport newsletter, bringing all the best stories, features and video right to your inbox. You can sign up here.

They are holding an inquiry into the procedure, used to ease incontinence, after some women experienced painful complications.
Last year, the then Scottish health secretary Alex Neil urged health boards to suspend use of the implants.
Adam Slater, a US compensation lawyer, said "women are not told the truth" about the devices.
He was lead trial counsel in the first pelvic mesh trial against Johnson & Johnson and Ethicon, resulting in a compensatory and punitive damages verdict of $11.1m.
Giving evidence to members of the Scottish Parliament committee via videoconference, Mr Slater said the closest comparison to the mesh implant situation he could think of was asbestos.
He said: "Something that was thought to be a wonderful invention for a long time and now everybody in the world knows it's something you wouldn't want to go near.
"It's the closest analogy I can find to these horrible devices that are now in so many women."
The committee is considering calls to suspend mesh implant operations. Operations continue to be carried out by some health boards despite the health secretary's call to stop using them.
Mr Slater has worked almost exclusively on mesh cases in the US since 2007 and is currently the lead counsel for more than 7,000 cases in New Jersey.
He said some doctors are now refusing to use the devices, with one even branding them a "social cancer".
He told MSPs that studies into the safety of mesh have been influenced by manufacturers, who enjoy a "close relationship" with regulators of the products.
The lawyer said thousands of women in the US are suing both manufacturers and doctors on the grounds that mesh devices are defective or that there was a failure to warn them about the dangers and complications.
The US regulator, the Food and Drug Administration (FDA), indicated last April that it plans to reclassify mesh for pelvic organ prolapse as "high risk", but is yet to do so.
Mr Slater urged UK regulator the Medicine and Healthcare products Regulatory Agency (MHRA) to "look at the studies that are relied on by the mesh manufacturers".
"Most of the studies they rely on are written or investigated by paid consultants," he said.
"I would throw those in the garbage immediately because if somebody is being paid by the manufacturer, there is a financial bias."
He added: "There has never been a high-level study...that has ever been done that has proven the mesh to be safe and effective."
Former world number one Rory McIlroy has remained defiant in the face of criticism over his controversial views on golf at the 2016 Rio Olympics.
[NEXT_CONCEPT]
Vaginal mesh implants are as dangerous as asbestos, MSPs on Holyrood's petitions committee have been told.