The Commission on Widening Access said there was an "undeniable case for change" on the issue.
But it acknowledged the move would be "divisive", with concerns that middle-class students could be displaced.
The commission was set up by the first minister last year.
It was tasked with advising the Scottish government on how best to achieve its goal of ending the big gap between the number of students from wealthy and deprived backgrounds going to university.
The commission's interim report said there had been "steady progress" on the issue over the past decade.
But it stressed that "very significant socioeconomic inequalities" remained which were "unfair, unsustainable and detrimental to Scotland", and said the country had a moral and economic duty to tackle the issue
The report said it had been suggested that admitting students from deprived backgrounds with lower grades could have a detrimental impact on the academic excellence of Scotland's universities.
But it stated: "There is increasingly strong evidence that with the right support, bright students from deprived backgrounds can enhance, rather than jeopardise, academic excellence."
The report said other opposing viewpoints included:
But it added: "Unless we are prepared to accept the notion that Scotland's talent is concentrated in its most affluent communities, it is clear that, through accident of birth, a whole section of Scottish society has nothing like an equal opportunity to maximise their talent and reap the benefits of higher education.
"We believe that this is fundamentally unfair and that the ultimate goal of widening access should be to eliminate socioeconomic inequality.
"Equality of access is not just a passport to a better life for individuals; it is also a passport to a fairer, better Scotland."
The report also said there was strong evidence that parental experience of higher education was one of the most influential factors in determining the likelihood of a child entering university.
It added: "This means that equal access is capable of transmitting the social and economic benefits of higher education between generations, breaking cycles of deprivation and contributing to a society that is healthier, wealthier and fairer.
"The evidence shows that a higher education is a passport to a better life. Graduates benefit from higher wages, significantly improved health outcomes and a higher life expectancy.
"We believe that Scotland has a moral duty to ensure that these opportunities are distributed fairly."
The commission's final report is expected to be submitted next year.
The interim report was welcomed by Education Secretary Angela Constance, who said more needed to be done to tackle inequality in education despite the progress that had been made in recent years.
A spokeswoman for Universities Scotland said: "Contextual admissions can help with this and it is one of many tools, but definitely not a silver bullet, that universities can use to help widen access.
"Universities will always look for the best and brightest applicants - our quality and excellence is very important to us and absolutely will not be compromised - but we are open-minded about what best and brightest actually means."
Scottish universities should consider admitting students from deprived backgrounds whose grades are lower than their more affluent counterparts, according to a report.