The 65-year-old is also widely reported to have been the country's first female Muslim judge.
New York police said they pulled her from the water and pronounced her dead on Wednesday after being alerted by an emergency call.
Her husband had reported her missing, police said.
Judge Abdus-Salaam's family identified her and a post-mortem examination will determine the cause of death, New York police say.
According to local media reports, police are treating the death as a possible suicide but an investigation is ongoing.
The body was fully clothed and showed no signs of trauma or criminality, police said.
Judge Abdus-Salaam's brother took his own life three years ago and she had been battling depression, according to reports.
Her husband told police she had not been heard from since Tuesday morning.
Justice Abdus-Salaam, born in 1952 in Washington DC, was appointed to the New York Court of Appeals by Governor Andrew Cuomo in 2013.
"Justice Sheila Abdus-Salaam was a trailblazing jurist whose life in public service was in pursuit of a more fair and more just New York for all," Mr Cuomo said in a statement.
New York Attorney General Eric Schneiderman called her death "an enormous loss for New York, our judiciary, and all who knew, respected and loved her".
He described her in a statement as a "thoughtful, thorough and fair jurist".
A graduate of New York's Columbia Law School, she started her legal career working for low-income people in Brooklyn and later served as a New York state assistant attorney general.
Justice Abdus-Salaam is identified in the Princeton Encyclopedia of American Political History as the first female Muslim to serve as a US judge.
US judge Sheila Abdus-Salaam, the first black woman to serve in New York's highest court, has been found dead in the city's Hudson River.