Contracts with 13 private prisons will be reviewed and and allowed to expire over the next five years.
"They do not save substantially on costs and ... they do not maintain the same level of safety and security," Deputy Attorney General Sally Yates said explaining the decision.
The majority of US prisoners are held in state-run prisons.
On Wall Street, the stocks of private prison companies declined sharply after the news was announced.
By Thursday afternoon, Corrections Corporation of America stock had plunged by nearly 50%.
An Inspector General's report released this month found that private prisons saw higher rates of violent incidents and rule infractions in comparison with government-run institutions.
Jonathan Burns, a spokesman for the Corrections Corporation of America, told BBC News that the report contained "significant flaws" and that other studies have shown their facilities "to be equal or better with regard to safety and quality".
David Fathi, who directs the National Prison Project for the American Civil Liberties Union (ACLU) told BBC News that the decision could have a trickle-down effect on state and local prisons, where more than 90% of US prisoners are held.
In prison systems "nobody ever wants to be the first" to make changes to policy, Mr Fathi says, so the states will be watching closely to see if the change in federal policy successfully cuts back on violence - and costs.
There were 94,365 prisoners being held in private facilities overseen by states in 2010, according to the Sentencing Project group.
During the Democratic presidential primary race, Hillary Clinton's main rival Senator Bernie Sanders made a campaign promise to end the "private, for-profit prison racket".
Senator Sanders sponsored a bill during his campaign attempting to end the use of private prisons in September 2015 saying "we cannot fix our criminal justice system if corporations are allowed to profit from mass incarceration".
In a statement after the decision, Senator Sanders called the move "an important step in the right direction", and that it is "an international embarrassment that we put more people behind bars than any other country on Earth... due in large part to private prisons".
Mrs Clinton's website states that she will "end the era of mass incarceration, reform mandatory minimum sentences, and end private prisons".
Mother Jones magazine published an expose earlier this month after one of their reporters took a job as a guard at a private prison in Louisiana for four months.
The magazine found the wide scale use of violence by prisoners and guards alike.
The federal government began to rely on private prisons in the 1990's during a period of overcrowding, with the inmate population peaking in 2013.
The decline in prisoner population over the past three years contributed to the Justice Department's decision, Ms Yates said.
President Barack Obama had made criminal justice reform a priority for his administration, and in July 2015 he became the first sitting president to visit a federal prison.
The Obama administration has pushed for shorter sentences for non-violent crimes, while separately Mr Obama has regularly reduced the prison sentences of large groups of non-violent inmates.
"Mass incarceration makes our entire country worse off, and we need to do something about it," Mr Obama said last year.
Earlier this month, Mr Obama commuted the sentences of 214 federal inmates, meaning he has now authorized more prisoner releases that the last nine previous presidents combined, accounting to analysts.
As of December, private prisons housed 22,660 federal inmates, out of a total of nearly 200,000.
The US Justice Department will phase out use of privately owned prisons, citing safety concerns.