Eventually , with little money and most of his London friends dead or drifted away , Brummell became fixated on the idea that if he could reconcile with the prince , he would be able to return to England .