By the 1860s, Brown was running a stylish brothel at the corner of University Place and 12th Street. Her neighbours included James Lenox, William H. Aspinwall, the socialite Mrs Peter A. Schemerhorn and New York Public Library. The brothel was tolerated because the house and her customers caused no disruption. Despite her illicit occupation, Brown was a darling of the New York upper class. She received invitations to social galas across New York City, and her admirers nicknamed her Princess Julia. She sometimes threw balls of her own in the winter as a way to attract new patrons. Charles Dickens reportedly visited her on a trip through America. The penny press also followed her exploits; the major criticism levied against her was that the cut she took from her girls' earnings was too large. She was reputed to be worth $11,500.