At the end of the 19th century , the neighborhood was called '' the Fifth Avenue of Brooklyn '' after the oil magnate Charles Pratt built mansions there for himself and his sons , and other wealthy residents followed suit .