Although his father taught school across the street from the museum cottage for a time and his aunt once lived in there , John Howard Payne was probably not even born when his family moved from East Hampton to Manhattan and what is now Chinatown .