Since he moved to Florida from New Jersey a few years ago , Mr. Brown said , he has been reminded of the isolation he felt early in his career as the lone black broker in the firm 's office in downtown Newark .