He is the longest-serving elected official in the city because of term limits , and during nearly four years as speaker , he has cultivated relationships with politicians and civic groups outside his Manhattan base , in part to dampen any suspicion that a white , well-off Upper East Sider can not adequately represent a city as racially and culturally diverse as New York .