For many years , Freedom House shared its headquarters at 20 West 40th Street in Manhattan , known as the Wendell Willkie Memorial Building , with the National Association for the Advancement of Colored People , the Anti-Defamation League and the Metropolitan Council of B ` nai B ` rith and other like-minded service organizations .