Patricia Kennedy was born on May 6 , 1924 , in Brookline , Mass . , the fourth daughter of Joseph P. and Rose Kennedy and the granddaughter of John F. Fitzgerald , the popular mayor of Boston who had earlier served in the House of Representatives .