Pillard is the daughter of Boston University psychiatry professor Richard Pillard and Cornelia Cromwell Tierney. She is married to Georgetown law professor and current ACLU Legal Director David D. Cole and has two children, Sarah and Aidan Pillard.