He was married to Doris Feigenbaum, and was a long-time member of Congregation Emanu-El in San Francisco. Their three sons continue to manage the business. He was a non-executive director of Vodafone from 1999–2000. Fisher was active in several public education causes, including being a major contributor to KIPP charter schools—a national network of low-income, high-achieving college preparatory public charter schools: he was the chairman of the board of trustees of the KIPP Foundation, the non-profit central organization of the KIPP network. He was also a contributor to Teach For America, GreatSchools.net, and EdVoice, a statewide coalition of California business leaders and others who support education reform. Fisher also served on the California State Board of Education. Fisher and his family donated a generous sum of money to Princeton University in 2006, and the Fisher Hall dormitory at Princeton's new residential college, Whitman College, is named for him. He has also donated to charter schools and museums in San Francisco, including the San Francisco Museum of Modern Art, and paid for public sculptures in San Francisco. In 2007, Fisher was honored as the Alumnus of the Year by the California Alumni Association at the University of California, Berkeley. He also contributed to many Bay Area Jewish causes, including the Jewish Community Center of San Francisco and the S.F.-based Jewish Community Federation and Endowment Fund. Since founding the Gap in 1969, Fisher and his wife Doris began collecting contemporary Western art. In 1993, "ARTnews" Magazine declared Fisher one of the top ten art collectors in the world. His collection, largely housed at the Gap headquarters in San Francisco, includes comprehensive, career-spanning works by Andy Warhol, Alexander Calder and Roy Lichtenstein, Ellsworth Kelly, Gerhard Richter, Anselm Kiefer, Chuck Close, and Claes Oldenburg and Coosje van Bruggen. On August 8, 2007, Fisher announced plans to build a museum in the San Francisco Presidio, tentatively named the Contemporary Art Museum of the Presidio, to house his art collection. The museum, if it had been built, would have been larger than the San Francisco Museum of Modern Art (SFMOMA). However, the plan engendered widespread skepticism and even outright antagonism among some historic preservationists in San Francisco. In July 2009, Fisher announced that he and his wife were abandoning their efforts to build the museum at San Francisco's Presidio, stating "Doris and I will take some time to consider the future of our collection and other possible locations for a museum, which could include other sites within the Presidio and elsewhere."