Miller grew up in a liberal-leaning Jewish family in Santa Monica, California.[15][16] He is the second of three children born to Michael D. Miller, a real estate investor, and Miriam (Glosser) Miller. His mother's family immigrated to the United States in the early 1900s from Belarus escaping the Anti-Jewish pogroms in the Russian Empire (1903-1906).[17][18] When his family arrived in the United States, his great-grandmother only spoke Yiddish, the historical language of the Ashkenazi Jews of Eastern Europe. However, in spite of their poverty, as did most poor immigrants, most of the family learned English, worked hard and opened successful businesses in Pennsylvania —"the classic immigration success story".[19]