whitmore was married four times; once to nancy mygatt in 1947 until they divorced in 1971. then he married audra lindley in 1972 until they divorced in 1979. later he once again married nancy mygatt in 1979 until they divorced once again in 1981. lastly whitmore married noreen nash in 2001 until whitmore's death in 2009. he only has three sons from his first marriage including; james whitmore, jr..