a member of clan murray, he was the second son of lord george murray, fifth son of john murray, 1st duke of atholl. his mother was amelia, daughter and heiress of james murray, a surgeon, of strowan. john murray, 3rd duke of atholl and george murray were his brothers; his uncle lord john murray, later his commanding officer, was his godfather. murray's father had been attainted and exiled for taking part in the jacobite rising of 1745, and in 1749 james joined him in the netherlands. he was educated at utrecht and besançon.