according to a story, he was initially a sarangi player, but decided to switch to vocals because of low status of sarangi players. in his early years, he used to sing with his brother abdul haq. they appeared before murtaza khan of the "maula bakhsh khandan" to appear before the maharaja (king) and were influenced by murtaza's style. the king of baroda state was impressed by this singing duo and made them the court musicians. this is where he met tarabai mane, who was the daughter of sardar maruti rao mane, a member of the royal family. when they decided to get married, they were ousted from baroda. the couple settled down in bombay. in 1922, tarabai mane left abdul karim khan, which apparently had a major impact on his music – making it pensive and meditative. karim khan's first wife, gafooran, was the sister of another kirana master abdul wahid khan, who was also his cousin.