he did n't mind me going to town or to visit grandma miriam at the home by myself but it was rare he left me at the house alone and , when he did , it was when whatever business he had did n't take him long .