It's not matter of trust, the Mibu clan was there already far before Daimyo came into picture :P. Mibu were traditional protectors of the village before it became nin village by Daimyo's design. So Daimyo may be irked up by the fact that Mibu clan already had 'stolen' perfect spot for his office.