Thanks for the reply. I used a similar solution, but changed the type inside of the ToTensor() function like this:
class ToTensor(object):
    def __call__(self, sample):
        image, label = sample['image'], sample['label']

        image = image.transpose((2, 0, 1))
        return {'image': torch.from_numpy(image).type(torch.FloatTensor),
                'label': torch.from_numpy(label).type(torch.FloatTensor)}

Now everything works fine.