If you’re using the same Dataset object, then yes, you’re using the same transforms for train and test. Therefore, you’re transforming the test set too. Check this tutorial: http://pytorch.org/tutorials/beginner/data_loading_tutorial.html 675
Normally you have 2 transforms like this:
train_transform = transforms.Compose([
        transforms.ToPILImage(),
        transforms.RandomCrop(60),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor()
    ])
test_transform = transforms.Compose([
        transforms.ToPILImage(),
        transforms.ToTensor()
    ])

and the custom Dataset object takes transform as a parameter so that you can use different transforms for training data and testing data. E.g.:
trainset = CustomDataset(data, transform=train_transform)
testset = CustomDataset(data, transform=test_transform)

This way, your test data is not transformed. Hope this helps.