If each epoch takes 30 minutes, it seems the GPU might not be used.
Could you run this dummy code snippet and report the time for each epoch you are seeing:
model = models.vgg16().to('cuda')
dataset = datasets.CIFAR10(
    root=ROOT,
    transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
        ]),
    download=True
)

loader = DataLoader(
    dataset,
    num_workers=4,
    pin_memory=True,
    shuffle=True,
    batch_size=64
)

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

for epoch in range(5):
    torch.cuda.synchronize()
    t0 = time.time()
    for data, target in loader:
        optimizer.zero_grad()
        data = data.to('cuda')
        target = target.to('cuda')
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
    torch.cuda.synchronize()
    t1 = time.time()
    print('Epoch {} took {}s'.format(epoch, t1-t0))

I’m getting approx. 36 seconds per epoch on a Titan V.