I did have varying shapes, but I solved the problem by converting both the model data, and the one hot vector to tensors individually, so my code looked like this:
# temp contains NumPy objects
dataset = []
for object in temp:
    dataset.append([torch.Tensor(torch.Tensor(object[0])), torch.Tensor(object[1])])
    # object[0] contains the data; object[1] contains the one-hot vector

Since the numpy array contains objects instead of float64, or any other primitive data types, is there a way in avoiding the creation of another array?