I looked at the code and think the dataset might be the issue, since it’s unnormalized and imbalanced.
There are several ways to deal with imbalanced datasets.
A good way is to use the WeightedRandomSampler 148.
Alternatively, since you are slicing the first 4000 samples, let’s just get 2000 samples of both classes.
y_one_idx = np.where(np.array(cdf['class']==1))[0][:2000]
y_zero_idx = np.where(np.array(cdf['class']==0))[0][:2000]
y_idx = np.hstack((y_one_idx, y_zero_idx))
np.random.shuffle(y_idx)

Y = torch.from_numpy(np.array(cdf['class'][y_idx], dtype=np.int64))
X = torch.from_numpy(np.column_stack((np.array(cdf['cap-shape'][y_idx], dtype=np.float32),
                                      np.array(cdf['cap-surface'][y_idx], dtype=np.float32),
                                      np.array(cdf['cap-color'][y_idx], dtype=np.float))))

Also we should try to normalize the data, because categorical data could also cause some problems in your model. Add these lines after loading the data:
X = X - torch.mean(X, dim=0)
X = X / torch.std(X, dim=0)

After some epochs, the model should start to learn. Printing the amount of 1s and 0s shows, that the output is also balanced.
However, the accuracy is not really good, so there is some work to do regarding the architecture.