I’m new to PyTorch, and I’m having trouble interpreting entropy. Suppose, we have a probability distribution [0.1, 0.2, 0.4, 0.3]
First, let’s calculate entropy using numpy.
import numpy as np

p = np.array([0.1, 0.2, 0.4, 0.3])
logp = np.log2(p)
entropy1 = np.sum(-p*logp)
print(entropy1)

Output:

1.846439

Next, let’s use entropy() from torch.distributions.Categorical
import torch
from torch.distributions import Categorical

p_tensor = torch.Tensor([0.1, 0.2, 0.4, 0.3])
entropy2 = Categorical(probs = p_tensor).entropy()
print(entropy2)

Output:

tensor(1.2799)

Why is entropy1 not equal to entropy2?