Differential Entropy with negative support Differential Entropy Estimation
Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. The underlying Gaussians have some standard deviation which reaches also into the negative part of the x-axis, if the mean is close to zero. This produces confusion:
I want to express the amount of information this distribution contains via entropy. I know, that all of the properties are strictly positive, so the intuitive way is to truncate the distribution, in such a way that the part of the support that is negative is truncated from the distribution. But:
Entropy is invariant to translation
Due to the fact that differential entropy is invariant to translation, and the amount of information is basically saying "how wide is the distribution", then I could just translate the distribution to the right and compute the entropy then (which would be the same as before the translation anyway, but at least theoretically valid) but this would no more mirror the prior knowledge of 'I know that properties are strictly positive.'
The question:
What is the correct approach? Computing the differential entropy from the whole distribution also containing the negative part of the support or simply limit the computation of entropy to the positive part (i.e. truncation)?
EDIT:
As mentioned in the accepted answer, the approach of truncating the distribution is not solid, here is why:

*

*The general intuition behind entropy and measurements is that after each measurement, the entropy of the distribution should either lower or stay the same. It should not increase, that would mean loss of information in some sense.

*In the first picture, it can be seen that the distribution truncated at 0 has lower entropy. This makes sense as it has smaller footprint.

*In the second picture a measurement has been made. The truncated distribution has higher entropy after measurement (H(X) = 8.29292) than before the measurement (H(X) = 8.01642) which is incorrect.

Conclusion
The truncation is incorrect approach and brings obscurity and haziness into the measurement.
 

In general, the approach is not solid. If a sizeable fraction of your model leaks into the forbidden region, then the model does a poor job at describing the true probability distribution, and, by extension, any metric derived from that model would incur a certain error. Cropping the probability distribution at zero and re-scaling may or may not be cheating. If you have reasons to believe that the cropped and re-scaled model is a good model of the underlying probability distribution, then go ahead. If not then you may be introducing errors.
My suggestion is as follows. Do it both ways and compare the results. If the amount of leak is small and the results agree to a good precision, then use either of the results. If the leak is sizeable and/or the results disagree significantly, then you have to change your model to a model that is explicitly non-zero
