On The Statistical Representation Properties Of The Perturb-Softmax And The Perturb-Argmax Probability Distributions
<|reference_start|>On The Statistical Representation Properties Of The Perturb-Softmax And The Perturb-Argmax Probability Distributions: The Gumbel-Softmax probability distribution allows learning discrete tokens in generative learning, while the Gumbel-Argmax probability distribution is useful in learning discrete structures in discriminative learning. Despite the efforts invested in optimizing these probability models, their statistical properties are under-explored. In this work, we investigate their representation properties and determine for which families of parameters these probability distributions are complete, i.e., can represent any probability distribution, and minimal, i.e., can represent a probability distribution uniquely. We rely on convexity and differentiability to determine these statistical conditions and extend this framework to general probability models, such as Gaussian-Softmax and Gaussian-Argmax. We experimentally validate the qualities of these extensions, which enjoy a faster convergence rate. We conclude the analysis by identifying two sets of parameters that satisfy these assumptions and thus admit a complete and minimal representation. Our contribution is theoretical with supporting practical evaluation.<|reference_end|>
@article{indelman2024on,
    title={On The Statistical Representation Properties Of The Perturb-Softmax And
  The Perturb-Argmax Probability Distributions},
    author={Hedda Cohen Indelman, Tamir Hazan},
    journal={arXiv preprint arXiv:2406.02180},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02180},
    primaryClass={cs.LG cs.AI stat.ML}
}