The Nature of Mathematical Modeling and Probabilistic Optimization
  Engineering in Generative AI
In this paper, we give an in-depth analysis on the mathematical problem
formulations and the probabilistic optimization explorations for some of the
key components in Transformer model [33] in the field of generative AI. We
explore and discuss some potential further enhancement for current state of the
art methods for some key underlying technologies of generative AI models from
algorithmic and probabilistic optimization perspective. In particular, we
present an optimal solution for sub-word encoding (SWE) based on similar
initial settings as that of byte-pair encoding (BPE) algorithm in [9] with
similar objectives as that of WordPiece approach in [28, 31] to maximize the
likelihood of the training data. We also present cross entropy optimization
method to optimize hyperparameters for word2vec model [17]. In addition, we
propose a factored combination of rotary positional encoding (RoPE) [32] and
attention with linear biases (ALiBi) [23] with a harmonic series. We also
present a probabilistic FlashAttention [6, 7] (PrFlashAttention) method with a
probability distribution over block distances in the matrix to decide which
block is likely to participate in a given round of attention computation while
maintaining the lower triangle shape of the tensor for autoregressive language
models by re-shaping the tensors. Finally, we present staircase adaptive
quantization (SAQ) of key-value (KV) cache for multi-query attention (MQA)
based on the framework presented in [16] to have gradual quantization
degradation while achieving reasonable model quality and cost savings.
671af9f682e53a391ca9cfed
Fulu Li
claimed_verified
2024-10-25T09:30:12.356Z
670d586a5cde0d9af5008ad0
/avatars/be510dee91e6101bb1d79978e76e65ee.svg
Fulu Li
user
Fulu2024
The Nature of Mathematical Modeling and Probabilistic Optimization
  Engineering in Generative AI
In this paper, we give an in-depth analysis on the mathematical problem
formulations and the probabilistic optimization explorations for some of the
key components in Transformer model [33] in the field of generative AI. We
explore and discuss some potential further enhancement for current state of the
art methods for some key underlying technologies of generative AI models from
algorithmic and probabilistic optimization perspective. In particular, we
present an optimal solution for sub-word encoding (SWE) based on similar
initial settings as that of byte-pair encoding (BPE) algorithm in [9] with
similar objectives as that of WordPiece approach in [28, 31] to maximize the
likelihood of the training data. We also present cross entropy optimization
method to optimize hyperparameters for word2vec model [17]. In addition, we
propose a factored combination of rotary positional encoding (RoPE) [32] and
attention with linear biases (ALiBi) [23] with a harmonic series. We also
present a probabilistic FlashAttention [6, 7] (PrFlashAttention) method with a
probability distribution over block distances in the matrix to decide which
block is likely to participate in a given round of attention computation while
maintaining the lower triangle shape of the tensor for autoregressive language
models by re-shaping the tensors. Finally, we present staircase adaptive
quantization (SAQ) of key-value (KV) cache for multi-query attention (MQA)
based on the framework presented in [16] to have gradual quantization
degradation while achieving reasonable model quality and cost savings.
The paper explores mathematical formulations and probabilistic optimization techniques for various components of Transformer models in generative AI, proposing enhancements such as sub-word encoding, cross entropy optimization, factored rotary positional encoding, probabilistic FlashAttention, and staircase adaptive quantization.
sub-word encoding byte-pair encoding WordPiece cross entropy optimization hyperparameters word2vec rotary positional encoding attention with linear biases FlashAttention probabilistic FlashAttention staircase adaptive quantization key-value cache multi-query attention