zq(x) = Woutek, where k = arg min

j

||ℓ2(Winze(x)) − ℓ2(ej)||2

Here, Win and Wout are projection matrices, with Win mapping the encoder’s output to an intermediate
representation, and Wout mapping this intermediate representation to the quantized representation
zq(x). Specifically,

Win ∈ RD×M and Wout ∈ RM×D

where D is the output dimension of the encoder, and M is the codebook dimension with M ≪ D.
The vector quantizer loss function is then defined to measure the reconstruction error and is given by:

zproj(x) = Win ze(x)

LVQ = ||sg[ℓ2(zproj(x))] − ℓ2(ek)||2

2 + β||ℓ2(zproj(x)) − sg[ℓ2(ek)]||2

2

where sg is the stop gradient operator, preventing the back-propagation of gradients through ek, and
β is a hyperparameter controlling the balance between the two terms in the loss function.

14
RVQGAN