Arham_Khan:

If I remember correctly, I had casted X to float because some of the layers in the model would not process X without it being a float. Is what you recommended essentially that for each sequence element I should pass a one-hot encoded vector to the model? So the model would be fed B batches, and each sequence of 1024 that I wished to pass to the model would be represented by 1024 one-hot encoded vectors of length 256, to represent the encoding space?
Could you comment on how the above change affects training? The way it currently works is that the input is quantized but even in float form the elements of the vector will still be of the form (120. , 243., …). In addition, how would we then process the one hot encoded vectors using convolutions? I see in your implementation you include an Embedding element, what is the use of this element in the context of wavenet? I see that it is a lookup table from the class to the embedding but what is it trained to embed each class into? Basically, how is the Embedding layer used in your implementation?


I use Embedding layer just to map 256 classes to differenct latent vectors with size equals to residual channels, no other meaning. If you feed one-hot encoded vectors into the first layer it’s actually do the same thing as embedding. Directly feeding the quatized value will lose some complexity on the input presentation but may not affect the performance too much, I guess.



 Arham_Khan:

Thanks for the thorough explanation! Sorry for the late reply but I’ve been busy with finals. The skip size parameter is something that has confused me as well but I saw it pop up in various implementations, including some which may be linked in my original post. It seems that they are pulling skip values for the last set of fully connected layers to act on, but I’m confused as to why we would truncate those vectors.


The trucation you see in other implementations I guess is to truncate those time steps that are not presented in the output sequence. Take your code for example, your truncation should be:
skip = skip[:,:,-1:] 

so the output shape is the same as (batch, 256, 1) just like your target.
Also, your current training method is quite inefficient. WaveNet is a fully convolutional model, a fully convolutional model means it can be parallized when training. My suggestion is, you can try feeding a sequence of samples as output instead of taking just one time sample. For example, if I want to use 5000 samples time sequences named x during training, my input and target output will be:
input = x[:, :, :4999]
target = x[:, :, 1:]          #right shift one sample

The truncation step can be removed because now the model output and target has the same shape.
Hope these will help you.