Thanks for the thorough explanation! Sorry for the late reply but I’ve been busy with finals. The skip size parameter is something that has confused me as well but I saw it pop up in various implementations, including some which may be linked in my original post. It seems that they are pulling skip values for the last set of fully connected layers to act on, but I’m confused as to why we would truncate those vectors.
If I remember correctly, I had casted X to float because some of the layers in the model would not process X without it being a float. Is what you recommended essentially that for each sequence element I should pass a one-hot encoded vector to the model? So the model would be fed B batches, and each sequence of 1024 that I wished to pass to the model would be represented by 1024 one-hot encoded vectors of length 256, to represent the encoding space?
Could you comment on how the above change affects training? The way it currently works is that the input is quantized but even in float form the elements of the vector will still be of the form (120. , 243., …). In addition, how would we then process the one hot encoded vectors using convolutions? I see in your implementation you include an Embedding element, what is the use of this element in the context of wavenet? I see that it is a lookup table from the class to the embedding but what is it trained to embed each class into? Basically, how is the Embedding layer used in your implementation?
Also, I now see why the padding on the causal convolution is incorrect - admittedly I had pulled that code from another post on these forums.
Are there any resources and books you’d recommend to get a good understanding of machine learning theory and model development? Particularly form the point of view of someone teaching how to design novel model architectures?