Noy_Uzrad:

Output size shall be [49, x, y].


If I understand correctly, since you want differently weighted spectrograms with each of the 49 probability vectors and 49 spectrograms, the output size should be [49, 49, x, y]. No?
i.e., In the spectrogram size [49, x, y], batch size = 49. The same batch size as in the probability matrix of [49, 49]. For a particular spectrogram, you have 49 probability values corresponding to different classes.