Thanks, I only have one more question. Since there are trainable parameters W, what is the most common way to initialize for this specific task?
Is this a good practice :
self.weights = nn.Parameter(torch.FloatTensor(self.n_patches)).unsqueeze(1).cuda()
nn.init.uniform_(self.weights)

And then to normalize so that their sum(w) = 1? Or its enough to initialize with uniform without normalization? Maybe other initialization procedure will be more suitable in this case?