Hello Blade!



 blade:




 KFrank:

Conceptually, what are the index values in W2 supposed to mean?


I have two networks that are trained together.
…
What you see is my downstream network: it takes in output of the upstream W
…
I want to turn them into either a one-hot vector or class labels W2


Once you turn the output of your upstream network into a one-hot
vector or a class label, you have done something that is not
differentiable and that breaks backpropagation.
So …   You have to do something else.
Why not just pass the output of your upstream directly to your
downstream network (i.e., directly to fc_5)?  What breaks?
If the output of your upstream network comes directly from a Linear
without any subsequent activation function, you will want an activation
function in between.  How about relu() or sigmoid() (or even
softmax())?
If passing a one-hot vector to your downstream vector makes the
most sense (ignoring the fact that it isn’t differentiable), perhaps you
should consider my observation that softmax() is a differentiable
approximation to the one-hot encoding of argmax().
You can sharpen this result – making softmax() “less soft” – by
scaling its argument, e.g., softmax (scale * W1).  As scale is
increased to approach +inf, the result of softmax (scale * W1)
will approach one_hot (argmax (W1)).
Best.
K. Frank