Hi Zhen!



 KenZhenLin:

Suppose the weight or kernel matrix is a 3 by 4 matrix W, and its elements are W_ij
I would like for each column j, there is one and only one nonzero W_ij, and W_ij = 1.


Again, this is an integer-programming problem – not a natural fit for
gradient descent.
But why do you want to do this?  Suppose you had weights that didn’t
have this constrained form, but gave a better-performing network, e.g.,
more accurate predictions.  Why would you not prefer those?


One possible solution I can think of is to put the following constraints using the penalty approach you mentioned:
W_1j + W_2j + W_3j = 1 for all j = 1,2,3,4
and
W_ij * (1-W_ij) = 0, for all i, j


I haven’t thought about it carefully, but these constraints do look like
they are consistent with what you described above, and can be turned
into penalties.


If this is not possible in Pytorch, can TensorFlow do it?


I know less about tensorflow, but I’m not aware of anything built into
it that would handle such constraints for you.  Tensorflow is also a
neural-network framework that is based on backpropagation and
gradient descent, so it is also not the natural tool for this part of your
use case.
A comment:  If your weight matrix, W, really is as described above, you
have 81 (3**4) choices for W.  You could freeze W to be each one of
those choices in succession, and train the rest of your network using
gradient descent while holding the specific choice of W constant.  You
will now have 81 trained networks.  Use the network, with its specific
W, that performs the best.  This is, of course, the “exhaustive search”
algorithm for the integer-programming problem, but with only 81 points
in your search space, it is perfectly doable.  (But it will scale horribly
should W grow in size beyond 3 by 4.)
Good luck.
K. Frank