Hi Zhen!



 KenZhenLin:

let’s say W is the weight in the output layer, is it possible to put constraints on W such that the optimal W is selected from the set S={W_1, W_2, …, W_n}, where each W_i is a given feasible value for W?


This is not a particularly natural thing to do in the context of neural
networks and gradient-descent optimization (and I am not aware of
anything built in to pytorch that will do this).
A general note outside of the context of neural networks and gradient
descent:  The kind of constraints you contemplate lead to a so-called
integer-programming problem – a known “hard” problem.
Back to pytorch:  Let’s say your feasible weights are the integers 1
through 10, S = {1, 2, ..., 10}.  Suppose – based on the learning
rate and the backpropagated gradient – your opt.step() wants to
set W = 1.23.  What do you do?  You could round W to 1, but that is
not necessarily satisfactory.
Suppose W happens to be 3 at a certain stage of training, but you
and I magically know that it should end up being 5.  But let’s say that
the learning rate is small so that your optimizer is only taking small
steps, say no larger than 0.01.  How is W suppose to get from 3 to 5,
when it can’t take a step of 2 or even 1.
The best I can imagine is to add a penalty to your loss function that is
zero when W = W_i, and greater than zero otherwise.  For example:
penalty = ((W - W_1) * (W - W_2) * ... * (W - W_n))**2
loss_with_penalty = regular_loss + alpha * penalty
opt.zero_grad()
loss_with_penalty.backward()
opt.step()

Here alpha is an adjustable parameter that you have start out at
zero, and then have increase as you train.  When you make alpha
very large, you will force W to be close to one of the W_i, but  while
alpha is still zero or small, the training is not prevented from moving
from one W_i to another.  But you will have to experiment in order to
determine how to schedule how alpha increases throughout training.
And at the end, when W is close to one of the W_i, you can always
round it to the nearest W_i if it has to be equal to one of the specified
values.
Good luck.
K. Frank