Hi Learner!



 learner47:

z is the binary representation of the class with m bits. As an example, for m=3 z runs from 000 to 111. So z has a particular structure in that it is the exhaustive list of combinations of 1s and 0s.


Then (barring other noteworthy details) I would understand this as a
multi-label, three-class classification problem.
To illustrate what I meant by “additional structure”, consider the mapping
between a set of three bits and the integers 0, …, 7, where, e.g., “011” is
the binary representation of the integer 3.
Let’s say your ground-truth z is “011”, while your predicted z is “100”.
Would you consider this a maximally bad prediction because all three
of the bits were predicted incorrectly?  Or would you consider it a pretty
good (but not perfect) predictions because the ground-truth z of “011”
corresponds to the integer 3, while the predicted z, “100”, corresponds
to 4, which is only off by 1?
I’m presuming the former, so viewing this as a multi-label classification
problem (rather than, say, a regression) is appropriate.


One, for this to work should the activation function at the end of my network be nn.Sigmoid


BCEWithLogitsLoss is the “go-to” loss function for the multi-label problem.
(There are other reasonable loss functions, but I would use this unless I
had good reason to think that another loss function would be better, and,
even then, I would want to demonstrate that the alternative loss function
did, if fact, perform better.)
If you use BCEWithLogitsLoss, then, no, you do not want a Sigmoid
for the output of your network.


(I checked the documentation and found that nn.Sigmoid is added internally in the loss function)?


You do not want the Sigmoid because, as you note, the Sigmoid is, in
effect, included in BCEWithLogitsLoss.  (The numerically-less-stable
BCELoss does not have the Sigmoid built in.)


And two, is this loss function optimal for the setting?


If my understanding of your problem as being a multi-label, multi-class
classification problem is correct, then, yes, BCEWithLogitsLoss is
reasonably likely to be your “optimal” loss function.


What I mean is does it lead to a trained network, which can perform as well as ||y-ax||^2 being the loss function?


I’m certainly not able to write a mathematical proof showing that one loss
function or another will be better – the whole business of training realistic
neural networks is sufficiently complicated that such choices  are generally
made based on rules of thumb and empirical results.
You can certainly try using both loss functions to train your network and
see which works better.  But don’t not try BCEWithLogitsLoss, at least
as a baseline.


The relation y=ax+b can produce infinite number of training samples, theoretically.


The beauty of this is that you don’t have to worry about overfitting – just
keep training until your model converges to an acceptable accuracy (or
reaches a plateau that could represent the best it can do).
And again, you might want to experiment with the architecture of your
network, probably including some shallower (fewer layers) networks.
Best.
K. Frank