Hi Learner!



 learner47:

The network that I am designing needs to able to get back the original m bit length vector (z) upon seeing y (y=ax+b) and a.


Yes, this clarifies things.  Your “ground truth” (your known “target” for
training) is a bit vector, z, of length m (say, m = 3).
Unless z has some additional structure / meaning (that you have not
shared with us, e.g., being the binary representation of an integer), then
you should understand your problem as being a so-called multi-label,
multi-class classification problem.  This means that you have m
classes, any number of which (including none or all of them) can
be “present” in any sample.  That is, any of the m bits can be
independently on.  (“Multi-class” means you have the m classes;
“multi-label” means that the classes are not exclusive, so that any
given sample can carry a label for multiple classes at a time.)
The most common approach to such a problem is to have your
final Linear layer output m logits (one for how strongly each of
the m bits is predicted to be “on”).  Then use BCEWithLogitsLoss
to compare your prediction with your ground-truth z.  The output
of your model will be of type float32 and have shape [nBatch, m].
Your ground-truth target will have the same shape and also be of
type float32 and will have value 1.0 whenever the corresponding
bit of z is set.
(As an aside, regardless of exactly how you choose to approach
your problem, I would start with a shallower architecture – maybe
just one or two hidden layers – and then add depth / layers if I
could show that doing so yielded better performance.)


My implementation through the neural network may be a round about approach to achieve what I want, because I am still a novice.


I’m assuming that this is a toy problem (given your m to n bit
“encoding” followed by the introduction of additive and multiplicative
noise).  If it is a toy problem, then carry on, and use it to practice
building neural networks.
If it represents a real problem, then I suspect that it has enough
structure that there would be a more optimal specialized algorithm
for recovering (filtering) z from y and a (not that I would know
what such an algorithm would be).


There can be at most 2^m classes for z and therefore x.


It is true that a multi-label, m-class problem can be understood as
a single-label 2^m-class problem.  As a general rule, however, you
will be better off treating is as a multi-label problem (unless it has
some special structure your network can take advantage of).
The problem, of course, is the 2^m classes, as you lose information
about how the classes are “correlated,” and the potentially large
number of classes means you have fewer samples per class.
Best.
K. Frank