Hi Arturas!



 Arturas_Druteika:

if it is a good, neutral or bad practice to specify loss as well as optimizer inside the model class


Including your loss and/or optimizer in your model class won’t break
anything or cause inefficiency.  So from a mechanical or technical
perspective, it’s okay.
However, I would say that doing so is bad practice from a stylistic
or design perspective.
The main idea is “separation of concerns.”  To my mind the purpose
of a model is to take an input and produce a prediction.  At inference
time (in contrast to training), you won’t be using a loss function nor
optimizer, so they don’t really belong in the model class.  In a similar
vein, when performing evaluation, you might choose to use a different
“loss” function (perhaps a non-differentiable accuracy function) than
the one you use for training.  Or you might use more than one accuracy
metric.
It is also true that an important purpose of a model is to be able to be
trained – that’s why models have Parameters that have grads and
layers that have backward() functions.  It seems to me to be a sound
stylistic / design choice to have these be part of the internals of the
model class.
But the details of how to train the model (e.g., what optimization
algorithm) can be separated from the infrastructure that makes a
model able to be trained (e.g., tensors that have a grad property),
and the guideline of “separation of concerns” suggests that when
things can be conceptually and straightforwardly separated, they
should be separated and placed in separate classes.
So “clean design” (whatever that means) suggests that the loss and
optimizer aren’t “naturally” part of the model, so they shouldn’t be
part of the model class.
Best.
K. Frank