Hi Metro!



 metro:

I’m looking for a loss function which will motivate a network to output on the last layer either near 0 or near 1 values.


I don’t understand your use case, but, in general, you can certainly
penalize outputs that are not close to 0 or 1.
Let x be a single output.  You could then try things like:
penalty_0_1 = (x * (1.0 - x))**2
penalty_0_1 = torch.abs (x * (1.0 - x))
penalty_0_1 = ((x * (1.0 - x))**2)**alpha

where the last version includes the parameter alpha that can be
used to tune the shape of the “wells” at 0 and 1.
(x would likely be a pytorch tensor, so you would typically sum
after calculating the penalty to get a single scalar penalty that would
push the individual elements of the tensor towards 0 or 1.)


it would allow the model to be “uncertain” i.e. output values which are extremely close to 0.5 e.g. 0.499999999 or 0.50000000001.


There is no (reasonable) way to avoid having an “uncertain” point
somewhere between 0 and 1 (most likely at 0.5).  But more importantly,
you don’t want the penalty at 0.5 to be so large that it is impractical
for the network output to move from near 0 to near 1 or vice versa.
Suppose that the network output “should” be (near) 1, but happens
to start out near 0.  As the network trains, you want it to be able to
get over the penalty hump, and not stay stuck near 0 forever.  (One
approach to this issue is to multiply your penalty with a weight.  At
the beginning of training start with a small weight so that the network
output can move easily between 0 and 1, but increase the weight as
training progresses so that the network output is forced increasingly
close to 0 or 1.)
Best.
K. Frank