If that does what you want, it probably is correct.
An easier way to express it would be something like
adiff = torch.abs(output-target)
batch_max = 0.2*torch.max(adiff).data[0]
sqdiff = (adiff*adiff+batch_max*batch_max)/(2*batch_max)
return adiff.clamp(max=batch_max)+(sqdiff-1).clamp(min=batch_max)

Note though that clamp is not differentiable w.r.t. the bounds, but your definition of batch_max isn’t, either. It probably does not matter much.
Best regards
Thomas