With Res net in particular the derivation is even easier as the actual layers are already implemented in a separate class that gets passed to a basic resnet.
I was hopping that there is a general approach, that i could apply to multiple models. The standard models typically don’t contain drop out, as they are usually trained with big datasets. But for small Training datasets which are pretty common in practice, dropout helps a lot. So having a function that would adds dropout before/after each relu would be very useful.
model_with_dropout = add_dropout(model, after=“relu”)