Could you give a small example, how these l_2 losses are calculated in the modules and where/how you would like to disable them?
Maybe detaching the critical tensors based on the condition would work.