Hi @HarrisonWilde!
Damn, it’s impossible to beat @ptrblck’s responsiveness, but I know the answer to this one! 
For a module to be supported by Opacus, the following conditions apply:

Modules with no trainable parameters (eg nn.ReLU, nn.Tanh)
Modules which are frozen. A nn.Module can be frozen in PyTorch by unsetting requires_grad in each of its parameters, ie for p in module.parameters(): p.requires_grad = False.
Explicitly supported modules (we keep a dictionary in opacus.SUPPORTED_LAYERS), eg nn.Conv2d.
Any complex nn.Module that contains only supported nn.Modules. This means that most models will be compatible, given that we support most of the common building blocks. This however also means that Opacus support depends on how a specific nn.Module is implemented. For example, nn.LSTM could be written by using nn.Linear (which we support), but its actual implementation does not use it (so that it can fuse operators and be faster). Any layer that needs a rewrite to be supported is in the /layers folder.

So in this case, the issue is that your Decoder class has a nn.Parameter of its own, the self.sigma!
Judging at how you use it, it looks like you are basically using it as InstanceNorm (correct me if I’m wrong!). We do support nn.InstanceNorm, nn.LayerNorm and nn.GroupNorm, so maybe you could normalize using one of those modules instead?