Hi,
Im using Opacus to make CT-GAN (GitHub - sdv-dev/CTGAN: Conditional GAN for generating synthetic tabular data.) differntial private.
There is already an implementation who does this:  (smartnoise-sdk/dpctgan.py at main · opendp/smartnoise-sdk · GitHub 6)
However, they use an older version of opacus (v0.9) and CTGAN(v.0.2.2.dev1).
I used their method to make the newest version of CTGAN differential private with the newest opacus version. Unfortunatly i run into the following error:
.../CTGAN_DP/DP_CTGAN.py", line 309, in fit
    loss_d.backward()
  File ".../lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File ".../lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File ".../python3.7/site-packages/opacus/grad_sample/grad_sample_module.py", line 197, in capture_backprops_hook
    module, backprops, loss_reduction, batch_first
  File ".../python3.7/site-packages/opacus/grad_sample/grad_sample_module.py", line 234, in rearrange_grad_samples
    A = module.activations.pop()
IndexError: pop from empty list

The function in question is:
 def rearrange_grad_samples(
        self,
        module: nn.Module,
        backprops: torch.Tensor,
        loss_reduction: str,
        batch_first: bool,
    ) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Rearrange activations and grad_samples based on loss reduction and batch dim

        Args:
            module: the module for which per-sample gradients are computed
            backprops: the captured backprops
            loss_reduction: either "mean" or "sum" depending on whether backpropped
            loss was averaged or summed over batch
            batch_first: True is batch dimension is first
        """
        if not hasattr(module, "activations"):
            raise ValueError(
                f"No activations detected for {type(module)},"
                " run forward after add_hooks(model)"
            )

        batch_dim = 0 if batch_first or type(module) is LSTMLinear else 1

        if isinstance(module.activations, list):
                A = module.activations.pop()
        else:
            A = module.activations

        if not hasattr(module, "max_batch_len"):
            # For packed sequences, max_batch_len is set in the forward of the model (e.g. the LSTM)
            # Otherwise we infer it here
            module.max_batch_len = _get_batch_size(module, A, batch_dim)

        n = module.max_batch_len
        if loss_reduction == "mean":
            B = backprops * n
        elif loss_reduction == "sum":
            B = backprops
        else:
            raise ValueError(
                f"loss_reduction = {loss_reduction}. Only 'sum' and 'mean' losses are supported"
            )

        # No matter where the batch dimension was, .grad_samples will *always* put it in the first dim
        if batch_dim != 0:
            A = A.permute([batch_dim] + [x for x in range(A.dim()) if x != batch_dim])
            B = B.permute([batch_dim] + [x for x in range(B.dim()) if x != batch_dim])

        return A, B

This does not happen with opacus v.09.
I investigated and found that module.activations is popped until only an empty list for the first module  is left and then produces this error.
I hacked my way around this issue as follows:
  if isinstance(module.activations, list):
            #print(len(module.activations))
            if len(module.activations) > 1:
                A = module.activations.pop()
            else:
                A = module.activations[0]
        else:
            A = module.activations

Meaning, if  module.activations is left with one element instead of an empty list at least training works.
My question is: Am I breaking anything important doing this and could this be a potential subcase which was not accounted for.
Or do I have to change something else in the model ?
The model I try to train with opacus is basically just a composition of  n*(nn.Liner nn.Relu nn.Dropout) which should be fine I think.
Thanks for a reply 
Have a great day.