@sayanghosh @amin-nejad sorry was busy last weeks. I can gladly share the snippets, concerning PR should I rather commit to a tutorial in a Jupyter Notebook or think about how to integrate it directly into the framework without the need to overwrite different hooks?
What I have for now (prototyping code, the hparams are the defined params in my PL Model):
class LightningCLI_Custom(LightningCLI):
[...]
def before_fit(self):
    """Hook to run some code before fit is started"""
    # possible because self.datamodule and self.model are instantiated beforehand
    # in LightningCLI.instantiate_trainer(self) -- see docs

    # TODO: why do I have to call them explictly here 
    #       -- in docs not mentioned (not found in .trainerfit())
    self.datamodule.prepare_data()
    self.datamodule.setup()

    if self.model.hparams.dp:
        if self.model.hparams.dp_tool == "opacus":
            # NOTE: for now the adding to the optimizers is in model.configure_optimizers()
            # because at this point model.configure_optimizers() wasn't called yet. 
            # That's also why we save n_accumulation_steps as a model parameter.
            sample_rate = self.datamodule.batch_size/len(self.datamodule.dataset_train)
            if self.model.hparams.virtual_batch_size >= self.model.hparams.batch_size: 
                self.model.n_accumulation_steps = int(
                    self.model.hparams.virtual_batch_size/self.model.hparams.batch_size
                )
            else: 
                self.model.n_accumulation_steps = 1 # neutral
                print("Virtual batch size has to be bigger than real batch size!")

            # NOTE: For multiple GPU support: see PL code. 
            # For now we only consider shifting to cuda, if there's at least one GPU ('gpus' > 0)
            self.model.privacy_engine = PrivacyEngine(
                self.model.model,
                sample_rate=sample_rate * self.model.n_accumulation_steps,
                target_delta = self.model.hparams.target_delta,
                target_epsilon=self.model.hparams.target_epsilon,
                epochs=self.trainer.max_epochs,
                max_grad_norm=self.model.hparams.L2_clip,
            ).to("cuda:0" if self.trainer.gpus else "cpu")
            # necessary if noise_multiplier is dynamically calculated by opacus
            # in order to ensure that the param is tracked
            self.model.hparams.noise_multiplier = self.model.privacy_engine.noise_multiplier
            print(f"Noise Multiplier: {self.model.privacy_engine.noise_multiplier}")

        else: 
            print("Use either 'opacus' or 'deepee' as DP tool.")

        # self.fit_kwargs is passed to self.trainer.fit() in LightningCLI.fit(self)
        self.fit_kwargs.update({
            'model': self.model
        })

        # in addition to the params saved through the model, save some others from trainer
        important_keys_trainer = ['gpus', 'max_epochs', 'deterministic']
        self.trainer.logger.experiment.config.update(
            {
                important_key:self.config['trainer'][important_key] 
                for important_key in important_keys_trainer
            }
        )
        # the rest is stored as part of the SaveConfigCallbackWandB
        # (too big to store every metric as part of the above config)
    
    # track gradients, etc.
    self.trainer.logger.experiment.watch(self.model)

Then in my model:
class LitModelDP(LightningModule):
    def __init__(...):
      [...]
      # disable auto. backward to be able to add noise and track 
      # the global grad norm (also in the non-dp case, lightning 
      # only does per param grad tracking)
      self.automatic_optimization = False

     # manual training step, eval, etc.

     def configure_optimizers(self):
        optims = {}
        # DeePee: we want params from wrapped mdoel
        # self.paramters() -> self.model.wrapped_model.parameters()
        if self.hparams.optimizer=='sgd':
            optimizer = torch.optim.SGD(
                self.model.parameters(), 
                **self.hparams.opt_kwargs,
            )
        elif self.hparams.optimizer=='adam':
            optimizer = torch.optim.Adam(
                self.model.parameters(), 
                **self.hparams.opt_kwargs,
            )

        if self.hparams.dp_tool=='opacus' and self.hparams.dp:
            self.privacy_engine.attach(optimizer)

        optims.update({'optimizer': optimizer})
        [...]
        return optims

Calling the LightningCLI at the end
cli = LightningCLI_Custom(model_class=LitModelDP, datamodule_class=[...])

I’m very open to feedback and would also gladly help to integrate that into PL.