[`Detr`] Fix detr BatchNorm replacement issue
# What does this PR do?

Fixes the current failing CI on #25077 / related failing jobs: https://app.circleci.com/pipelines/github/huggingface/transformers/69452/workflows/999f3686-2d9a-4324-bed6-1c858f4d8246/jobs/871127 

In #25077 I decided to [add a property method `current_adapter`](https://github.com/younesbelkada/transformers/blob/peft-integration-attempt-2/src/transformers/adapters/peft_mixin.py#L156) to easily switch between adapters. This leads to failing CI because `PreTrainedModel` will inherit from `AdapterMixin` (that will contain that attribute) and `replace_batch_norm` loops over `dir(model)` and calls `getattr(model, attr_str)`, therefore checks for all available attributes including `current_adapter`. 

I can also change the property method to an instance method to avoid this issue, but I find it cleaner to do the module replacement in a pure PyTorch manner rather than using `dir(model)` which can cause weird behaviours in the future .

Can confirm slow DETR / DETA integration tests pass with this change

cc @sgugger @amyeroberts 
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25230). All of your documentation changes will be reflected on that endpoint.