Using affine_grid and grid_sample will give you a behaviour like spatial transformer network. Which works well for some use cases.
The learning of the bounding boxes is a hard problem, especially if you don’t have a ground truth bounding box to learn from.
You may want to look at different attention mechanisms that exist to find ideas.