Without seeing more of the code it’s a little hard to say for sure, but hearing that model is taking both the inputs and the targets is a little unusual.  Normally, we want the model to only take the inputs.  The loss function should be something separate.
This is some pseudo-code with a little more of the best practice.  Note that the loss_fn is separate from the model.  The reason for this is we don’t really care about keeping the loss function when storing the model – we just want to use it.
Example:
def train_one_epoch(model, optimizer, data_loader, device, validation):    
    model.train()
    loss_fn = torch.nn.BCELoss()
    for i, values in enumerate(data_loader):
        images, targets = values
        images = list(image.to(device) for image in images)
        targets = [{'boxes' : t['boxes'] , 'labels' : t['labels'].to(device)} for t in targets ]

        if validation:
            torch.cuda.empty_cache()
            model.eval()  # Optional, or use the torch.no_grad()
            with torch.no_grad():
                predictions = model(images)
                loss_dict = loss_fn(images, targets)
                losses = sum(loss for loss in loss_dict.values())
                # Do something with losses.
            model.train()
        else:
            optimizer.zero_grad()
            predictions = model(images)
            loss_dict = loss_fn(images, targets)
            losses = sum(loss for loss in loss_dict.values())
            losses.backward()
            optimizer.step()

That being said, you may be able to work around this issue with only a few code changes by doing any of the following:

Reduce your batch size to something smaller.
Perform a validation on a smaller set of items.

Those would be band-aids, but they’re something to try.  I’d do the no_grad thing first.