Maybe try this? And directly use sample as the input to the network.
channels = 3
height, width = 224, 224
views = 2
cameras = 4
frames = 1

tgt_rgb = torch.randn(1, frames, channels, height, width)
src_rgb = torch.randn(1, views, frames, channels, height, width)
tgt_bg = torch.randn(1, channels, height, width)
src_bg = torch.randn(1, views, channels, height, width)

src_w2c = torch.randn(1, views, 4, 4)
src_K = torch.randn(1, views, 3, 3)
tgt_w2c = torch.randn(1, 4, 4)
tgt_K = torch.randn(1, 3, 3)
bd = torch.FloatTensor([1, 100])

sample = dict()
# Pack data
sample['tgt_rgb'] = tgt_rgb.to(device)
sample['src_rgb'] = src_rgb.to(device)

sample['src_bg'] = src_bg.to(device)
sample['tgt_bg'] = tgt_bg.to(device)

sample['src_w2c'] = src_w2c.to(device)
sample['src_K'] = src_K.to(device)
sample['tgt_w2c'] = tgt_w2c.to(device)
sample['tgt_K'] = tgt_K.to(device)

sample['bd'] = torch.FloatTensor([
    torch.min(bds[0]) * .9,
    torch.max(bds[0]) * 2.,
]).unsqueeze(0).to(device)

However, one thing is that randomly initializing everything could produce weird outcome since the calculation is based on projective geometry.