Appreciate the feedback @erroneus!  I had a question regarding your comment about it not working for the cross-modal part because A and B are drawn independently.  I’m working with medical imaging, and A and B are different modalities but are of the same patient, so the distributions shouldn’t be totally independent despite the difference in imaging modality.  (see picture)

NetworkSchema.png1268×399 130 KB

I borrowed part of your diagram from appendix A.2 Figure 6 for sake of clarity. No plagiarism intended, happy to repost without it if you’d prefer.
Or are you referring to my bit of code:
loss1 = loss_calc(loc_moving, glob_atlas)
loss2 = loss_calc(loc_atlas, glob_atlas)
loss = (loss2 + loss1)

I’m not supposed to concatenate the local feature maps together, right? i.e.
loc_maps = torch.cat((loc_atlas, loc_moving))
loss = loss_calc(loc_maps, glob_atlas) 

I will certainly do some tuning to optimize receptive field size as well. Thanks again!
-Drew