Thanks for your reply peter. So consider two inputs to this network. One is the Image and the other is a graph, the graph is basically the linkage and features of text on the image. I am using the pytorch faster rcnn as my cnn for object detection on that image. I modify the roi heads in there (specifically the two MLP Heads ) the batch size that comes out from that FasterRcnn is 512 (We use batch size of 1), and the  batch size of the graph network comes out to be different on every pass (depending on the nodes we have in the graph).