Because your are using BCEWithLogitsLoss (seemingly reasonable
for your use case), you should get rid of the Sigmoid after your final
Linear layer, fc . ( Sigmoid converts logits to probabilities, likely not
what you want.)

Thank you K. Frank!! I face palmed hard after reading this, I hadn’t considered the effects of that activation layer…
Changing that activation function to LeakyReLU helped, the loss now goes down to 0.05 after 100 epochs over 2 samples.

Other than that, I don’t have much intuition about your use case, or
what “NUM_IMAGES” (apparently 6 ) might mean conceptually.

The network’s input is a tensor of image features representing panoramic views along a path (36 views separated by 30 degrees, with 12 views looking up, 12 looking forward, 12 looking down). NUM_IMAGES is the length of the path (could have been named better), in the example case there are 6 points. The goal of the network is to find which objects are present at a given time step.
The problem is that I don’t have a ground truth for the objects present at each point in the path, but I do know which objects appear in the path. The idea is that a model trained to predict objects present in the path, with an independence assumption between time steps aggregated using max pooling, could be used to classify single points afterwards.

Having said that, the shenanigans where you reshape your data
into a “pseudo-batch” of size batch_size * length , pass it through
your network, and then reshape it back to a batch of batch_size
strike me as a little odd.

To process each point in the path separately, I decided to use a 1d convolution, but that required reshaping the output. Rather than flatten it and lose some of the benefits of the current structure, I tried what you described above.
Originally the plan was to use a 2d convolution, but it seems to require iterating through the data to avoid using multiple points of the path at once.
Since the images do capture a panoramic view, I wanted to capture “interactions” between neighboring images. A fully connected layer could do the trick, but it would be big ~(36 x 2048 x HIDDEN_SIZE), hence the use of convolutions.