I think I got it right now.
According to docs https://pytorch.org/docs/stable/nn.html#batchnorm1d 7 and quoting:

Parameters


num_features – C from an expected input of size (N,C,L) or L from input of size (N,L)


So I figured out that as we need 3d input and BatchNorm1d uses C as the num_features in three dimensional input, the singular dimension should be the last one.
So instead of
feat = self.bottleneck(global_feat.unsqueeze(1))   # Which gives [bs, 1, 2048] 
I just did:
feat = self.bottleneck(global_feat.unsqueeze(-1))   # Which gives [bs, 2048, 1]
No more errors and training seems to run smoothly with SyncBatchNorm as well. Hope, this helps someone.