Thanks so much! I think I got something working now. Only problem is that inference is super slow because it’s autoregressive.
I’m doing:
        tgt = torch.ones(x.size(0), 1, 128) * -1

        for i in range(x.size(1)):
            tgt = torch.randn(tgt.size())
            pred = reconstruct_spect_model(x, tgt)

            tgt = torch.cat((tgt, pred[:, -1, :].unsqueeze(1)), 1)

I assume I don’t need any tgt_mask during inference?