I’m trying to understand how to use Transformers for OCR starting with something very basic like MNIST. My thought was to extract image patches from each MNIST image and then feed those into a Transformer with a classifier on the output. For example:
model = torch.nn.Sequential(
          torch.nn.Conv2d(1,16,3,3),
          torch.nn.ReLU(),
          torch.nn.Conv2d(16,32,3,3),
          torch.nn.ReLU()
        )

model(images).shape
> torch.Size([64, 32, 3, 3])

So here is a batch of 64 32-layer patches of 3x3. Given that the input to the Transformer is supposed to be a sequence of vectors, my thought is I should “unroll” the 3x3 patches to end up with a sequence of 32 9-dimensional vectors. Does this make sense?