Pepi:

How might I improve the model architecture?


You could search for similar use cases and check if some architectures were successfully used before.
If you cannot find such a model, you could try to run experiments and check which change in the architecture is beneficial for the use case. There are also meta-learning approaches, which could try to find a suitable architecture, but I’m unsure if the current methods would suite your use case.
Anyway, lectures such as FastAI provide also some best practices to create models.