How is your training accuracy? Is it also at approx. 50%?
It’s a bit strange to use 320 filters in the first conv layer, than back to 64 and up again to 1024.
Did you want to use 640 instead?
If so, it also seems to be a bit high, but would maybe work better than the model now.
Could you plot your training loss and accuracy?
This would make it a bit more easy to debug.