Thanks for that, it helped a lot to know that the code fundamentally is not wrong. I figured out that the issue is simply that there are not enough conv-relu-pool layers to reduce the feature map to a reasonable size…the model had 900 million parameters haha.