I would start with a very simple model (e.g. just conv - pool - relu - linear) and try to overfit a single sample. If this isn’t even working then your training code might have a bug I couldn’t see so far.