I see, so this objective is requiring a lot out of your network. I suspect a wider decoder will eventually be useful, but I’d not be concerned with that yet.  Try a full scale train/validation and see how it goes.  If you pulled an off-the-shelf U-net model it should mostly work for your application.