It looks like you already tried to double the width of your conv filters. Did it improve anything?
If not, it’s just a wild idea and could fail totally, but have you thought about cutting the image and use the second part as an additional channel?
E.g. your current input size [1, 48, 84] would become [2, 48, 42].