I have this dataset which contains some 10000 shirts and their color (blue, black,red…) and pattern (checks, stripes, solid…). There are total 12 different classes in color and 8 in pattern. I am able to make a model of resnet18 (training all layers) to identify color and another resnet18 to identify pattern. For these two models I use Cross-entropy loss and the results are good. There are 12 outputs for color-resnet and 8 outputs for pattern-resnet.
But when I want to do both things using single resnet (12+8 outputs), using BCEwithlogits loss, the network is not giving proper results (infact the results are very bad) . Any idea as why is this happening?