I would like to train an image classifier that uses a positive floating point number (1-dimensional data of size 1) as the second input during training and inference. I am not an expert on deep learning, but I have identified from literature that what I want to do is probably called “early fusion”, i.e. combining two different types of data (or modalities) when extracting features from an image.
The purpose of supplying the float is to help improve classification accuracy; the number on its own is not enough to make a prediction. A similar example would perhaps be providing the gender of a patient whose medical image is used to train a tumor classifier. The only difference would be that gender comes only in a few discrete values.
Examples I’ve seen in this forum appear to be more complicated than mine. For example, the second input might contain enough information to extract features or even train a second classifier, such as an MLP and then merge the predictions [1, 2, 3, 4]. Therefore, the only solution I could think of is that I concatenate the float y to a flattened image, like so:
# subclassed torchvision.models resnet50
def forward(self, x, y):
...
x = self.avgpool(x)
x = torch.flatten(x, 1)
x = torch.cat((x, y), dim=1)
x = self.fc(x)  # nn.Linear(in_features + 1, num_classes)
return x

However, doing this has actually caused a drop in classification accuracy after training, when compared with a resnet50 model that accepts only image input. My intuition also tells me that the 1D float input should somehow be normalized.
I would be grateful for suggestions on how to approach this problem when writing the forward method. I have already written my own dataset with a custom __getitem__ method.