Hey guys,
These are really good responses.
I am also doing something similar. I am training a Language Model using RoBERTa on a fashion items’ description corpus. This way the model should learn embeddings for many common fashion terms like dresses, pants etc. and more specifically, their sub-types like floral dress, abstract dress, animal dress etc. The embeddings obtained in this way should be context-aware since they were trained on such specific data.
Next, I have an image classifier which tells me given a product image which category i.e. floral, animal etc. it belongs to. I was thinking of using these text embeddings along with the feature vectors of the image classifier to obtain a dense representation for a particular product.
So, my question is on the same lines as okkular. @lewtun, one way as you mentioned was to aggregate the subword embeddings to get one single embedding; however you’ve mentioned in your reply to okkular



 lewtun:

so trying both approaches would be a good test 


Can you please elaborate what is the other approach please?
Also @BramVanroy does the approach that I mentioned above take into account the context of the fashion language for the downstream task of dense vector representation for a product? I can understand using pretrained BERT/ RoBERTa might not be a good idea but after fine-tuning it this way, it should capture the context alright, correct?
Thanks & Regards,
Vinayak.