I am trying to do kind of video captioning but not exactly that.
In my dataset, each image has exactly two objects and one relationship among them.
For example:- person-ride-bicycle i.e. each image contains following pair: object1-relationship-object2
object 1 and object 2 have 35 classes/labels to choose from
relationship have 85 classes/labels to choose from
More about dataset-
let’s consider first folder 0000. It has 30 images, these are frames of a same video (all will have same relationship let’s say: person-ride-bicycle)
i have annotations corresponding to each the folder. for eg,
for the folder 0000, i have annotation 12-45-7 (object1-relationship-object2)
for the folder 0001, i have annotation 22-54-30
and so on…
Architecture looks something like this…
relationship below is dog-plays-frisbee

image927×444 97.8 KB