  In recent years, developing AI for robotics has raised much attention. The
interaction of vision and language of robots is particularly difficult. We
consider that giving robots an understanding of visual semantics and language
semantics will improve inference ability. In this paper, we propose a novel
method-VSGM (Visual Semantic Graph Memory), which uses the semantic graph to
obtain better visual image features, improve the robot's visual understanding
ability. By providing prior knowledge of the robot and detecting the objects in
the image, it predicts the correlation between the attributes of the object and
the objects and converts them into a graph-based representation; and mapping
the object in the image to be a top-down egocentric map. Finally, the important
object features of the current task are extracted by Graph Neural Networks. The
method proposed in this paper is verified in the ALFRED (Action Learning From
Realistic Environments and Directives) dataset. In this dataset, the robot
needs to perform daily indoor household tasks following the required language
instructions. After the model is added to the VSGM, the task success rate can
be improved by 6~10%.

Robotics;Artificial Intelligence;Computer Vision and Pattern Recognition