Multimodal deep learning isn’t one of the easiest domain to handle as the first project.
For an introduction you could start to look at some material in:

  
      

      GitHub
  

  
    

GitHub - pliang279/awesome-multimodal-ml: Reading list for research topics in... 1

  Reading list for research topics in multimodal machine learning - GitHub - pliang279/awesome-multimodal-ml: Reading list for research topics in multimodal machine learning