ViT+CLIP+NeRF: Fewshot Learning, Putting NeRF on a Diet

Is anyone interested in Computer Vision Field? 
Our team project goal is the implementation of this paper:
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis :
  
      

      arXiv.org
  

  
    

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis 31

  We present DietNeRF, a 3D neural scene representation estimated from a few
images. Neural Radiance Fields (NeRF) learn a continuous volumetric
representation of a scene through multi-view consistency, and can be rendered
from novel viewpoints by ray...


  

  
    
    
  

  


With this project, We will code a 3D neural scene representation (NeRF: Neural Radiances Field) estimated from a few images. Which is based on extracting the semantic information using a pre-trained visual encoder such as CLIP, a Vision Transformer.

1. Languages and Skills

Languages : Python, Pytorch, JAX
Other Skills : Git, Github, Cloud Computing Experience


2. Model / Baseline Code


ViT, CLIP : https://github.com/openai/CLIP


Meatlearning-NeRF (Learned Initialization Paper : JAX Based NeRF Code) :
https://github.com/tancik/learnit


NeRF (Pytorch Based NeRF Code) : https://github.com/yenchenlin/nerf-pytorch



3. Datasets


Phototourism dataset:
https://www.cs.ubc.ca/~kmyi/imw2020/data.html


Realistic Synthetic dataset :
 https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1


DTU multi-view stereo (MVS) dataset :
http://roboimagedata.compute.dtu.dk/?page_id=36



4. Challenges

JAX based NeRF Code
Data Preprocessing
CLIP / NeRF training with TPU
Augmentation CLIP to NeRF with JAX based code


5. Project Goal Output
The desired project output is an implementation of the Fewshot learning NeRF model which is using CLIP+ViT (JAX based code)

6. References

NeRF : https://www.matthewtancik.com/nerf

Putting NeRF in a diet : https://arxiv.org/abs/2104.00677

Learned Initialization : https://arxiv.org/abs/2012.02189



7. Team Mindset
-Enjoy! 
-Passionate 
-Collaborative 
-Learning Oriented