readme: # Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
  
  [![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
  
  <p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
  
  This project attempted to implement the paper **[Putting NeRF on a Diet](https://arxiv.org/abs/2104.00677)** (DietNeRF) in JAX/Flax.
  DietNeRF is designed for rendering quality novel views in few-shot learning scheme, a task that vanilla NeRF (Neural Radiance Field) struggles.
  To achieve this, the author coins **Semantic Consistency Loss** to supervise DietNeRF by prior knowledge from CLIP Vision Transformer. Such supervision enables DietNeRF to learn 3D scene reconstruction with CLIP's prior knowledge on 2D views.  
  
  Besides this repo, you can check our write-up and demo here:
  - ✍️ **[Write-up in Notion](https://steep-cycle-f6b.notion.site/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)**: more details of DietNeRF and our experiments
  - ✨ **[Demo in Hugging Face Space](https://huggingface.co/spaces/flax-community/DietNerf-Demo)**: showcase our trained DietNeRFs by Streamlit
  
  ## 🤩 Demo
  1. You can check out [our demo in Hugging Face Space](https://huggingface.co/spaces/flax-community/DietNerf-Demo)
  2. Or you can set up our Streamlit demo locally (model checkpoints will be fetched automatically upon startup)
  ```shell
  pip install -r requirements_demo.txt
  streamlit run app.py
  ```
  
  <p align="center"><img width="600" height="400" alt="Streamlit Demo" src="assets/space_demo.png"></p>
  
  ## ✨ Implementation
  
  Our code is written in JAX/ Flax and mainly based upon [jaxnerf](https://github.com/google-research/google-research/tree/master/jaxnerf) from Google Research. The base code is highly optimized in GPU & TPU. For semantic consistency loss, we utilize pretrained CLIP Vision Transformer from [transformers](https://github.com/huggingface/transformers) library.
  To learn more about DietNeRF, our experiments and implementation, you are highly recommended to check out our very detailed **[Notion write-up](https://www.notion.so/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)**!
  
  <p align="center"><img width="500" height="600" alt="스크린샷 2021-07-04 오후 4 11 51" src="assets/report_thumbnail.png"></p> 
  
  ##  🤗 Hugging Face Model Hub Repo
  You can also find our project on the [Hugging Face Model Hub Repository](https://huggingface.co/flax-community/putting-nerf-on-a-diet/).
  
  Our JAX/Flax implementation currently supports:
  
  <table class="tg">
  <thead>
    <tr>
      <th class="tg-0lax"><span style="font-weight:bold">Platform</span></th>
      <th class="tg-0lax" colspan="2"><span style="font-weight:bold">Single-Host GPU</span></th>
      <th class="tg-0lax" colspan="2"><span style="font-weight:bold">Multi-Device TPU</span></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td class="tg-0lax"><span style="font-weight:bold">Type</span></td>
      <td class="tg-0lax">Single-Device</td>
      <td class="tg-0lax">Multi-Device</td>
      <td class="tg-0lax">Single-Host</td>
      <td class="tg-0lax">Multi-Host</td>
    </tr>
    <tr>
      <td class="tg-0lax"><span style="font-weight:bold">Training</span></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
    </tr>
    <tr>
      <td class="tg-0lax"><span style="font-weight:bold">Evaluation</span></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
      <td class="tg-0lax"><img src="http://storage.googleapis.com/gresearch/jaxnerf/check.png" alt="Supported" width=18px height=18px></td>
    </tr>
  </tbody>
  </table>
  
  ## 💻 Installation
  
  ```bash
  # Clone the repo
  git clone https://github.com/codestella/putting-nerf-on-a-diet
  # Create a conda environment, note you can use python 3.6-3.8 as
  # one of the dependencies (TensorFlow) hasn't supported python 3.