Fix typos and update README.md
Browse files
README.md
CHANGED
@@ -1,18 +1,17 @@
|
|
1 |
# Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
|
|
|
2 |
[](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
|
3 |
|
4 |
<p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
|
5 |
|
6 |
Welcome to Putting NeRF on a Diet Project!
|
7 |
-
This project is
|
8 |
-
The
|
9 |
-
The
|
10 |
-
The Diet NeRF result outperforms the original NeRF in 3D reconstruction and neural rendering with only few images.
|
11 |
|
12 |
|
13 |
-
## 🤗 Hugging Face Hub Repo
|
14 |
-
|
15 |
-
[https://huggingface.co/flax-community/putting-nerf-on-a-diet/](https://huggingface.co/flax-community/putting-nerf-on-a-diet/)
|
16 |
|
17 |
Our JAX/Flax implementation currently supports:
|
18 |
|
@@ -53,36 +52,31 @@ Our JAX/Flax implementation currently supports:
|
|
53 |
|
54 |
- Streamlit Space Demo
|
55 |
|
56 |
-
You can check our Streamlit Space demo
|
57 |
With any input camera pose, we can render the novel view synthesis.
|
58 |
-
[https://huggingface.co/spaces/flax-community/DietNerf-Demo](https://huggingface.co/spaces/flax-community/DietNerf-Demo)
|
59 |
|
60 |
- Colab Demo
|
61 |
|
62 |
-
Moreover, we
|
63 |
-
You need colab pro account for running our model on the colab(
|
64 |
-
[https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
|
65 |
-
](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
|
66 |
-
)
|
67 |
-
|
68 |
|
69 |
## 💻 Installation
|
70 |
|
71 |
```bash
|
72 |
# Clone the repo
|
73 |
-
|
74 |
# Create a conda environment, note you can use python 3.6-3.8 as
|
75 |
# one of the dependencies (TensorFlow) hasn't supported python 3.9 yet.
|
76 |
conda create --name jaxnerf python=3.6.12; conda activate jaxnerf
|
77 |
# Prepare pip
|
78 |
conda install pip; pip install --upgrade pip
|
79 |
# Install requirements
|
80 |
-
pip install -r
|
81 |
# [Optional] Install GPU and TPU support for Jax
|
82 |
# Remember to change cuda101 to your CUDA version, e.g. cuda110 for CUDA 11.0.
|
83 |
-
pip install --upgrade jax
|
84 |
# install flax and flax-transformer
|
85 |
-
pip install flax
|
86 |
```
|
87 |
|
88 |
## ⚽ Dataset
|
@@ -93,8 +87,7 @@ in the place you like. Let's assume they are placed under `/tmp/jaxnerf/data/`.
|
|
93 |
|
94 |
## 💖 Methods
|
95 |
|
96 |
-
You can check
|
97 |
-
* 👉👉 VEEEERY Detail DietNeRF Explaination Docs : https://www.notion.so/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745
|
98 |
|
99 |
<p align="center"><img width="400" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/124376591-b312b780-dce2-11eb-80ad-9129d6f5eedb.png"></p>
|
100 |
|
@@ -120,7 +113,7 @@ python -m train \
|
|
120 |
```
|
121 |
You can toggle the semantic loss by “use_semantic_loss” in configuration files.
|
122 |
|
123 |
-
## 💎
|
124 |
|
125 |
### ❗ Rendered Rendering images by 8-shot learned Diet-NeRF
|
126 |
|
@@ -133,9 +126,9 @@ DietNeRF has a strong capacity to generalise on novel and challenging views with
|
|
133 |
|
134 |
### ❗ Rendered GIF by occluded 14-shot learned NeRF and Diet-NeRF
|
135 |
|
136 |
-
We made
|
137 |
The reconstruction quality can be compared with this experiment.
|
138 |
-
|
139 |
|
140 |
#### Training poses
|
141 |
<img width="1400" src="https://user-images.githubusercontent.com/26036843/126111980-4f332c87-a7f0-42e0-a355-8e77621bbca4.png">
|
@@ -163,9 +156,9 @@ Diet NeRF shows better quailty than Original NeRF when It is occulused.
|
|
163 |
| Project Managing | [Stella Yang](https://github.com/codestella) To Watch Our Project Progress, Please Check [Our Project Notion](https://www.notion.so/Putting-NeRF-on-a-Diet-e0caecea0c2b40c3996c83205baf870d) |
|
164 |
| NeRF Team | [Stella Yang](https://github.com/codestella), [Alex Lau](https://github.com/riven314), [Seunghyun Lee](https://github.com/sseung0703), [Hyunkyu Kim](https://github.com/minus31), [Haswanth Aekula](https://github.com/hassiahk), [JaeYoung Chung](https://github.com/robot0321) |
|
165 |
| CLIP Team | [Seunghyun Lee](https://github.com/sseung0703), [Sasikanth Kotti](https://github.com/ksasi), [Khali Sifullah](https://github.com/khalidsaifullaah) , [Sunghyun Kim](https://github.com/MrBananaHuman) |
|
166 |
-
| Cloud TPU Team | [Alex Lau](https://github.com/riven314), [Aswin Pyakurel](https://github.com/masapasa)
|
167 |
|
168 |
-
* Extremely Don't Sleep Contributors
|
169 |
|
170 |
## 😎 What we improved from original JAX-NeRF : Innovation
|
171 |
- Neural rendering with fewshot images
|
@@ -218,16 +211,14 @@ This project is based on “Putting NeRF on a Diet”.
|
|
218 |
|
219 |
## ❤️ Special Thanks
|
220 |
|
221 |
-
|
222 |
-
Our Project is started in the HuggingFace X GoogleAI (JAX) Community Week Event.
|
223 |
-
https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104
|
224 |
|
225 |
Thank you for our mentor Suraj and organizers in JAX/Flax Community Week!
|
226 |
Our team grows up with this community learning experience. It was wonderful time!
|
227 |
|
228 |
<img width="250" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126369170-5664076c-ac99-4157-bc53-b91dfb7ed7e1.jpeg">
|
229 |
|
230 |
-
Common Computer AI(https://comcom.ai/
|
231 |
Thank you so much for your support!
|
232 |
<img width="250" alt="스크린샷" src="https://user-images.githubusercontent.com/77657524/126914984-d959be06-19f4-4228-8d3a-a855396b2c3f.jpeg">
|
233 |
|
|
|
1 |
# Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
|
2 |
+
|
3 |
[](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
|
4 |
|
5 |
<p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
|
6 |
|
7 |
Welcome to Putting NeRF on a Diet Project!
|
8 |
+
This project is a JAX/Flax based code implementation of the paper [Putting NeRF on a Diet](https://arxiv.org/abs/2104.00677). The model generates the novel view synthesis rendering (NeRF: Neural Radiances Field) with Few-shot learning scheme.
|
9 |
+
The semantic loss uses the pre-trained CLIP Vision Transformer embedding. This information can give a 2D supervision for 3D.
|
10 |
+
The Diet NeRF result outperforms the original NeRF in 3D reconstruction and neural rendering with only few training images.
|
|
|
11 |
|
12 |
|
13 |
+
## 🤗 Hugging Face Model Hub Repo
|
14 |
+
You can also find our project on the [Hugging Face Model Hub Repository](https://huggingface.co/flax-community/putting-nerf-on-a-diet/).
|
|
|
15 |
|
16 |
Our JAX/Flax implementation currently supports:
|
17 |
|
|
|
52 |
|
53 |
- Streamlit Space Demo
|
54 |
|
55 |
+
You can check our [Streamlit Space demo](https://huggingface.co/spaces/flax-community/DietNerf-Demo).
|
56 |
With any input camera pose, we can render the novel view synthesis.
|
|
|
57 |
|
58 |
- Colab Demo
|
59 |
|
60 |
+
Moreover, we have also prapared a [Colab notebook](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing).
|
61 |
+
You need colab pro account for running our model on the colab (To avoid out-of-memory issue)
|
|
|
|
|
|
|
|
|
62 |
|
63 |
## 💻 Installation
|
64 |
|
65 |
```bash
|
66 |
# Clone the repo
|
67 |
+
git clone https://github.com/codestella/putting-nerf-on-a-diet
|
68 |
# Create a conda environment, note you can use python 3.6-3.8 as
|
69 |
# one of the dependencies (TensorFlow) hasn't supported python 3.9 yet.
|
70 |
conda create --name jaxnerf python=3.6.12; conda activate jaxnerf
|
71 |
# Prepare pip
|
72 |
conda install pip; pip install --upgrade pip
|
73 |
# Install requirements
|
74 |
+
pip install -r requirements.txt
|
75 |
# [Optional] Install GPU and TPU support for Jax
|
76 |
# Remember to change cuda101 to your CUDA version, e.g. cuda110 for CUDA 11.0.
|
77 |
+
!pip install --upgrade jax "jax[cuda110]" -f https://storage.googleapis.com/jax-releases/jax_releases.html
|
78 |
# install flax and flax-transformer
|
79 |
+
pip install flax transformers[flax]
|
80 |
```
|
81 |
|
82 |
## ⚽ Dataset
|
|
|
87 |
|
88 |
## 💖 Methods
|
89 |
|
90 |
+
* 👉👉 You can check VEEEERY detailed explanation about our project on [Notion Report](https://www.notion.so/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)
|
|
|
91 |
|
92 |
<p align="center"><img width="400" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/124376591-b312b780-dce2-11eb-80ad-9129d6f5eedb.png"></p>
|
93 |
|
|
|
113 |
```
|
114 |
You can toggle the semantic loss by “use_semantic_loss” in configuration files.
|
115 |
|
116 |
+
## 💎 Experimental Results
|
117 |
|
118 |
### ❗ Rendered Rendering images by 8-shot learned Diet-NeRF
|
119 |
|
|
|
126 |
|
127 |
### ❗ Rendered GIF by occluded 14-shot learned NeRF and Diet-NeRF
|
128 |
|
129 |
+
We made artificial occlusion on the right side of image (Only picked left side training poses).
|
130 |
The reconstruction quality can be compared with this experiment.
|
131 |
+
DietNeRF shows better quality than Original NeRF when It is occluded.
|
132 |
|
133 |
#### Training poses
|
134 |
<img width="1400" src="https://user-images.githubusercontent.com/26036843/126111980-4f332c87-a7f0-42e0-a355-8e77621bbca4.png">
|
|
|
156 |
| Project Managing | [Stella Yang](https://github.com/codestella) To Watch Our Project Progress, Please Check [Our Project Notion](https://www.notion.so/Putting-NeRF-on-a-Diet-e0caecea0c2b40c3996c83205baf870d) |
|
157 |
| NeRF Team | [Stella Yang](https://github.com/codestella), [Alex Lau](https://github.com/riven314), [Seunghyun Lee](https://github.com/sseung0703), [Hyunkyu Kim](https://github.com/minus31), [Haswanth Aekula](https://github.com/hassiahk), [JaeYoung Chung](https://github.com/robot0321) |
|
158 |
| CLIP Team | [Seunghyun Lee](https://github.com/sseung0703), [Sasikanth Kotti](https://github.com/ksasi), [Khali Sifullah](https://github.com/khalidsaifullaah) , [Sunghyun Kim](https://github.com/MrBananaHuman) |
|
159 |
+
| Cloud TPU Team | [Alex Lau](https://github.com/riven314), [Aswin Pyakurel](https://github.com/masapasa), [JaeYoung Chung](https://github.com/robot0321), [Sunghyun Kim](https://github.com/MrBananaHuman) |
|
160 |
|
161 |
+
* Extremely Don't Sleep Contributors 🤣: [Seunghyun Lee](https://github.com/sseung0703), [Alex Lau](https://github.com/riven314), [Stella Yang](https://github.com/codestella), [Haswanth Aekula](https://github.com/hassiahk)
|
162 |
|
163 |
## 😎 What we improved from original JAX-NeRF : Innovation
|
164 |
- Neural rendering with fewshot images
|
|
|
211 |
|
212 |
## ❤️ Special Thanks
|
213 |
|
214 |
+
Our Project is started in the [HuggingFace X GoogleAI (JAX) Community Week Event](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104).
|
|
|
|
|
215 |
|
216 |
Thank you for our mentor Suraj and organizers in JAX/Flax Community Week!
|
217 |
Our team grows up with this community learning experience. It was wonderful time!
|
218 |
|
219 |
<img width="250" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126369170-5664076c-ac99-4157-bc53-b91dfb7ed7e1.jpeg">
|
220 |
|
221 |
+
[Common Computer AI](https://comcom.ai/en/) sponsored multiple V100 GPUs for our project!
|
222 |
Thank you so much for your support!
|
223 |
<img width="250" alt="스크린샷" src="https://user-images.githubusercontent.com/77657524/126914984-d959be06-19f4-4228-8d3a-a855396b2c3f.jpeg">
|
224 |
|