benchmark/README.md

:bar_chart: Benchmark

We provide in this release (benchmark.zip) with the following 17 entries as a benchmark to evaluate NVS models. We hope this will help standardize the evaluation of NVS models and facilitate fair comparison between different methods.

Dataset	Split	Path	Content	Image Preprocessing	Image Postprocessing
OmniObject3D	S (SV3D), O (Ours)	omniobject3d	train_test_split_*.json	center crop to 576	\
GSO	S (SV3D), O (Ours)	gso	train_test_split_*.json	center crop to 576	\
RealEstate10K	D (4DiM)	re10k-4dim	train_test_split_*.json	center crop to 576	resize to 256
	R (ReconFusion)	re10k	train_test_split_*.json	center crop to 576	\
	P (pixelSplat)	re10k-pixelsplat	train_test_split_*.json	center crop to 576	resize to 256
	V (ViewCrafter)	re10k-viewcrafter	images/*.png,transforms.json,train_test_split_*.json	resize the shortest side to 576 (--L_short 576)	center crop
LLFF	R (ReconFusion)	llff	train_test_split_*.json	center crop to 576	\
DTU	R (ReconFusion)	dtu	train_test_split_*.json	center crop to 576	\
CO3D	R (ReconFusion)	co3d	train_test_split_*.json	center crop to 576	\
	V (ViewCrafter)	co3d-viewcrafter	images/*.png,transforms.json,train_test_split_*.json	resize the shortest side to 576 (--L_short 576)	center crop
WildRGB-D	Oₑ (Ours, easy)	wildgbd/easy	train_test_split_*.json	center crop to 576	\
	Oₕ (Ours, hard)	wildgbd/hard	train_test_split_*.json	center crop to 576	\
Mip-NeRF360	R (ReconFusion)	mipnerf360	train_test_split_*.json	center crop to 576	\
DL3DV-140	O (Ours)	dl3dv10	train_test_split_*.json	center crop to 576	\
	L (Long-LRM)	dl3dv140	train_test_split_*.json	center crop to 576	\
Tanks and Temples	V (ViewCrafter)	tnt-viewcrafter	images/*.png,transforms.json,train_test_split_*.json	resize the shortest side to 576 (--L_short 576)	center crop
	L (Long-LRM)	tnt-longlrm	train_test_split_*.json	center crop to 576	\

• For entries without images/*.png and transforms.json, we use the images from the original dataset after converting them into the reconfusion format, which is then parsable by ReconfusionParser (seva/data_io.py). Please note that during this conversion, you should sort the images by sorted(image_paths), which is then directly indexable by our train/test ids. We provide in benchmark/export_reconfusion_example.py an example script converting an existing academic dataset into the the scene folders.
• For evaluation and benchmarking, we first conduct operations in the Image Preprocessing column to the model input and then operations in the Image Postprocessing column to the model output. The final processed samples are used for metric computation.

Acknowledgment

We would like to thank Wangbo Yu, Aleksander Hołyński, Saurabh Saxena, and Ziwen Chen for their kind clarification on experiment settings.