  Accurate dietary intake estimation is critical for informing policies and
programs to support healthy eating, as malnutrition has been directly linked to
decreased quality of life. However self-reporting methods such as food diaries
suffer from substantial bias. Other conventional dietary assessment techniques
and emerging alternative approaches such as mobile applications incur high time
costs and may necessitate trained personnel. Recent work has focused on using
computer vision and machine learning to automatically estimate dietary intake
from food images, but the lack of comprehensive datasets with diverse
viewpoints, modalities and food annotations hinders the accuracy and realism of
such methods. To address this limitation, we introduce NutritionVerse-Synth,
the first large-scale dataset of 84,984 photorealistic synthetic 2D food images
with associated dietary information and multimodal annotations (including depth
images, instance masks, and semantic masks). Additionally, we collect a real
image dataset, NutritionVerse-Real, containing 889 images of 251 dishes to
evaluate realism. Leveraging these novel datasets, we develop and benchmark
NutritionVerse, an empirical study of various dietary intake estimation
approaches, including indirect segmentation-based and direct prediction
networks. We further fine-tune models pretrained on synthetic data with real
images to provide insights into the fusion of synthetic and real data. Finally,
we release both datasets (NutritionVerse-Synth, NutritionVerse-Real) on
https://www.kaggle.com/nutritionverse/datasets as part of an open initiative to
accelerate machine learning for dietary sensing.

Computer Vision and Pattern Recognition;Artificial Intelligence