Recently, many studies and commercial applications have been introduced for automated food identification from visual data to quantitatively estimate calories and macronutrients. However, a reliable diet recommendation system (DRS) must be able to infer not only types of food but also more fine-grained information, such as ingredients, to adjust its dietary analysis on micronutrients to be expected from consumption based on the actual contents that have been added.

Existing models that were designed to predict recipes and ingredients from food were only trained on images and recipes sourced primarily from internet resources on cooking, so their predictions may easily fail in realistic scenarios, in which foods are highly customizable with numerous combinations of ingredients. The goal of this project is to extend successful Year 1 and 2 efforts to construct an end-to-end explainable DRS as a useful tool for healthy food consumption.