HSIFoodIngr-64: A Dataset for Hyperspectral Food-Related Studies and a Benchmark Method on Food Ingredient Retrieval

Food-related issues have attracted increasing attention recently due to its various applications in our daily life, for example, restaurant service and dietary assessment, which are essential for human health management and understanding food characteristics. Many existing datasets are proposed for various food-relevant tasks, e.g., food detection and segmentation. However, most of the datasets are labelled with only dish-level annotations and lack detailed information for the corresponding ingredients. Hyperspectral imaging (HSI), which can explore the emissive characteristics of different objects with a long region of spectral bands, is of great potential for food and ingredient analysis. In the present work, a new food image dataset HSIFoodIngr-64 containing 3,389 pairs of HSI and RGB images with 21 dish classes and 64 ingredient categories was established. It is the first HSI-based food image dataset, of which all images are labelled with dish-level and ingredient-level annotations and corresponding pixel-wise ingredient masks. Therefore, our dataset can be applied to different food-centric tasks. Furthermore, this research proposed a benchmark method on the food ingredient retrieval task, which consists of two sub-networks called IPSN for ingredient prediction sub-network and ERSN for edge refinement sub-network. Extensive experiments were conducted on HSIFoodIngr-64 dataset and the effectiveness of our proposed method was verified. HSIFoodIngr-64 dataset is expected to provide a new perspective on food analysis and inspire more efforts on various food-related issues.

Automated summary

Food-related issues have gained attention due to their wide applications in daily life, such as restaurant service and dietary assessment, which are crucial for human health management and food understanding. Hyperspectral imaging (HSI) can explore the emissive characteristics of different objects and has significant potential for food and ingredient analysis. In this study, a new dataset called HSIFoodIngr-64 was established, containing 3,389 pairs of HSI and RGB images with both dish-level and ingredient-level annotations. A benchmark method for food ingredient retrieval was proposed and demonstrated the effectiveness on the HSIFoodIngr-64 dataset. This dataset provides a new perspective on food analysis and encourages further research on food-related issues.