FoodVision: Multi-Food Recognition with Calorie Insights

Project Domain / Category

Machine Learning/Image Processing

Abstract / Introduction

Food recognition is becoming very important in areas such as digital health, dietary monitoring, and food delivery apps. Existing food classification systems can identify food items from images, but they usually stop there — they don’t tell you about portion sizes or calorie counts.

This project aims to build a deep learning–based system that not only detects and classifies multiple food items in a single image but also estimates portion sizes and calculates approximate calorie values.

The system will use object detection models (such as YOLOv8) to recognize food items like roti, dal, rice, salad, eggs, fruits, and drinks. A post-processing step will then map each detected food item to a calorie database. Portion size will be estimated using bounding box area (or segmentation masks), and calories will be calculated accordingly.

By combining food detection, portion size estimation, and calorie calculation, this project creates a complete end-to-end application for diet tracking and nutrition analysis.

Objectives

The main objectives of the project are:

        Build and preprocess a multi-food image dataset (minimum 400 images) with bounding box annotations.

        Train and fine-tune an object detection model (YOLOv8n or YOLOv8s) for detecting multiple food items.

        Design a calorie-mapping module that links detected food items to standard nutrition values.

        Estimate portion sizes from bounding box area (or segmentation masks) and calculate calories accordingly.

        Develop a web-based application where users can upload an image and view:

        Detected food items with bounding boxes and labels.

        A side panel showing calorie and nutrient breakdown (per item + total).

        Document the complete process and evaluate system performance with metrics such as mAP, precision, recall, and calorie estimation accuracy.

Note: Dear Students, kindly read the Prerequisites and Methodology carefully and evaluate your skills before selecting this project. If you do not have the required knowledge, or if you are not willing to learn it wholeheartedly, you will not be able to complete this project successfully. Therefore, be very careful and thoughtful while choosing this project.

Prerequisites (what you must know / install) Students should have a basic understanding of:

        Python Programming: Variables, loops, functions, file handling, and working with libraries like os, numpy, pandas, matplotlib.

        Image Processing Basics: Reading, resizing, and displaying images (OpenCV, PIL), pixels, and image formats.

Page 56 of 167

        Machine Learning & Deep Learning

         Basics of supervised learning (train/test split, accuracy, evaluation).

         CNN (Convolutional Neural Networks) – how images are classified.

         Object detection concepts – bounding boxes, IoU, precision/recall.

         Awareness of YOLO, Faster R-CNN, Mask R-CNN (at least conceptually).

        Dataset Preparation & Annotation

         How to annotate bounding boxes using tools (LabelImg, Roboflow).

         How datasets are structured for object detection (images + labels in .txt or .xml).

        Nutrition & Calorie Mapping (Basic)

         General idea of how food calorie values are measured.

         Awareness that portion size affects calories.

         Example: 1 roti ≈ 70 kcal, 1 cup rice ≈ 200 kcal, 1 boiled egg ≈ 75 kcal.

Methodology

        Learn the Basics of Python and Machine Learning

Before starting the project, you must learn Python programming, deep learning basics, and object detection models.

You should cover the following topics:

         Python basics: variables, loops, functions, file handling.

         Python libraries: NumPy, Pandas, Matplotlib.

         Deep Learning basics: CNN (Convolutional Neural Networks).

         Object Detection concepts: bounding boxes, mAP, YOLO.

If you are unable to understand these concepts do not choose this project.

Suggested Resources:

            https://docs.python.org/3/tutorial/

            https://www.coursera.org/specializations/python

            https://developers.google.com/machine-learning/crash-course

            https://www.coursera.org/specializations/deep-learning

            https://docs.ultralytics.com/

            https://blog.roboflow.com/annotating-images/

        Setup your development environment:

To implement the project, students must set up a proper development environment:

            Download and install the python

            Install a development environment

            Install the necessary libraries that helps you with image processing, data handling, and model training.

        Data Collection:

            Collect a diverse collection of multi-food images featuring different meal composition. You can collect manually using mobile cameras or use the free dataset (Food-101 Dataset, UECFOOD100 Dataset, Vireo Food-172 Dataset) and modify accordingly.

Page 57 of 167

Following are the requirements of Dataset:

            10 classes (Roti/Chapati, Egg, Drink, Fruits, Vegetables, Salad, Pizza Slice, Pulao, Bread Slice).

            40 images per class → 400 total images.

            Each image must contain 2 different food items.

            Label images with bounding boxes or segmentation masks for each food item using tools like

LabelImg or Roboflow.

            Ensure that the dataset is structured properly with annotations and labels for object detection. Suggested Resources:

1. https://blog.roboflow.com/annotating-images/

        Load and visualize the data:

Before working on the model, students need to understand the dataset by loading the images and plotting samples from different categories. Visualization tools like Matplotlib and OpenCV can be used for this purpose.

        Data Pre-processing:

In this step, you have to clean and prepare the data for analysis. Resize all images to a standard size (e.g., 416x416 pixels for YOLO). Normalize pixel values to a range between 0 and 1 for efficient processing.

Apply data augmentation (rotation, contrast adjustments, brightness variations) to improve model robustness.

Suggested Resources: https://docs.opencv.org/master/d6/d00/tutorial_py_root.html https://docs.ultralytics.com/datasets/

        Split the Data into Training and Testing Sets

Students need to divide the dataset into training and testing sets: Training Set: 80% of the data, used for model training.

Testing Set: 20% of the data, used to evaluate the model’s performance on unseen images.

Suggested Resources: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

        Build a Model:

Students need to select an appropriate object detection model YOLOv8n (Nano) or YOLOv8s (Small) for food classification.

Suggested Resources: https://docs.ultralytics.com/modes/train/?utm_source=chatgpt.com https://www.youtube.com/watch?v=Grir6TZbc1M

        Model Training:

Train the object detection model on the prepared dataset.

Fine-tune with annotated dataset to improve accuracy and generalization.

Page 58 of 167