Revolutionizing Culinary Experiences with AI: Introducing FIRE (Food Image to REcipe generation) 🔥 | by Prateek Chhikara | Jan, 2024

1. Recipe Customization

Recipe customization is important on account of connection between meals, customs, and specific particular person preferences. Furthermore, it turns into essential when addressing allergy signs or dietary restrictions. Surprisingly, whatever the evident demand, current literature lacks devoted efforts in recipe customization. Our work objectives to bridge the evaluation gap by enabling custom-made recipe customization, considering specific particular person model profiles and dietary restrictions.

To data future evaluation on this house, we showcase the facility of FIRE to assist a recipe customization technique that focuses on a wide range of issues (e.g., ingredient various, model adjustment, calorie adjustment, cooking time adaptation) to test few-shot effectivity completely. As confirmed throughout the purple part of Decide 5, we take away components to trim the potatoes from the recipe. Two sentences related to potatoes are deleted throughout the modified mannequin, and one sentence is modified to verify consistency. Notably, we stock out ingredient addition to change ‘cheese’ with ‘cheddar cheese’ and acknowledge that it must be added sooner than baking, ensuing throughout the modified sentence ‘Sprinkle half each of cheddar cheese and onions.’

2. Producing Machine Code for Image-based Recipes

Altering recipes to machine code permits automation, scalability, and integration with diversified current strategies, thus lowering handbook intervention, saving labor costs, and lowering human errors whereas getting ready the meals. To facilitate this exercise, we combine FIRE’s recipe period vitality with the facility of huge LMs to regulate code-style prompts for structural duties [14]. We current an occasion technique for producing Python-style code representations of recipes developed by FIRE, by prompting GPT-3 (please seek the advice of with orange half in Decide 5).

We launched FIRE, a method tailored for meals computing, specializing in producing meals titles, extracting components, and producing cooking instructions solely from image inputs. We leveraged newest CV and language modeling developments to understand superior effectivity in the direction of sturdy baselines. Furthermore, we demonstrated wise functions of FIRE for recipe customization and recipe-to-code period, showcasing the adaptability and automation potential of our technique.

We report three challenges that must be addressed in future evaluation:

1. Present and proposed recipe period fashions lack a reliable mechanism to substantiate the accuracy of the generated recipes. Typical evaluation metrics fall fast on this aspect. Subsequently, we want to create a model new metric that assesses the coherence and plausibility of recipes, providing a additional thorough evaluation.
2. The variability and availability of recipes are influenced by geographical, climatic, and religious components, which might limit their applicability. Incorporating knowledge graphs that account for these contextual components and ingredient relationships can present totally different ingredient options, addressing this problem.
3. Hallucination in recipe period using language and imaginative and prescient fashions poses a serious drawback. Future work would uncover the state-tracking methods to boost the period course of, ensuring the manufacturing of additional wise and proper recipes.

I hope this overview has supplied you the notion into the inspiration and progress of FIRE, our revolutionary instrument for altering meals photographs into detailed recipes. For a additional in-depth exploration of our technique, I invite you to check out our full paper, which is revealed throughout the IEEE/CVF Winter Conference on Capabilities of Computer Imaginative and prescient (WACV) — 2024. If our evaluation contribute to your work, we is perhaps glad in case you cite it. 😊

Paper Hyperlink: https://openaccess.thecvf.com/content/WACV2024/html/Chhikara_FIRE_Food_Image_to_REcipe_Generation_WACV_2024_paper.html

@InProceedings{Chhikara_2024_WACV,
creator    = {Chhikara, Prateek and Chaurasia, Dhiraj and Jiang, Yifan and Masur, Omkar and Ilievski, Filip},
title     = {FIRE: Meals Image to REcipe Period},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Capabilities of Computer Imaginative and prescient (WACV)},
month     = {January},
yr      = {2024},
pages     = {8184-8194}
}

[1] Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. A survey on meals computing. ACM Comput. Surv., 52(5), sep 2019.

[2] Sutter Nicely being. Consuming Correctly for Psychological Nicely being. https://www.sutterhealth.org/health/nutrition/eating-wellfor-mental-health. Accessed on March 24, 2023.

[3] Kiely Kuligowski. 12 Causes to Use Instagram for Your Enterprise. https://www.business.com/articles/10-reasons-touse-instagram-for-business/. Accessed on May 12, 2023.

[4] Dim P. Papadopoulos, Enrique Mora, Nadiia Chepurko, Kuan Wei Huang, Ferda Ofli, and Antonio Torralba. Learning program representations for meals photographs and cooking recipes, 2022.

[5] Sundaram Gunasekaran. Computer imaginative and prescient experience for meals prime quality assurance. Tendencies in Meals Science & Know-how, 7(8):245–256, 1996.

[6] Yoshiyuki Kawano and Keiji Yanai. Meals image recognition with deep convolutional choices. In Proceedings of the 2014 ACM Worldwide Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, pages 589– 593, 2014

[7] Amaia Salvador, Michal Drozdzal, Xavier Giro-i Nieto, and ´ Adriana Romero. Inverse cooking: Recipe period from meals photographs. In Proceedings of the IEEE/CVF Conference on Computer Imaginative and prescient and Pattern Recognition, pages 10453– 10462, 2019.

[8] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and period. In Worldwide Conference on Machine Learning, pages 12888– 12900. PMLR, 2022.

[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16×16 phrases: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.

[10] Prateek Chhikara, Ujjwal Pasupulety, John Marshall, Dhiraj Chaurasia, and Shweta Kumari. Privateness aware questionanswering system for on-line psychological nicely being hazard analysis. In The twenty second Workshop on Biomedical Pure Language Processing and BioNLP Shared Duties, pages 215– 222, Toronto, Canada, July 2023. Affiliation for Computational Linguistics.

[11] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the bounds of change finding out with a unified text-to-text transformer. The Journal of Machine Learning Evaluation, 21(1):5485–5551, 2020.

[12] Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzman, Luke Zettlemoyer, and Marjan ´ Ghazvininejad. Detecting hallucinated content material materials in conditional neural sequence period. In Findings of the Affiliation for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, 2021.

[13] Mehrdad Farahani and Kartik Godawat and Haswanth Aekula and Deepak Pandian and Nicholas Broad. Chef Transformer. https://huggingface.co/flax-community/t5- recipe-generation. Accessed on April 12, 2023.

[14] Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language fashions of code are few-shot commonsense learners. In Findings of the Affiliation for Computational Linguistics: EMNLP 2022, 2022