Vision-Based Cooking Assistance System for Visually Impaired People

Abstract

Cooking simple meals is an essential skill for independent living. Our user research showed that visually impaired people felt strong anxiety and danger about cooking, especially when they used induction cooktops due to fear of burns from fire, difficulty in locating high temperature objects, and absence of cookware adapted for them. This study proposes a vision sensing system to assist visually impaired people so that they can use an induction cooktop safely and easily. Our system is designed to first detect the induction cooking zone, cookware, and the cook’s hands using a vision-based deep-learning algorithm of Python with a YOLO v5 model, and then to give real-time feedback on the relative position of cookware to users. The algorithm is composed of (1) a hob position notification that guides the direction to move pots or pans to the accurate position on the hob, and (2) a hand position notification that guides the relative position of the users’ hands. The feedback is given to the user using sound effects and voice guidance designed based on user preference. Our usability assessment indicates that the proposed system increased the accessibility to stove-like appliances for visually impaired people and encourages their confidence to try cooking. Our final prototype includes light feedback along with audible guides to cover various types of visual impairments. We expect that this system will encourage visually impaired people to get a clearer sense of location information around hobs, and have higher confidence in cooking on their own. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022.