Identifying UI widgets of mobile applications from sketch images

Abstract

In this study, we present a statistical method for identifying UI widgets of mobile applications from sketch images using geometric and text analysis features. We classify the widgets into three components according to their characteristics: basic component, composition component and container component. We investigate the geometric feature: the relationship among graphic elements of widgets and the text analysis feature: the parts of speech of the widget text. We incorporate the geometric features of widgets in the rules through inclusion, combination and repetition relations in a statistical manner. The proposed method comprises the extraction of graphic elements such as text or shapes from the input sketch image using the Optical Character Recognition (OCR) technique and edge detection. The widgets are then identified by comparing them using the predefined rules. The method also provides meaningful features from the text of widgets. This can be useful in feedback about its recognitions to change the inference results. © Medwell Journals, 2018.