Severity of usability problems and system usability scale (Sus) scores on augmented reality (ar) user interfaces

Abstract

Severity scales are commonly employed to determine the level of priority of usability problems (UPs) found during a usability inspection, so they can be even-tually resolved in the redesign process. Different types of qualitative and quantitative data, besides a list of UPs and severity ratings, can be obtained from usability studies by employing numerous tools. The system usability score (SUS) is one of the most used questionnaires to measure perceived usability of systems and services. Finding relation-ships between different types of data obtained from a usability study is important because it can help practitioners and developers to have a deeper insight into the system and evaluation procedures. In this study, we examined if the number of UPs reported through usability inspection can predict SUS scores. A total of 24 participants took part in the experiment, where 4 augmented reality (AR) user interfaces (UIs) were inspected and SUS answers were obtained at the end. After the experiment, severity ratings related to the UPs reported were collected from 4 experts. Ultimately, 272 UPs were divided into 3 severity levels: low, moderate, and high. Based on the number of UPs reported by an evaluator related to each severity level, multiple linear regression analysis was performed. Results showed that only the number of high-severity UPs could predict SUS scores. Fur-thermore, the effects of previous experience with AR UIs and gender (combined with the number of high-severity UPs) on SUS scores were investigated. Results allowed a deeper view into the AR UIs, their issues, and how evaluator characteristics could be related to the usability evaluation. © ICIC International 2021.