Rethinking ROUGE Scores for Video Game Review Summarization

Abstract

Recall-oriented understudy for gisting evaluation (ROUGE) is a prevalent evaluation measure in the field of natural language processing, especially for text summarization. However, ROUGE's reliability has been continuously debated because a high ROUGE score does not guarantee a high-quality summary and vice versa. As an empirical study in the video game review summary, we address that existing state-of-the-art summarization techniques fail in generating high-quality game review summaries, and ROUGE scores for those summaries are quite problematic. To this end, game review data are newly collected, and BERT based automatic review summarizations are performed on the dataset to reconsider ROUGE use in video game review summarizations. Especially, we provide an in-depth discussion between the ROUGE score and the scores of human annotators in terms of game review factors.