사전 학습된 언어 모델에서의 성별 편향 측정

Abstract

Purpose This study aims to analyze and quantify gender bias in pre-trained language models by examining their behavior and proposing new metrics for bias assessment. Methods In this study, we utilized the WinoBias benchmark dataset to measure gender stereotype and skew through tasks evaluating model outputs in pro- and anti-stereotypic scenarios. Unlike traditional methods that measure genter stereotype and skew using model outputs, this study proposes a new bias measurement approach based on the model's probability values. Results Unlike previous studies that showed a conflicting relationship between gender stereotype and skew, measurements using the proposed metric revealed a positive correlation between the two, diverging from the results of conventional metrics. Conclusion The study highlights the need to adopt and improve alternative metrics to better assess and address bias in language models.