The Relationship between Embedding and Deductive Reasoning Ability on BERT

Abstract

BERT is a pre-Trained language model that has been shown to be effective for a variety of tasks, including natural language inference (NLI) and textual entailment. However, it is not clear how well BERT can identify logical correctness or inconsistency. In this paper, we investigate the ability of BERT to distinguish between logically correct sentences and those with logical contradictions. We perform this by comparing the similarity scores of these two types of sentences using a variety of BERT models. Our results were conducted to understand the relationship between BERT embeddings and ability of logical reasoning. To this end, we identified that BERT is able to capture some aspects of logical relationships between words or sentences. However, our results also show that the logical reasoning ability of BERT varies depending on the complexity of the logic involved. Our findings contribute to the understanding of how language models can be used for deductive reasoning tasks. In addition, we are the first to introduce a new zero-shot evaluation dataset for deductive reasoning. We believe that our works could be used to improve the performance of language models for these tasks. © 2024 IEEE.