Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-Shot In-Context Learners
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Cho, Hyunsoo | - |
dc.contributor.author | Kim, Hyuhng Joon | - |
dc.contributor.author | Kim, Junyeob | - |
dc.contributor.author | Lee, Sang-Woo | - |
dc.contributor.author | Lee, Sang-goo | - |
dc.contributor.author | Yoo, Kang Min | - |
dc.contributor.author | Kim, Tae Uk | - |
dc.date.accessioned | 2023-09-11T01:53:54Z | - |
dc.date.available | 2023-09-11T01:53:54Z | - |
dc.date.created | 2023-07-20 | - |
dc.date.issued | 2023-06 | - |
dc.identifier.issn | 2159-5399 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/190393 | - |
dc.description.abstract | Through in-context learning (ICL), large-scale language models are effective few-shot learners without additional model fine-tuning. However, the ICL performance does not scale well with the number of available training samples as it is limited by the inherent input length constraint of the underlying language model. Meanwhile, many studies have revealed that language models are also powerful feature extractors, allowing them to be utilized in a black-box manner and enabling the linear probing paradigm, where lightweight discriminators are trained on top of the pre-extracted input representations. This paper proposes prompt-augmented linear probing (PALP), a hybrid of linear probing and ICL, which leverages the best of both worlds. PALP inherits the scalability of linear probing and the capability of enforcing language models to derive more meaningful representations via tailoring input into a more conceivable form. Throughout in-depth investigations on various datasets, we verified that PALP significantly enhances the input representations closing the gap between ICL in the data-hungry scenario and fine-tuning in the data-abundant scenario with little training overhead, potentially making PALP a strong alternative in a black-box scenario. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | Association for the Advancement of Artificial Intelligence | - |
dc.title | Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-Shot In-Context Learners | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Kim, Tae Uk | - |
dc.identifier.doi | 10.1609/aaai.v37i11.26495 | - |
dc.identifier.scopusid | 2-s2.0-85164138550 | - |
dc.identifier.bibliographicCitation | AAAI Conference on Artificial Intelligence, v.37, no.11, pp.12709 - 12716 | - |
dc.relation.isPartOf | AAAI Conference on Artificial Intelligence | - |
dc.citation.title | AAAI Conference on Artificial Intelligence | - |
dc.citation.volume | 37 | - |
dc.citation.number | 11 | - |
dc.citation.startPage | 12709 | - |
dc.citation.endPage | 12716 | - |
dc.type.rims | ART | - |
dc.type.docType | Proceeding | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | scopus | - |
dc.subject.keywordPlus | Black boxes | - |
dc.subject.keywordPlus | Context learning | - |
dc.subject.keywordPlus | Fine tuning | - |
dc.subject.keywordPlus | In contexts | - |
dc.subject.keywordPlus | Language model | - |
dc.subject.keywordPlus | Large-scales | - |
dc.subject.keywordPlus | Learning performance | - |
dc.subject.keywordPlus | Linear probing | - |
dc.subject.keywordPlus | Scalings | - |
dc.subject.keywordPlus | Training sample | - |
dc.subject.keywordAuthor | SNLP | - |
dc.subject.keywordAuthor | Language Models, SNLP | - |
dc.subject.keywordAuthor | Text Classification | - |
dc.identifier.url | https://ojs.aaai.org/index.php/AAAI/article/view/26495 | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.