Contrasting Multi-Modal Similarity Framework for Video Scene Segmentation
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Park, Jinwoo | - |
dc.contributor.author | Kim, Jungeun | - |
dc.contributor.author | Seok, Jaegwang | - |
dc.contributor.author | Lee, Sukhyun | - |
dc.contributor.author | Kim, Junyeong | - |
dc.date.accessioned | 2024-03-27T00:30:24Z | - |
dc.date.available | 2024-03-27T00:30:24Z | - |
dc.date.issued | 2024-02 | - |
dc.identifier.issn | 2169-3536 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/73035 | - |
dc.description.abstract | This paper proposes a video scene segmentation framework referred to as a Contrasting Multi-Modal Similarity (CMS). Video is composed of multiple scenes which are short stories or semantic units of video, with each scene consisting of multiple shots. The task of video scene segmentation aims to semantically segment long videos, such as movies, into the sequence of scenes by identifying the boundaries of each scene transition. Current video scene segmentation frameworks have primarily relied on comparing only the visual cues of adjacent shots to identify scene boundaries. These frameworks have focused on two major approaches: 1) comparing only the visual cues of adjacent frames to distinguish between scenes and 2) performing clustering based on visual cues for distinction among scenes. However, within videos, there exist numerous scenes that are difficult to distinguish using visual information alone, as they often appear similar or ambiguous. Taking inspiration from the aforementioned issues, we propose a framework referred to as CMS that leverages not only visual cues (i.e., shots) but also textual cues (i.e., captions) to semantically distinguish scenes. The new framework, CMS, leverages visual cues and text cues as follows: 1) Generate captions corresponding to each shot using a zero-shot captioning model (Caption Generation). 2) Construct similarity score matrices for each modality to measure semantic similarities (Similarity Score Calculation). 3) Based on the above matrix, select similar shots and dissimilar shots for contrastive training (Similarity Score-based Sampling). Our experiments show that the CMS framework advances the performance to exceed the previous state-of-the-art methods with a relatively simple approach without complex model architectures. © 2013 IEEE. | - |
dc.format.extent | 12 | - |
dc.language | 영어 | - |
dc.language.iso | ENG | - |
dc.publisher | Institute of Electrical and Electronics Engineers Inc. | - |
dc.title | Contrasting Multi-Modal Similarity Framework for Video Scene Segmentation | - |
dc.type | Article | - |
dc.identifier.doi | 10.1109/ACCESS.2024.3370676 | - |
dc.identifier.bibliographicCitation | IEEE Access, v.12, pp 32408 - 32419 | - |
dc.description.isOpenAccess | Y | - |
dc.identifier.wosid | 001177917000001 | - |
dc.identifier.scopusid | 2-s2.0-85186972253 | - |
dc.citation.endPage | 32419 | - |
dc.citation.startPage | 32408 | - |
dc.citation.title | IEEE Access | - |
dc.citation.volume | 12 | - |
dc.type.docType | Article | - |
dc.publisher.location | 미국 | - |
dc.subject.keywordAuthor | contrastive learning | - |
dc.subject.keywordAuthor | multi-modal reasoning | - |
dc.subject.keywordAuthor | Visual scene segmentation | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Telecommunications | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.relation.journalWebOfScienceCategory | Telecommunications | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
84, Heukseok-ro, Dongjak-gu, Seoul, Republic of Korea (06974)02-820-6194
COPYRIGHT 2019 Chung-Ang University All Rights Reserved.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.