Detection Method for Distributed Web-Crawlers: A Long-Tail Threshold Model
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Ro, Inwoo | - |
dc.contributor.author | Han, Joong Soo | - |
dc.contributor.author | Im, Eul Gyu | - |
dc.date.accessioned | 2022-07-11T05:21:33Z | - |
dc.date.available | 2022-07-11T05:21:33Z | - |
dc.date.created | 2021-05-12 | - |
dc.date.issued | 2018-10 | - |
dc.identifier.issn | 1939-0114 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/149220 | - |
dc.description.abstract | This paper proposes an advanced countermeasure against distributed web-crawlers. We investigated other methods for crawler detection and analyzed how distributed crawlers can bypass these methods. Our method can detect distributed crawlers by focusing on the property that web traffic follows the power distribution. When we sort web pages by the number of requests, most of requests are concentrated on the most frequently requested web pages. In addition, there will be some web pages that normal users do not generally request. But crawlers will request for these web pages because their algorithms are intended to request iteratively by parsing web pages to collect every item the crawlers encounter. Therefore, we can assume that if some IP addresses are frequently used to request the web pages that are located in the long-tail area of a power distribution graph, those IP addresses can be classified as crawler nodes. The experimental results with NASA web traffic data showed that our method was effective in identifying distributed crawlers with 0.0275% false positives when a conventional frequency-based detection method shows 2.882% false positives with an equal access threshold. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | WILEY-HINDAWI | - |
dc.title | Detection Method for Distributed Web-Crawlers: A Long-Tail Threshold Model | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Im, Eul Gyu | - |
dc.identifier.doi | 10.1155/2018/9065424 | - |
dc.identifier.scopusid | 2-s2.0-85058873014 | - |
dc.identifier.wosid | 000453811300001 | - |
dc.identifier.bibliographicCitation | SECURITY AND COMMUNICATION NETWORKS, v.2018 | - |
dc.relation.isPartOf | SECURITY AND COMMUNICATION NETWORKS | - |
dc.citation.title | SECURITY AND COMMUNICATION NETWORKS | - |
dc.citation.volume | 2018 | - |
dc.type.rims | ART | - |
dc.type.docType | Article | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | Y | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Telecommunications | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Telecommunications | - |
dc.identifier.url | https://www.hindawi.com/journals/scn/2018/9065424/ | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.