Next Task Size Prediction Method for FP-Growth Algorithm

Abstract

Frequent pattern (FP) mining is useful for deriving meaningful information from data. Among related methods, FP-growth is one of the most widely used algorithms but has time and physical restrictions when applied to big data mining. FP-growth repeatedly generates conditional FP-trees (CFP-trees) on memory and mines frequent patterns. As a result, memory shortage may occur when processing big data with FP-growth. If memory overflow occurs during FP-growth, mining will no longer be performed afterward. If memory overflow is predicted in advance, the error can be handled using methods such as switching to secondary memory-based mining method or waiting until the memory is released and subsequently proceeding with the mining. In this paper, the next task size prediction method is proposed to predict whether the main memory is sufficient to operate the next task during FP-growth process. Furthermore, it can also help in the parallel and distributed processing of FP-growth. The next task size prediction method can help solve the load balancing problem by allocating the remaining memory-sized task in each workstation or appropriately sized tasks to the workstations.