A schema generator for collected data from wearable devices for reliable data ingestion

Abstract

We are living in an era, where we have smart devices all around us, like smart phones, smart watches, and other devices. Data produced by these devices has a great importance, used either for providing services or for analyzing purpose. These devices produce data in large amount, with many different formats, which makes data ingestion a very challenging task, as this massive amount of data causes a tremendous problem in both structuring and storage of data. Furthermore, due to the different formats and unpredictable nature of devices, data validation is also required. Although, there are many different data ingestion tools available which serves the purpose of structuring the data, but validation of data is still a big challenge. Therefore, we proposed a Schema Generator which aims to auto generate a schema of the input file for a new device added to the system, and validates this schema against the input file using available ingestion tool. Additionally, the idea is to provide a user interface for creating a configuration file for ingestion tool, as creating it manually is a very strenuous process for new device data. © 2019 ACM.