DataCon: Lessons Learned Enabling Easier Data Sharing, Exploration and Fusion Building a DataCon AutoGenerator Module

Abstract

Data is transforming the world. Individuals, organizations, companies and governments are rushing to build technologies that generate, manage and analyze ever-increasing amounts of data. However, sharing, exploring and fusing datasets remain difficult and painful processes. We previously proposed a "DataCon" system that supports easier data sharing, exploration and fusion of many types of datasets and announced a 3 years, 1 million USD project funded by the Korean government to develop a DataCon-based data sharing platform. We now describe the lessons learned during our first phase of development: a proof of concept DataCon AutoGenerator Module which takes in arbitrary datasets and automatically generates corresponding DataCon objects. Specifically, the study describes several potential use cases for a DataCon-based data sharing platform, explores how several popular data repositories organize their datasets, sketches a preliminary data taxonomy to organize the DataCon repository, maps out a tentative technological development roadmap, recounts lessons learned implementing the initial proof of concept and lists several potential avenues for future research. We will use this study as a blueprint for future development and hope it also informs the work of others who want to make working with data easier, accelerating our collective ability to transform the world with data.