Big data integration: an evolutionary perspective

Simona Dinu

doi:10.1117/12.2570740

31 December 2020 Big data integration: an evolutionary perspective

Simona Dinu

Proceedings Volume 11718, Advanced Topics in Optoelectronics, Microelectronics and Nanotechnologies X; 117180L (2020) https://doi.org/10.1117/12.2570740
Event: Advanced Topics in Optoelectronics, Microelectronics and Nanotechnologies 2020, 2020, Online Only

Abstract

This study aims at providing an efficient method to solve the statistical file merging issue: merging two or more files containing distinct datasets, from different sources, where the items of the datasets do not overlap (the percentage of units that are common to all datasets is insignificant). The problem is modeled as a network transportation problem and is solved using an adaptive Genetic Algorithm based on fuzzy logic controller, which dynamically calibrates algorithm parameters. This evolutionary technique is convenient for large-scale optimization problems with a significant number of variables and logical constraints. It also provides researchers in the fields of statistics and optimization a valuable instrument for achieving a correct balance between the quality of solutions and the processing time. This is a critical demand in most of the practical optimization problems. Numerical experiments on selected test instances validate this generalized merging approach and proves that the results of this file merging simulation can be extrapolated for better integration of various datasets generated by numerous government, administrative and statistical agencies. Apart from better central financial reporting for example, this will help to expand cooperation between private sector and government institutions.

Citation Download Citation

Simona Dinu "Big data integration: an evolutionary perspective", Proc. SPIE 11718, Advanced Topics in Optoelectronics, Microelectronics and Nanotechnologies X, 117180L (31 December 2020); https://doi.org/10.1117/12.2570740

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
10 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Fuzzy logic

Data integration

Genetic algorithms

Data fusion

Distance measurement

Data modeling

Statistical analysis

Keywords/Phrases

Search In:

Publication Years