Metadata-Driven ETL Pipelines: A Framework for Scalable Data Integration Architecture
DOI:
https://doi.org/10.32628/CSEIT241061224Keywords:
Metadata-Driven Architecture, ETL Pipeline Design, Data Integration Systems, Pipeline Orchestration, Data Governance FrameworkAbstract
This article comprehensively analyzes metadata-driven data pipelines in Extract, Transform, and Load (ETL) processes, examining their architectural patterns, implementation strategies, and business impact. The article explores how metadata-driven approaches enhance pipeline flexibility, maintainability, and scalability compared to traditional ETL implementations. The article investigates the theoretical foundations of metadata-driven architectures and presents a framework for implementing reusable pipeline components through metadata templates. The article evaluates performance characteristics and resource utilization patterns across different implementation scenarios, providing insights into optimization strategies. Additionally, the article examines the integration of business rules and governance models within metadata-driven pipelines, demonstrating how this approach facilitates consistent data quality management and regulatory compliance. The findings suggest that metadata-driven pipelines significantly reduce development overhead, improve maintenance efficiency, and enhance the adaptability of ETL processes in dynamic business environments. This article contributes to the growing knowledge in data integration architecture and provides practical guidelines for organizations seeking to modernize their data pipeline infrastructure.
📊 Article Downloads
References
A. Munappy, J. Bosch, and H. Holmström Olsson, "Data Pipeline Management in Practice: Challenges and Opportunities," Lecture Notes in Computer Science, vol. 12562, pp. 168-184, 2020. DOI: 10.1007/978-3-030-64148-1_11 Link: https://research.chalmers.se/publication/523476/file/523476_Fulltext.pdf DOI: https://doi.org/10.1007/978-3-030-64148-1_11
A. Munappy, J. Bosch, and H. Holmström Olsson, "Modelling Data Pipelines," in Proceedings - 46th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2020, pp. 13-20. DOI: 10.1109/SEAA51224.2020.00014 Link: https://research.chalmers.se/publication/521248/file/521248_Fulltext.pdf DOI: https://doi.org/10.1109/SEAA51224.2020.00014
A. Ismail, M. S. Joy, J. E. Sinclair, and M. I. Hamzah, "A Metametadata Taxonomy to Support Semantic Searching Algorithms in Metadata Repository," in Proceedings - 2009 International Conference on Electrical Engineering and Informatics, 2009. DOI: 10.1109/ICEEI.2009.5254702 Link: https://ieeexplore.ieee.org/document/5254702 DOI: https://doi.org/10.1109/ICEEI.2009.5254702
M. Bushong, "Metadata Driven Pipelines for Dynamic Full and Incremental Processing in Azure SQL," Microsoft Community Hub, 2023. Link: https://techcommunity.microsoft.com/blog/azuredatafactoryblog/metadata-driven-pipelines-for-dynamic-full-and-incremental-processing-in-azure-s/3925362
Databricks Community, "Metadata-Driven ETL Framework in Databricks (Part-1)," 2024. Link: https://community.databricks.com/t5/technical-blog/metadata-driven-etl-framework-in-databricks-part-1/ba-p/92666
M. Bisson, E. Phillips, and M. Fatica, "A CUDA implementation of the pagerank pipeline benchmark," in IEEE High Performance Extreme Computing Conference (HPEC), 2016. Link: https://ieeexplore.ieee.org/abstract/document/7761620 DOI: https://doi.org/10.1109/HPEC.2016.7761620
H. Chihoub and C. Collet, "A Scalability Comparison Study of Data Management Approaches for Smart Metering Systems," in 45th International Conference on Parallel Processing (ICPP), 2016. Link: https://ieeexplore.ieee.org/document/7573850 DOI: https://doi.org/10.1109/ICPP.2016.61
T. Ishihara, K. Hotta, Y. Higo, and S. Kusumoto, "Reusing Reused Code," in 20th Working Conference on Reverse Engineering (WCRE), 2013. Link: https://ieeexplore.ieee.org/document/6671322 DOI: https://doi.org/10.1109/WCRE.2013.6671322
N. Deepa, B. Prabadevi, L.B. Krithika, and B. Deepa, "An Analysis on Version Control Systems," in 2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE), 2020. Link: https://ieeexplore.ieee.org/abstract/document/9077781
J. Zhang, J. Yang, and J. Li, "When Rule Engine Meets Big Data: Design and Implementation of a Distributed Rule Engine Using Spark," in IEEE Third International Conference on Big Data Computing Service and Applications (BigDataService), 2017. Link: https://ieeexplore.ieee.org/abstract/document/7944919 DOI: https://doi.org/10.1109/BigDataService.2017.17
C. Cichy and S. Rass, "An Overview of Data Quality Frameworks," in IEEE Access, 2019. Link: https://ieeexplore.ieee.org/document/8642813 DOI: https://doi.org/10.1109/ACCESS.2019.2899751
Downloads
Published
Issue
Section
License
Copyright (c) 2024 International Journal of Scientific Research in Computer Science, Engineering and Information Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
