Enhancing Data Ingestion Efficiency in Cloud-Based Systems: A Design Pattern Approach

This paper aims to defne design patterns specifcally for data ingestion techniques within cloud-based architectures, addressing the challenges associated with high-volume data processing. The approach utilizes a fexible, metadata-driven framework that enhances adaptability and ease of use. This framework supports both incremental and full refresh methods, allowing for seamless changes to ingestion types, schema updates, table additions, and the incorporation of new data sources with minimal intervention from data engineers. The proposed design patterns were validated through experiments conducted on the Azure and Google Cloud platforms. The experiments demonstrate that the proposed design patterns signifcantly reduce data ingestion time, showcasing their efectiveness in managing high-volume data ingestion. This paper contributes to the feld of data management by presenting a comprehensive defnition of design patterns tailored for data ingestion in cloudbased architectures, efectively addressing key challenges in high-volume data processing.