Optimizing Real-time Data Ingestion and Transformation Workflows in Cloud-native Environments Using Serverless Computing and Microservices-based Data Integration Patterns
Keywords:
Cloud-native, serverless computing, microservices, real-time data ingestion, data transformation, workflow optimization, event-driven architecture, data pipelinesAbstract
In modern cloud-native environments, optimizing real-time data ingestion and transformation is critical for achieving scalability, flexibility, and efficiency. This paper explores serverless computing and microservices-based data integration patterns to improve real-time data workflows. By leveraging serverless architectures, organizations can reduce operational overhead and dynamically allocate resources. Additionally, microservices-based patterns facilitate seamless integration, enhancing data consistency and performance. This study presents a literature review of approaches developed before 2023, identifies key challenges, and proposes an optimized framework for real-time data transformation. The paper also includes experimental analyses, case studies, and visualizations that highlight performance improvements using the proposed approach.
References
Jonas, E., Schleier-Smith, J., Sreekanti, V., et al. (2019). Cloud Programming Simplified: A Berkeley View on Serverless Computing. USENIX Association.
Baldini, I., Castro, P., Chang, K., et al. (2018). Serverless Computing: Current Trends and Open Problems. Springer Cloud Computing.
Vinay, S. B. (2024). AI-Driven Patent Mining: Unveiling Innovation Patterns through Automated Knowledge Extraction. International Journal of Super AI (IJSAI), 1(1), 1-11.
McGrath, G., & Brenner, P. (2017). Serverless Computing: Design, Implementation, and Performance. IEEE IC2E.
S.Sankara Narayanan and M.Ramakrishnan, Software As A Service: MRI Cloud Automated Brain MRI Segmentation And Quantification Web Services, International Journal of Computer Engineering & Technology, 8(2), 2017, pp. 38–48.
Nivedhaa, N. (2024). Towards Efficient Data Migration in Cloud Computing: A Comparative Analysis of Methods and Tools. International Journal of Artificial Intelligence and Cloud Computing (IJAICC), 2(1), 1-16.
Dhanekulla, P. (2024). Blockchain and distributed ledger technology in modernizing insurance systems: Enhancing transparency and reducing fraud. International Journal of Computer Engineering and Technology (IJCET), 15(6), 275–290.
Newman, S. (2018). Building Microservices. O'Reilly Media.
K. Vasudevan, The Influence of AI-Produced Content on Improving Accessibility in Consumer Electronics. Indian Journal of Artificial Intelligence and Machine Learning (INDJAIML), 2(1), 2024, 1-11.
Dhanekulla, P. (2024). The role of middleware in integrating legacy systems with modern policy administration. International Journal of Scientific Research in Computer Science, Engineering and Information Technology (IJSRCSEIT), 10(6). https://doi.org/10.32628/CSEIT24106180
Fowler, M., & Lewis, J. (2020). Microservices: A Definition of This New Architectural Term. ThoughtWorks.
Gannon, D., Barga, R., & Sundaresan, N. (2020). Cloud-Native Data Science: Platforms and Integration. ACM Computing Surveys.
Taibi, D., Lenarduzzi, V., & Pahl, C. (2019). Microservices Anti-patterns: A Taxonomy. Springer Computing.
K K Ramachandran, The Evolution of Recurrent Neural Networks in Handling Long-Term Dependencies in Sequential Data. International Journal of Neural Networks and Deep Learning (IJNNDL), 1(1), 2024, pp. 1-10.
N.Kannan, Exploring Robustness and Generalization in Data Science Models through Multi-Fidelity Simulations and Transfer Learning. International Journal of Data Scientist (IJDST), 1(2), 2024, pp. 1-11.
Mukesh, V. (2025). Architecting intelligent systems with integration technologies to enable seamless automation in distributed cloud environments. International Journal of Advanced Research in Cloud Computing (IJARCC), 6(1),5-10
Dhanekulla, P. (2023). Intelligent claims management in insurance: The role of AI in modern policy systems. International Journal of Application or Innovation in Engineering and Management, 12(1), Article No-1, 17–2.
Meng, W., et al. (2019). Event-Driven Architecture for Streaming Data Pipelines. IEEE Big Data.
Zaharia, M., Chowdhury, M., Franklin, M., et al. (2016). Apache Spark: A Unified Engine for Big Data Processing. ACM SIGMOD.
Chen, Y., Hu, X., & Wu, W. (2018). Scalable Event Processing in Cloud-Native Environments. IEEE Cloud Computing.
Dhanekulla, P. (2024). Modernizing legacy systems in insurance: Strategies for seamless integration of cloud-based policy administration solutions. International Journal of Research in Computer Applications and Information Technology (IJRCAIT), 7(2), 1484–1498.
Chen, F., et al. (2017). Optimizing Serverless Computing for Large-Scale Workloads. ACM SoCC.
Gorton, I., & Klein, J. (2018). Understanding Performance Trade-offs in Microservices. IEEE Software.
Xu, J., Zhang, Y., & Lin, H. (2019). Real-Time Processing with Function-as-a-Service. ACM Middleware.
Dhanekulla, P. (2024). Real-Time Risk Assessment in Insurance: A Deep Learning Approach to Predictive Modeling. International Journal for Multidisciplinary Research (IJFMR), 6(6), 1–11.
Luo, X., et al. (2020). Cloud-Native Streaming Analytics for IoT. Springer IoT Journal.
Rindos, A., et al. (2021). Architecting Data Pipelines for Serverless and Microservices. IEEE CloudNet.
Kumar, R., & Arora, A. (2019). Fault Tolerance in Distributed Data Processing. ACM Journal of Cloud Computing.
Hellerstein, J., et al. (2022). Future of Data Engineering with Serverless Architectures. Springer Data Science Review.
Dhanekulla, P. (2023). The Use of Robotic Process Automation (RPA) In Policy Administration Systems.International Journal of Social Impact, 8(3), 168-176. DIP: 18.02.019/20230803, DOI: 10.25215/2455/0803019
Adhikari, K., et al. (2020). Scalable Stream Processing with Kubernetes and Kafka. IEEE Transactions on Cloud Computing.
Nezhad, B., et al. (2018). Autoscaling Strategies for Serverless Data Pipelines. ACM SIGMETRICS.
Gupta, S., et al. (2019). A Case for Combining Serverless and Microservices. IEEE Internet Computing
