Designing & Building Data Analytics and Gen AI Platform using LLM

As a Data Engineer and Data Platform Solution Architect, I was responsible for designing and building a platform-agnostic data engineering platform. My key achievements in this role include:- Building a platform to run big data applications like Spark, Presto/Trino, Hive Metastore Service, Apache Airflow, and JupyterHub on Kubernetes, which can be easily lifted and shifted to any public cloud provider or on-premise environment.- Migrating Seagate Enterprise Data Platform from AWS to Seagate Lyve Cloud, ensuring seamless transition of data and applications.- Migrating thousands of production Spark jobs from AWS EMR to Spark Operator on Kubernetes, achieving better performance, cost efficiency, and resource utilization.- Optimizing Spark on Kubernetes performance, cost, and resource utilization by using NMVe, S3A Committers, and enabling graceful decommission.- Using new Spark 3 features like AQE and Dynamic Partition Pruning to boost performance of existing jobs and reduce processing time.- Extracting important Spark driver and executor JMX metrics to Prometheus and using those to optimize resource utilization, identify bottlenecks, and fine-tune the platform.- Conducting AWS cost optimization across EMR, EKS, S3, and EC2, achieving significant savings for the organization.- Building a CI/CD pipeline for Kubernetes application deployments and data pipelines, ensuring fast and reliable delivery of new features and updates.- Performing POC with innovative data products like Varada, Alluxio, Iguazio, Neuroblade, and evaluating their potential to enhance the platform and address specific use cases.- Segregating compute and storage, scaling them independently, and solving latency issues by implementing suitable and smart caching solutions.

As a Data Engineer, I was responsible for designing and building platform-agnostic data engineering pipelines. My key achievements in this role include:- Designed and built data pipelines using the latest open source tools such as Airflow, Spark, Livy, Presto, and Kubernetes, to enable efficient data processing and analysis.- Developed a generic ETL framework using Spark and Presto to allow for seamless data integration across various sources.- Successfully migrated data pipelines from on-premises MapReduce to Spark on AWS EMR, resulting in improved scalability and performance.- Deployed a Presto cluster using AWS EC2 ASG with auto scaling and graceful shutdown features to ensure high availability and reduce operational costs.- Conducted Presto cluster stabilization and optimization activities to improve query performance, reduce latency, and ensure reliable operation.

As a Data Engineer at Barclays, I achieved the following:- Designed and developed a highly scalable Slowly Changing Dimension (SCD) engine using Spark and Scala, which improved data quality and reduced processing time.- Developed a reconciliation tool using Spark to compare large volumes of data from different sources, which helped ensure data accuracy and consistency across the organization.- Contributed to various data warehouse and data mart projects utilizing Big Data technologies, demonstrating my ability to work with diverse teams and successfully deliver high-quality data solutions.

As a Data Engineer at Cognizant, I achieved the following:- Worked on various Big Data projects for Barclays, contributing to their successful implementation and completion- Involved in data migration and transformation projects, ensuring smooth transition of data between systems- Designed and developed multiple automation utilities as part of the innovation team, improving overall efficiency and productivity- Utilized skills in big data technologies and programming languages such as Hadoop, Spark, Scala, and Python to build scalable and effective solutions- Collaborated with cross-functional teams, including data scientists, business analysts, and software developers, to deliver high-quality results