As a Senior Data Engineer, I have developed and maintained complex data pipelines using GCP services (Dataproc, Compute Engine, Cloud Storage, Cloud SQL, Cloud Functions, Pub/Sub, Dataflow, Cloud Tasks, BigQuery, and Google Kubernetes Engine), and optimized ETL processes with Python and SQL. I created Hive tables on HDFS, utilized Terraform for Compute Engine provisioning, and automated data transitions from Firestore to BigQuery. My work includes real-time data streaming with Kafka and Spark, data integration with Snowflake, and implementing BigQuery solutions. Additionally, I automated workflows using Apache Airflow, developed ETL pipelines with Matillion, and performed database maintenance. I also leveraged Looker for data visualization and managed the ELK Stack for log analysis.

As a Data Engineer at Shell, I built real-time streaming ETL pipelines using AWS Kinesis Data Streams and migrated on-premises SQL databases to Amazon Redshift with AWS Glue. I created reusable Glue jobs, developed workflows for data transfer from Amazon S3 to Amazon RDS, and implemented analysis solutions using AWS Lake Formation and Databricks on AWS. I optimized ETL processes in AWS Glue, developed high-performance data ingestion pipelines, and created dynamic Glue workflows. I used SQL and AWS Data Pipeline for efficient data import to Amazon Redshift, upgraded Redshift clusters, and developed fact and dimension tables. Additionally, I created QuickSight and Amazon Managed Grafana reports, used AWS Glue DataBrew for data preparation, and utilized Apache Spark on EMR to build data models from Kinesis Data Streams for real-time persistence into Cassandra and Redshift.

As a Big Data Engineer at Cigniti Technologies, I built ETL pipelines, developed data applications with HDFS, Hive, Spark, and Scala, and moved data from the data lake to Teradata. I created catalog tables with complex joins for large datasets, automated data validation with shell scripts, and optimized hive scripts. I troubleshot pipeline issues with MapReduce, Tez, and Spark, resolved data discrepancies, and improved algorithm performance. Additionally, I migrated data to Google Cloud Platform (GCP), maintained a Global Data Lake on GCP, and transformed business requirements into technical specifications. I also focused on data mapping, performance tuning, and data modeling in data warehouse environments.