• Worked in Agile software development methodology, collaborated with business partners, stakeholder, Project Manager and build the Epics, feature, and user stories. Involved in Sprint plannings, showcases and Retrospectives.• Involved in team project meeting, designed, and developed data solutions. • Designed, developed, and deployed DataLakes, Data Marts and Datawarehouse using AWS cloud like AWS S3, AWS RDS and AWS Redshift and terraform. • Designed, developed, and deployed Datawarehouse, AWS Redshift, applied my best practices. • Designed, developed, and deployed both batch and streaming pipelines using Confluent Kafka, Kubernetes, docker and AWS Cloud.

• Migrated existing Realtime streaming and batch pipelines from on-perm to AWS cloud. Utilized Kafka, Docker, and AWS Services like AWS S3, Lambda, IAM, Glue, Athena, RDS, Redshift, EMR, etc.,• Designed, developed, and deployed DataLakes, Data Marts and Datawarehouse using AWS cloud like AWS S3, AWS RDS and AWS Redshift and terraform. • Designed, developed, and deployed self-service ELT and ETL pipelines using AWS services like, Lambda, Glue, EMR, events, SNS, Redshift, S3, IAM, etc. • Created several Airflow dags used various operators, developed custom hooks and plugins. • Designed and developed Airflow Dag to extract data from source, load data into DataLake, process the data and load data into Datawarehouse.• Built a series of Spark Applications and Hive scripts to produce various analytical datasets needed for digital marketing teams.

• Designed, developed, and deployed DataLake, Data Warehouse, using Google cloud services like Google Cloud Storage and BigQuery. • Designed, developed, and deployed streaming and batch pipelines using Google Cloud services like, Cloud Function, Cloud Storage, BigQuery, DataFlow, DataProc, Compute Engine, Kubernetes. • Designed and developed several batch pipelines using Airflow, Python, Spark and Google cloud services like Cloud Function, Cloud Storage, BigQuery, DataFlow, DataProc, etc.,• Designed and developed streaming pipelines using Apache Kafka, docker and Kubernetes and Cloud Storage. • Designed, developed, and deployed several airflow dags to read data from source, load data into DataLake and process the data and load into Datawarehouse.

• Designed and developed data flow from sources to our data Lake Apache HDFS.• Designed and developed both Batch and streaming Jobs using Apache Hadoop ecosystem.• Developed and scheduled Hive scripts and Spark jobs using Apache Airflow.• Used Spark SQL to load complex nested JSON into hive tables.• Performed masking and unmasking if the data as per the requests.• Developed Spark-Streaming applications to consume the data from Kafka topics and to insert the processed streams to Apache Hive tables.