Oils and Petroleum industry

• Serverless ETL using Glue: Migrated/Created the ETL built on EC2 instances to the serverless using AWS Glue and monitored and triggered by Apache Airflow which does the cost reduction by 40-45 %• Enhanced ETL Performance: ETL job execution performance enhanced drastically while using Pyspark connectors for extracting data from salesforces by 8x to 10x for few jobs.• Real Time Data Pipelines: Worked on real time data pipeline using Kafka stream, EMR with Auto-Scaling and store in S3 as Data Lake which size around 300 Tb.• ETL and Pipeline Migration: Migrated/created Real-Time jobs and Batch jobs from Spark 2.x and python 2.x to Spark 3.x and Python 3.x in serverless environment for cost saving and performance efficiency.• Tools and Technology Used: Python, Pyspark, SparkSQL, Kafka, Data Lake, EMR, Airflow, Tableau, AWS Redshift, Glue, SQS, S3, Lambda, DynamoDB, Athena, Spectrum, Code Commit, GCP, Big Query, Domo

Develop, construct and maintain architectureswith business requirementsCreate Data pipeline for different business requirements Data warehousing,Data ModellingDevelop data set processesUse programming language and tools for ETLIdentify ways to improve data reliability, efficiency and qualityDeliver end to end data solutions Use large data sets to address business issuesDeploy sophisticated analytics programs, machine learning and statistical methodsPrepare data for predictive and prescriptive modellingHandling a team of 5 data engineers Find hidden patterns using dataUse data to discover tasks that can be automatedDeliver updates to stakeholders based on analyticsReporting-dash boarding and story telling using Tableau and Domo

Design schema ER diagram, implement business logic in technical termsDevelop and implement Common Data Model for different clients Develops and design the customer life cycle Management Engine to enhance customer behaviour and send personalised communications Develop and maintain ETL job using pentaho Data migration from one platform to another platform Creating stores procedures, triggers , functions and views to meet the requirements Used pandas, numpy, matplotlib and scipy Deliver End to End data driven decision Intract with client directly and deliver the solutions in the given timelines

Perform different types of Analysis Win back Analysis Churn out Analysis Impact Analysis Customer Retentions Analysis Customers Behaviour Analysis Customers Feedback Analysis EOSS Analysis Campaign Response Analysis Festival Impact Analysis Done customer segmentation, cross selling, Up Selling, Lifting incremental revenue Customer Life Cycle Management