-Built multiple data extraction pipelines using large dataset to assist with the data analysis requests-Build a PySpark based customized clustering algorithm with unsupervised learning technique to identify small merchants-Build the entire data processing logic to enrich the data for the stakeholder analysis for identifying merchants efficiently-Build the logic and wrote code to create dataset for the stakeholder analysis for identifying merchants in the USA and Canada. -Scheduled the data extraction jobs using Airflow like inhouse job management tool-Also extensively wrote PySpark SQL code to extract the global data for analysis-Build label datasets for the Data science team to train their models-Worked closely with technology team to build and implement solutions identified during the data analysis-Migrated multiple onprem ETL data pipeline to AWS cloud.-Build data pipelines using AWS services like S3,Glue,EMR,Redshift etc-Worked on project to process streaming data using Kinesis and Kafka.

Hadoop Developer-Created ETL pipelines - Integration and Migration, import data using Sqoop to load data from Oracle to HDFS-Hands-on experience with Hadoop and associated technologies such as MapReduce, Pig and Hive-Written complex Hive queries for data analysis to meet the business requirements-Solved performance issues in Hive with joins, groups and aggregation-Monitor and manage Hadoop cluster using Cloudera Manager -Involved in ETL, Data Integeration and Migration, Import data using Sqoop to load data from Oracle to HDFS on regular basis.- Importing and Exporting data into HDFS from Oracle Database and vice versa using Sqoop.