• Collecting the business requirements and drafting in to the design format/architecture diagram.• Migrating the on-prem informatica processing to azure cloud.• Writing the SQL stored procedures in to pyspark code.• Writing the common utilities in python code.• Developing the databricks notebooks.• Applied various business logics using pyspark.• Implemented optimized file format Delta.• Applied various optimizations while implementing the transformation… Show more • Collecting the business requirements and drafting in to the design format/architecture diagram.• Migrating the on-prem informatica processing to azure cloud.• Writing the SQL stored procedures in to pyspark code.• Writing the common utilities in python code.• Developing the databricks notebooks.• Applied various business logics using pyspark.• Implemented optimized file format Delta.• Applied various optimizations while implementing the transformation logic.• Writing back the transformed data to the processed layer (ADLS).• Orchestrate all pyspark jobs using ADF.• Enable the proper alerting mechanism, and data quality checks in ADF.• Enabled the logging mechanism.• Deployed the code on various environments using azure devops (CI/CD). • Prepared Unit test cases • Involved in CI/CD pipeline process.• Prepared sprint planning and achieved the goals.• Managed a team of 7 people. Show less

• Extract the data from various source systems and land it in ADLS (landing layer).• Converting the raw data in a unified file format in staging layer.• Start reading the data from staging layer using databricks notebooks.• Applied various business logics using pyspark.• Thorough understanding of mapping document while applying transformation logic.• Implemented optimized file formats like Avro, ORC, Parquet and Delta.• Applied various optimizations while implementing… Show more • Extract the data from various source systems and land it in ADLS (landing layer).• Converting the raw data in a unified file format in staging layer.• Start reading the data from staging layer using databricks notebooks.• Applied various business logics using pyspark.• Thorough understanding of mapping document while applying transformation logic.• Implemented optimized file formats like Avro, ORC, Parquet and Delta.• Applied various optimizations while implementing the transformation logic.• Writing back the transformed data to the processed layer(ADLS).• Orchestrate all pyspark jobs using ADF.• Enable the proper alerting mechanism, and data quality checks in ADF.• Properly handled exceptions when we are dealing with huge amount of data.• Enabled the logging mechanism • Prepared Unit test cases • Prepared DoD (Definition of Done)• Prepared proper transition plan to Ops(Operations) team. Show less

• Used the Cloudera as a distribution platform for Hadoop.• Created HIVE Tables on top of datasets which are in staging layer.• Used Various Hive performance optimization techniques.• Created Impala Tables as the target tables to load data into the integration layer.• Extensively used Cloudera Stream-sets to transfer Raw data files into HDFS foundation layer.• Used Parquet file format in all target tables in Impala for better performance.• Used Spark core and Spark SQL… Show more • Used the Cloudera as a distribution platform for Hadoop.• Created HIVE Tables on top of datasets which are in staging layer.• Used Various Hive performance optimization techniques.• Created Impala Tables as the target tables to load data into the integration layer.• Extensively used Cloudera Stream-sets to transfer Raw data files into HDFS foundation layer.• Used Parquet file format in all target tables in Impala for better performance.• Used Spark core and Spark SQL for data transformations.• Involved in Building data frames and RDD’s using Spark SQL and Spark Core Show less