More then. 6 years of experience working as a Data Engineer for multiple organizations to develop different data solutions.Collaborated effectively as a team member in the migration project, overseeing the seamless transition of the Big Data platform from Azure HDInsight 4.0 to Azure Databricks, ensuring minimal disruption to operations.Resolved compatibility issues encountered during the migration of jobs from Azure HDInsight 4.0 to Azure Databricks, ensuring smooth continuity of data processing and analysis.Played a key role in optimizing and enhancing data pipelines on Azure Databricks, resulting in a significant 40% increase in system performance and a notable 30% reduction in cost, thereby improving overall efficiency.Provided comprehensive support to the Data Science/Data Analytics teams, actively assisting them with their daily activities that required the use of the Big Data Platform, enabling streamlined workflows and facilitating data-driven decision-making.Developed and integrated automated ETL pipelines using shell scripting, PySpark, and Scala, effectively automating the loading process into the data lake storage using Azure Databricks, thereby improving data availability and accuracy.Implemented data reconciliation and control processes by collaborating with business analyst to establish data authenticity.Worked with business teams in development of life and annuity experience studies assumptions algorithms to predict key business indicators.