➡️ Developed spark jobs that ingest data to ADLS gen2 from multiple data sources and executes the notebook to clean, process and segregate the data based on the business requirements➡️ Worked on high level transformations using Databricks to transform the data as required by the business➡️ Applied various segregation logics for data preparation which further helped in processing the data efficiently➡️ Optimized the existing data pipelines to run faster by applying various optimization techniques➡️ Worked on performance tuning for the spark jobs that brought down the processing time by 50%➡️ Led the migration of data from multiple sources to ADLS gen2 thereby empowering teams to analyze it➡️ Achieved proficiency in Azure Data Factory, Databricks, ADLS Gen2, Delta Lake and other Azure services.➡️ Collaborated with client in requirement gathering and analysis before developing the data pipelines

➡️ Loading the files into DM6 (internal ETL tool), generating fact and dimension tables➡️ Based on the request adding business rules, filters and transformations to the process, generating report and sharing with the client➡️ Identified a scope to improve, discussed with the team and implemented the incremental data loading which saved the cost and server runtime by 70%

➡️ Developing ETL process and implementing Data warehouse system in SQL Server based on requirements gathered➡️ Designing and achieving star schema structure for data warehouse system➡️ Creating Datamarts for reporting solutions➡️ Performing Root cause analysis and fine tuning