As a dedicated and results-oriented Lead Data Engineer with 8 years of extensive experience, I have successfully led a team of 6 professionals in delivering robust and scalable data solutions. My expertise spans across cloud platforms (Azure, AWS), big data technologies (Spark, Hadoop), and programming languages (Scala, Python). I am proficient in version control using Git and have a strong background in ETL processes, data warehousing, and real-time data processing. I have developed and deployed data solutions on Azure and AWS, ensuring high availability and scalability, and implemented and optimized ETL processes using Spark and Scala, achieving significant performance improvements. I have collaborated with cross-functional teams to define data requirements and deliver actionable insights, established best practices for code management, version control, and continuous integration using Git, and mentored team members while providing technical guidance to foster a culture of continuous learning and improvement. With a Bachelor’s Degree in Computer Science, Engineering, or a related field, and relevant certifications such as AWS Certified Solutions Architect and Microsoft Certified: Azure Data Engineer Associate, my technical prowess, combined with strong leadership skills, positions me to effectively drive data initiatives and contribute to the success of any organization. I am ready for immediate joining to contribute my skills and lead transformative projects.

Migrating data from your existing HDFS deployment to Ozone for benefits such as scaling the metadata to billions offiles; supporting denser, more cost-effective storage nodes as a part of a strategy to segregate storage and compute;or cold storage and archival storage needs.we have an existing HDFS cluster from which you want a part or whole data to be moved to Ozone. TheOzoneservice can exist in the same cluster as HDFS or in a different cluster.

Involved in ETL workflow using AWS ADF, with PySpark as per business requirements, which includesextraction of data from Relation Database and loading it to Azure Sql DB.Extensively involved in writing transforming data in PySpark and pushing into ADLS Gen2.Storing the resulted data into Azure SQL DB for PowerBI and Sportfire Consumption.Experienced in data migration from On-Prem to Azure Cloud with Databricks using Spark api.Attend daily scrum calls and update the ADO user stories on daily basis.Monitor the pipeline jobs and report/fix them if any issues.

Providing AWS Cloud services including development, migration of On-premises Project into AWS Cloud and implementation of business rules. Starting from data-Analysis, data-design documents and data modelling to solve business problem.Mapping, gathering & eliciting business requirements and formulated error/defect management & unit testing approaches for developed solutions.Using Python and AWS Components such as EC2, EMR, Redshift, S3, Lambda, Athena & Step Function to develop AWS SolutionsDevelop various projects pipelines using Spark and AWS concepts using various Intuit owned frameworks.Managing and supervising a team of 5 people to carry on various errands.Migrating existing SQL, HQL and different DataMart’s, Data lakes and Data-Warehouses to Spark using AWS Cloud Environment.Meeting business application requirements & data sources by designing data extraction processes.Standardizing & consolidating existing Salesforce Orgs into a single instance and analyzing existing set-up for data loading intoSalesforce Objects, Contact, Account & Financial Account to provide a fundamental service for both individual & workspace solutionsIdentifying performance bottlenecks and applying performance tuning methods to resolve the same.Developing & implementing Data Integration & Warehouse Solutions; delivering technical expertise in supporting production lifecycle; executing ETL job scheduling and monitoring job runsDeployment and production for various crucial job on Tidal. Creating and maintaining job on tidal for scheduling suchpipelines.