Pavan Kumar Email & Phone Number

Canada

• Integrated Azure Data Factory (ADF) into the data engineering ecosystem, orchestrating, and automating data movement tasks, data transformations, and workflows between on-premises sources, Azure storage, and Snowflake.
• Leveraged Azure Databricks and PySpark for complex data transformations, data wrangling, and data processing tasks within the Azure cloud environment. This included implementing efficient algorithms and distributed.
• Developed and optimized PySpark jobs within Azure Databricks for data cleansing, aggregation, and enrichment, ensuring data quality and consistency before loading into Snowflake's data warehouse.
• Developed and maintained complex SQL queries and scripts using TSQL and PostgreSQL to extract, transform, and load (ETL) large volumes of data for business analytics and reporting purposes.
• Designed and implemented database schemas, tables, views, and stored procedures to support data modeling and analysis requirements.
• Designed and implemented scalable data marts to support business intelligence and analytics, improving reporting efficiency and data accessibility.

• Led the successful migration of native Spark workloads to Databricks, leveraging Databricks' advanced features for scalability, reliability, and performance improvement.
• Implemented robust data governance solutions, including data cataloging using Databricks Catalog, metadata management, and data lineage tracking.
• Ensured adherence to regulatory requirements and data governance policies across Wells Fargo's data assets.
• Developed ETL processes to extract, transform, and load data into data marts, ensuring data quality and consistency across multiple sources.
• Implemented optimization techniques such as Auto Compaction, Ordering, and Vacuuming in Databricks to enhance data usability, storage efficiency, and query performance.
• Designed and implemented cost-effective infrastructure configurations in Databricks, optimizing resource utilization and minimizing operational costs for Wells Fargo.

• Data migration from On-premises to AWS S3 using Amazon transfer family and RDS.
• Developed data pipeline to ingest data into Apache Druid from different data sources.
• Involved in converting Hive/SQL queries into Spark transformations using Spark SQL, Spark Data Frames
• Deployed Hadoop clusters and Spark on Kubernetes.
• Created Python-based tools and scripts for ad-hoc data analysis and reporting, utilizing visualization libraries like Matplotlib and Seaborn.
• Developed dashboards and reports to present data insights to stakeholders.