Pavan Kumar Email & Phone Number

Canada

• Integrated Azure Data Factory (ADF) into the data engineering ecosystem, orchestrating, and automating data movement tasks, data transformations, and workflows between on-premises sources, Azure storage, and Snowflake.• Leveraged Azure Databricks and PySpark for complex data transformations, data wrangling, and data processing tasks within the Azure cloud environment. This included implementing efficient algorithms and distributed computing techniques to handle large-scale data processing requirements.• Developed and optimized PySpark jobs within Azure Databricks for data cleansing, aggregation, and enrichment, ensuring data quality and consistency before loading into Snowflake's data warehouse.• Developed and maintained complex SQL queries and scripts using TSQL and PostgreSQL to extract, transform, and load (ETL) large volumes of data for business analytics and reporting purposes.• Designed and implemented database schemas, tables, views, and stored procedures to support data modeling and analysis requirements.• Designed and implemented scalable data marts to support business intelligence and analytics, improving reporting efficiency and data accessibility.• Collaborated with cross-functional teams to optimize database performance and query execution times, resulting in a 25% reduction in report generation time.• Conducted data quality assessments and implemented data cleansing and validation processes to ensure data accuracy and consistency across systems.• Participated in database migration projects, including upgrading SQL Server versions, and migrating data to cloud-based PostgreSQL instances, ensuring seamless data continuity and minimal downtime.• Provided technical expertise and support to junior team members, offering guidance on SQL best practices, query optimization techniques, and database administration tasks.

• Led the successful migration of native Spark workloads to Databricks, leveraging Databricks' advanced features for scalability, reliability, and performance improvement.• Implemented robust data governance solutions, including data cataloging using Databricks Catalog, metadata management, and data lineage tracking.• Ensured adherence to regulatory requirements and data governance policies across Wells Fargo's data assets.• Developed ETL processes to extract, transform, and load data into data marts, ensuring data quality and consistency across multiple sources.• Implemented optimization techniques such as Auto Compaction, Ordering, and Vacuuming in Databricks to enhance data usability, storage efficiency, and query performance.• Designed and implemented cost-effective infrastructure configurations in Databricks, optimizing resource utilization and minimizing operational costs for Wells Fargo.• Utilized Massive Parallel Processing (MPP) techniques in Spark SQL and PySpark within Databricks to optimize data processing performance and query execution speed.• Developed a robust orchestration layer using Databricks Jobs and Azure Data Factory (ADF) for automating data workflows, scheduling, and dependency management.• Implemented Continuous Integration and Continuous Deployment (CICD) pipelines for Databricks using Azure DevOps, ensuring automated testing and deployment of code changes.• Implemented near real-time data processing pipelines using Delta Lake and Structured Streaming in Databricks, enabling rapid data ingestion, transformation, and analysis.• Implemented comprehensive security measures in Delta Lake within Databricks, including access controls, encryption, and data masking techniques, to ensure data protection.

• Data migration from On-premises to AWS S3 using Amazon transfer family and RDS.• Developed data pipeline to ingest data into Apache Druid from different data sources.• Involved in converting Hive/SQL queries into Spark transformations using Spark SQL, Spark Data Frames• Deployed Hadoop clusters and Spark on Kubernetes.• Created Python-based tools and scripts for ad-hoc data analysis and reporting, utilizing visualization libraries like Matplotlib and Seaborn.• Developed dashboards and reports to present data insights to stakeholders.• Integrated Python with big data frameworks such as Apache Spark or Hadoop for large-scale data processing and analytics.• Developed PySpark scripts for distributed data processing tasks.• Worked with HDFS Sink Connector to transfer data from Kafka topics to files on HDFS clusters.• Built and administered Kafka cluster of 10 nodes and implemented Kafka mirror maker.• Used Flink Streaming for pipelined Flink engine to process data streams to deploy new API including definition of flexible windows.• Developed Data Lake platform with Apache Drill, Spark, AWS Athena, Glue, Redshift and AWS S3• Developed and implemented Hive scripts for transformations such as evaluation, filtering, and aggregation.• Developed Spark Streaming applications with Kafka for real time data processing using Scala and Spark• Have sound knowledge of Kafka message partitioning and setting up of replication factor in Kafka cluster.• Reprocessed failed messages in Kafka topics using offset id.• Actively monitored performance metrics of enterprise Kafka cluster and analyzed and debugged logs and resolved issues in production and non-production environments.• Reduced the latency of Spark Jobs by tweaking the spark configurations and follow the performance and optimization techniques.