Pradeep Kumar Email & Phone Number

Topeka, Kansas, United States

As a Senior Data Engineer, I led the implementation of large-scale Lambda architectures using Azure Data Platform services such as Azure Data Lake, Azure Data Factory, and Databricks. I developed scalable ETL pipelines in PySpark and Spark SQL to handle large datasets efficiently, ensuring data quality and consistency. Additionally, I played a key role in optimizing NoSQL databases, Kafka clusters, and managing PostgreSQL security features to safeguard sensitive data.I also implemented various data transformation frameworks within Snowflake to streamline data processing and created insightful Tableau dashboards to visualize complex data trends. My work involved managing and optimizing Hadoop jobs and integrating Kafka for real-time data processing. These efforts significantly improved data governance, compliance reporting, and operational efficiency.I ensured smooth collaboration with cross-functional teams by managing multiple GitHub repositories and optimizing version control processes. This enhanced team productivity and ensured the integrity of data engineering projects, allowing the organization to meet regulatory standards and data-driven objectives.

Rochester, Minnesota, United States

At Mayo Clinic, I migrated on-premise applications to AWS, leveraging EC2 and S3 for efficient data processing and storage. I designed and developed ETL workflows using AWS Glue, enabling seamless migration of data to AWS Redshift and ensuring efficient data processing using PySpark and Kafka. My efforts led to a more robust data infrastructure capable of supporting real-time analytics and reporting.I developed complex Spark SQL queries to handle large datasets and built custom PySpark transformations for data aggregation and validation. Additionally, I implemented scalable Snowflake solutions and streamlined data pipelines using Kafka, ensuring real-time data integration and processing across multiple systems. My contributions significantly enhanced the organization’s ability to analyze large volumes of data in real time.In this role, I also developed interactive Tableau dashboards to present complex data insights and helped optimize reporting systems for better performance. My expertise in Snowflake, Hadoop, and Kafka ensured the scalability and efficiency of data processes, leading to improved decision-making capabilities within the clinic.

Bengaluru, Karnataka, India

I managed the end-to-end development of data pipelines using Azure Data Factory, Spark SQL, and Azure Data Lake Analytics. I played a crucial role in implementing real-time data processing solutions using Azure Stream Analytics, ensuring data integrity and seamless ingestion into Azure SQL databases. This enhanced the organization's ability to make data-driven decisions with timely and accurate insights.I leveraged PySpark and Kafka to build robust ETL pipelines capable of handling large-scale data processing tasks. My focus on optimizing MS SQL and NoSQL queries ensured the efficient retrieval and transformation of data across systems. Additionally, I integrated Apache HBase with Hadoop to improve the handling of unstructured data and developed real-time data pipelines for various reporting purposes.I also led the development of interactive Power BI dashboards, enabling stakeholders to visualize key metrics and make informed decisions. My experience in Power BI, coupled with a solid foundation in data engineering tools, enabled me to deliver high-quality reports that aligned with the organization's strategic objectives.

Mumbai, Maharashtra, India

I worked as a Data Analyst, focusing on integrating data from various sources using AWS services such as S3, EC2, and EMR for data transformation and analytics. I developed and maintained Python-based ETL pipelines to streamline data extraction, transformation, and loading processes, ensuring efficient handling of large datasets. Additionally, I managed AWS RDS databases, optimizing performance and implementing backup and recovery strategies to ensure data integrity.I played a key role in writing advanced SQL queries, including stored procedures and dynamic SQL, to support complex reporting requirements. I also designed MySQL databases to improve data retrieval and storage processes, and developed automated Excel reports using Pivot Tables, VLOOKUP, and complex formulas to enhance data analysis for business decisions.In addition, I utilized Power BI and Hadoop ecosystem components to create interactive dashboards and perform large-scale data processing tasks. My ability to integrate various data sources and provide actionable insights through data visualization tools allowed the organization to make data-driven decisions more effectively.