At Banner Health, I spearheaded the development of scalable data pipelines using Apache Kafka, Spark, and AWS for real-time patient data processing. I designed a distributed data processing framework with Hadoop and Hive, automated ETL processes using Airflow, and established a centralized data lake on AWS S3 for advanced analytics. I implemented real-time analytics with Spark Streaming and Kafka, migrated legacy systems to AWS Redshift, and ensured HIPAA compliance through robust data governance. I optimized critical data pipelines for performance, collaborated with cross-functional teams to deliver data-driven solutions, and mentored junior engineers in best practices. Additionally, I developed and maintained robust data governance and data quality processes, ensuring compliance with HIPAA regulations and industry best practices for handling sensitive healthcare data. I also conducted performance tuning and optimizations for critical data pipelines, resulting in significant improvements in data processing times and resource utilization.

At HCLTech, I led the implementation of an AWS-based Data Lake using S3 and Glue, optimized SQL queries, and developed a security framework for S3 using Lambda and DynamoDB. I integrated Airflow with AWS for ML workflows, validated data fields, and converted SQL queries to ETL tasks. I used Databricks for data analysis, ingested data into Snowflake using Snowpipe, and implemented stringent security measures. I managed large datasets with PySpark and Pandas, scheduled Oozie workflows for Hadoop, and maintained a 99.9% uptime for critical data pipelines. Additionally, I performed information purging and applied changes utilizing Databricks and Spark information analysis and extensively utilized Databricks notebooks for interactive analysis utilizing Spark APIs. I also implemented stringent security measures by utilizing Snowflake’s integration with identity providers or Single Sign-On (SSO) and Multi-Factor Authentication, including encryption in transit and at rest, and ensured compliance with data protection regulations.