Senior Lead Data Engineer
Current• Designed and implemented scalable data streaming pipelines by leveraging Databricks, Kinesis, S3, and Snowflake capabilities to enable real-time data analytics.• Implemented various proof of concepts (POCs) by utilizing newly integrated features in Databricks to enhance functionality and significantly reduce data processing time by 60% while also lowering costs.• Implemented a complex workflows in Databricks integrated with DBT jobs.• Developed a Databricks framework to monitor crucial business files in S3 and SFTP, and implemented an alerting mechanism to send the file status to end-user.• Experience in writing complex DBT Models to transform the structured data into business-oriented datasets.• Led multi-team collaboration initiatives to gather requirements and resolving cross-functional, technical challenges.