Data Engineer
Current- Led seamless migration of Hadoop infrastructure to Azure, collaborating cross-functionally to minimize disruptions.
- Designed and implemented Azure-based data lake architecture, accommodating existing Hadoop ecosystem for scalability and flexibility.
- Developed and maintained efficient Python and PySpark-based data pipelines on Azure Databricks, processing large volumes effectively.
- Implemented data quality checks and monitoring mechanisms to enhance reliability and integrity of pipelines, ensuring data trustworthiness.
- Leveraged Azure services like SQL Data Warehouse and Synapse Analytics for optimized data warehousing solutions, enhancing resource utilization.
- Assisted in designing and developing data pipelines to facilitate the ingestion, processing, and storage of healthcare data.