Data Engineer
Bengaluru, Karnataka, India
- Managed, configured, and scheduled resources across the cluster utilizing Azure Kubernetes Service.
- Worked extensively on Azure Data Factory, encompassing data transformations, Integration Runtimes, Azure Key Vaults, Triggers, and migrated data factory pipelines to higher environments using ARM Templates.
- Monitored Spark cluster using Log Analytics and Ambari Web UI, transitioned log storage from Cassandra to Azure SQL Data Warehouse for improved query performance.
- Utilized Azure Data Factory to integrate on-premises data sources (such as MySQL and Cassandra) and cloud data sources (Blob storage and Azure SQL DB), applied transformations, and loaded data into Azure Synapse.
- Created dashboards and visualizations for data analysis, with a focus on Microsoft products like SQL Server Reporting Services (SSRS) and Power BI.
- Demonstrated proficiency in utilizing R for data analysis and statistical modeling in cloud-based environments, enriching the data engineering toolkit and facilitating advanced analytics capabilities.