System Engineer - Azure Data Engineer
Bengaluru, Karnataka, India
Worked on data migration projects for Europe-based insurance clients, focusing on both data and on-premises ETL tool migration to Azure using Azure analytics services and advanced data engineering techniques. Key contributions include:Azure Data Factory: Utilized for ETL and data orchestration, achieving significant performance improvements of over 25%.Azure Databricks: Implemented transformation logic using Medallion and Delta Lakehouse architecture with Unity Catalog.Azure Synapse: Used for data warehousing and data mart use cases.ADLS Gen2: Employed for raw data storage and integration with the Lakehouse architecture.Azure T-SQL: Queried data within data marts and created optimized stored procedures.Azure Analytics Service: Logged services with data queried using KQL in Log Analytics.Azure Event Hubs: Captured CDC (Change Data Capture) from Qlik Enterprise Manager to ingest source changes.Power BI: Designed data visualizations and reports, ensuring high-quality content and presentation.Azure DevOps: Managed deployments across environments, version control with repositories, and created CI/CD pipelines.Slowly Changing Dimensions (SCD): Worked primarily with SCD Type 2 for storing and analyzing historical data.Spark SQL: Queried data within the Lakehouse architecture.PySpark: Developed optimized transformation logic using techniques such as bucketing, coalescing, and repartitioning.Hands-on end-to-end project experience, from design to production release and support.Worked with APIs, various data ingestion methods, and both batch and stream data processing.Used ServiceNow as a ticketing and incident management tool for major incidents.Followed Agile Scrum methodologies, managing projects with Jira and Rally, documenting pipeline configurations in Confluence, and ensuring data governance and GDPR compliance in ETL processes.