Sr Data Engineer
CurrentDesigned, built, tested, and maintained end-to-end data pipelines, data integration, ETL processes, and data management delivery within Azure Cloud using Azure Data Factory, Azure Data Lake Storage and Azure Databricks.Ingested/Migrated data, applied transformation logic, and performed continuous data quality checks on data from various sources.Determined the lifecycle from analysis to production, focusing on data validation, defining logic, performing transformations, and creating end-to-end ETL data pipelines.Manipulated semi-structured and unstructured data using Azure Databricks into Bronze-Silver-Gold Zones using PySpark as programming language.Developed Spark applications using Scala and Spark SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.Created optimized data pipelines using Spark with Scala and Python.Utilized Synapse's integration with Azure Data Factory and Azure Databricks to create automated data pipelines for ingesting and transforming data from diverse sources.Created and developed Stored Procedures, Joins, and Triggers to handle complex business rules within Azure environment.Worked in Agile Teams, participated in sprints, daily standups, backlog grooming, and used JIRA for progress tracking.Successfully migrated applications from Cassandra DB to Azure Data Lake Storage Gen 1 using Azure Data Factory.Utilized Apache Airflow for automating ETL processes between Azure SQL Database, Azure Data Lake Storage, and other Azure data services.Organized Azure resources into resource groups using Terraform.Incorporated Azure services into Airflow DAGs using Azure-related operators and hooks.Enhanced security and credential management by integrating Apache Airflow with Azure Key Vault.Implemented RBAC policies using Terraform for security and access control across Azure resources.
