Azure Data Engineer
Current• Lead the design and implementation of Azure Data Factory pipelines, automating data ingestion and processing tasks in manual data handling• Designed and implemented scalable data solutions on Microsoft Azure, leveraging services such as Azure Data Factory, Azure SQL Data Warehouse, and Azure Databricks, resulting in optimized data processing workflows• Implemented data security and access controls for Azure SQL Database and Azure Cosmos DB to ensure compliance with regulatory requirements• Managed Azure Synapse Analytics to design and maintain data warehousing solutions, optimizing data storage and query performance for analytical workloads• Monitored robust data security measures within Azure services and Snowflake, ensuring compliance with industry standards and data governance policies to safeguard sensitive information and maintain data integrity• Utilized PySpark to preprocess and analyze large-scale datasets, applying data transformations, filtering, and aggregations to prepare data for downstream analytics, resulting in enhanced data insights and model accuracy• Integrated Azure SQL Database instances, including performance tuning, query optimization, and scaling to manage increased workloads, ensuring high availability and minimal downtime for critical applications• Created pipelines in ADF using linked services to extract, transform and load data from data sources like Azure SQL, and Blob storage• Identified and implemented workflow optimizations within Azure Databricks, including cluster auto-scaling, resource tuning, and code optimization, leading to substantial cost savings and increased processing efficiency for complex data workloads• Developed the insights of data by optimizing Power BI reports and dashboards for performance, ensuring fast loading times and responsiveness