Azure Data Engineer
Current•Maintained data pipelines using Azure Data Factory and Azure Data Factory Data Flows. Schedule and orchestrate data workflows for efficient ETL (Extract, Transform, Load) processes. Manage and organize data in Azure Data Lake Storage and Azure Blob Storage, ensuring data is stored securely and efficiently. • Responsible for data partitioning and optimization strategies to enhance storage performance. Utilize Azure Synapse Analytics and Azure HDInsight for processing and analyzing large volumes of data. Develop and optimize data processing jobs, including data transformations and aggregations. Leverage Azure Databricks for advanced analytics and machine learning tasks. •Collaborated with data scientists to build and deploy machine learning models. Implement real-time data processing solutions using Azure Stream Analytics to handle streaming data. Configure and manage event hubs, IoT hubs, and event-driven architectures. •Adapted workflows and automation using Azure Logic Apps to streamline data-related tasks and trigger notifications. Maintaining metadata catalogs using Azure Data Catalog and Azure Purview for data discovery, lineage tracking, and governance. •Managed and administered databases using Azure SQL Database, Azure Cosmos DB, and Azure Data Explorer. Ensure data security, backup, and high availability for databases. •Continuously monitored and optimized data pipelines, queries, and data storage for performance and cost efficiency up to 70%. Implement caching and indexing strategies as needed. • Effective in data quality checks and data validation processes to ensure data accuracy and reliability. • Documented data engineering processes, data models, and workflows. Collaborate with data scientists, data analysts, and business stakeholders to gather data requirements and deliver solutions. •Involved in Implementing monitoring, logging, and alerting solutions using Azure Monitor to proactively detect and address data-related issues.