Data Engineer
Current• Successfully migrated data from traditional database systems to Azure, enhancing scalability and efficiency.• Generated ER diagrams, architectural design diagrams, process flow documents, data dictionaries, and mapping documents, facilitating project understanding and collaboration.• Wrote complex transformation scripts to populate dimension tables, enhancing data processing efficiency.• Engineered end-to-end metadata-driven pipelines, ensuring atomicity and re usability for streaming, batch, and event-driven processing. (Using ADF, Data bricks, Power Bi etc.)• Create Views, stored procedures and write complex SQL queries as per the requirements.• Designed and implemented metadata models for centralized custom logging and data quality framework, enhancing data integrity across sources. (Using Data bricks)• Implemented auto integration between Azure Active Directory and Data Lake, streamlining user access and permissions.• Created data and analytical models for all zones within the data lake, optimizing data utilization and analysis.• Developed a comprehensive package utilizing predefined client inputs, deploying robust batch processing, and streaming pipelines for multiple sources.• Proficiently managed Azure Data Factory (ADF) pipelines and integrated with various Azure services, ensuring seamless data management and delivery.