Data Engineer
CurrentDeveloped real-time ETL pipelines using Azure Data Factory (ADF) to ingest and process large volumes of machine sensor data, ensuring accurate and timely data availability for the production analytics dashboard.Designed and implemented data transformation workflows to clean and aggregate sensor data from various factory machines, enabling seamless integration with analytical models.Collaborated with data scientists and engineers to integrate predictive models into the production environment, allowing for real-time insights into equipment health and operational performance.Monitored and optimized ETL pipelines for efficient data processing, reducing latency and ensuring that factory managers received up-to-date machine performance metrics.Implemented alerting systems within Databricks to notify factory operators of impending machine failures, enabling proactive maintenance actions to minimize downtime.Integrated the analytics dashboard with Power BI, providing factory managers with easy-to-understand visualizations of machine performance, production efficiency, and maintenance predictions.Ensured data accuracy and consistency by conducting rigorous data validation and cleansing, guaranteeing that sensor data was reliable and suitable for analysis.Worked closely with operations and maintenance teams to define key performance indicators (KPIs) and metrics for machine health and production efficiency, ensuring that the analytics dashboard aligned with operational goals.Developed CI/CD pipelines using Azure DevOps to automate the deployment of updates to the ETL pipelines, predictive models, and dashboard, ensuring continuous improvement of the system.Provided thorough documentation on the system architecture, data pipelines, predictive models, and maintenance procedures, facilitating future scalability and maintenance.