Data Engineer
Current• Designed and implemented internal process improvements by automating manual tasks, optimizing data delivery, and redesigning infrastructure for scalability and security, while enhancing the analytics platform within the BI architecture to prioritize data security, quality, timeliness, and extensibility.• Led the onboarding and implementation of an Azure CI/CD pipeline integrated with Git, Docker, JFrog, Apache Airflow, Prometheus, and Grafana for the tech team. Successfully migrated… Show more • Designed and implemented internal process improvements by automating manual tasks, optimizing data delivery, and redesigning infrastructure for scalability and security, while enhancing the analytics platform within the BI architecture to prioritize data security, quality, timeliness, and extensibility.• Led the onboarding and implementation of an Azure CI/CD pipeline integrated with Git, Docker, JFrog, Apache Airflow, Prometheus, and Grafana for the tech team. Successfully migrated all jobs to the new system, enhancing version control, improving turnaround time, and providing greater operational visibility.• Commissioned ~50+ servers for automation, integration, and transformation solutions with comprehensive network and application configurations. Set up the Airflow workflow engine, PostgreSQL, Prometheus, Grafana, and other microservices from scratch.• Designed and implemented a comprehensive solution to monitor the health, configuration, and operational status of IoT camera and controller devices across India. Developed an automated ticketing system to facilitate efficient issue resolution by field engineers, including root cause analysis and automated report generation for HQ and field teams to optimize system performance.• Proficient in designing and implementing complex ETL workflows for real-time and batch processing using open-source technologies, enabling ad-hoc analysis, dashboarding, and seamless integrations. Show less
