Data Engineer
Current* Data Pipeline Development and Optimization: Developed and maintained80+ efficient data pipelines in Databricks, significantly improving dataprocessing speed and reliability.* Cross-Functional Collaboration: Collaborated with clients to develop andimplement alternative end-to-end data pipelines, reducing dependency onexternal data products and improving data reliability.* Technology Migration and Modernization: Successfully migrated AWS EKScron jobs, leveraging a combination of shell scripts, SQL, Python, and othertechnologies to Databricks jobs, enhancing scalability and maintainability.* Infrastructure Development and Standardization: Developed a genericpipeline infrastructure to reduce development time and effort for similarpipelines, promoting standardization and consistency.* Team Leadership and Mentorship: Lead a team of 6 developers, applyingtechnical expertise and project management skills to deliver and maintain200+ pipelines. Fostered a collaborative team environment through Scrummaster and mentorship.* Data Monitoring and Alerting: Created Splunk dashboards to monitor datacount trends and job run status, enabling proactive issue identification andresolution.
