Software Engineer 2
Current• Contributed to the design of a robust, scalable and fault-tolerant big data system, responsible for generating datasets for Azure Customers’ consumption. Played a pivotal role in integration, end-to-end testing as well as collaboration with multiple data consumers for data absorption contract discussions.• Owned the development of data integration pipelines utilizing Scala, Spark, and SQL for processing Consumption data. Implemented data quality checks and alert mechanisms to guarantee the accuracy and completeness of data.• Proactively enhanced system stability and scalability by diligently identifying anomalies in datasets and system behavior. Significantly reduced live sites by ~20 per month and improved processing time by over 50%.