Senior Software Engineer
CurrentMachine Learning Platforms TeamML Platforms (part of AI team) manages the AI team’s infrastructure and data storage/manipulation workflows.– Scoped and provisioned all of AI team’s infrastructure via Terraform for new EU data center ahead of schedule. Worked with cross-team stakeholders to understand and prioritize their needs. Performed opportunistic improvements, such as user provisioning via SCIM and security improvements to Databricks.– Designed and implemented GitOps continuous deployment framework for AI team’s compute resources on Databricks using Github Actions and Scala. Prepared RFC to gather feedback. Communicated with ML Platforms, Data Science, and SRE teams to deliver lean solution that satisfies all teams’ needs.– Utilized Databricks Change Data Feed to capture row-level changes in Spark structured streaming, avoiding costly file reprocessing for updated/deleted data. Wrote thorough integration tests to avoid data corruption.– Proposed and led intern project to create configurable data synthesis framework. Collaborated with data scientists to add features such as non-IID columns and rows, custom generators, and type validation.