Senior Software Engineer
CurrentWorked closely with external field engineers to fine-tune ~50 Databricks Jobs responsible for ingesting/maintaining a sustained 500k records-per-second into a multi-petabyte Delta Lakehouse that will soon power Segment's customer data profiles across all product offerings (Profiles Data Lake).Drove down AWS S3 costs (storage + API requests) for the Profiles Data Lake by 300%.Owned operational readiness and excellence for Profiles Data Lake "go-live", ensuring a resilient Data Platform complete with: real-time systems monitoring via DataDog; custom dashboard panels; Slack alerting; GDPR/CCPA/BCR compliance; PagerDuty integration; & internal documentation/runbooks.Introduced & productionalized concurrent patterns in Scala/Spark allowing multiple tenants/customers to process at the same time; ultimately decreasing job runtimes by 44% on average while increasing cluster worker/executor CPU utilization by 8x on compute-heavy tasks.Assisted in the initial setup of a cloud-to-cloud migration process using Python, Apache Airflow, GCP, AWS [Athena/Glue/MWAA/S3/Lambda/StepFunctions], Scala, and Databricks; ultimately allowing our team to automate our migration & backfill processes.