Senior Data Engineer
CurrentAs a Senior Data Engineer (B2C) I turn business needs from internal leaders into data pipelines used across teams within Glassdoor to support our internal data infrastructure. Using tools such as Airflow, I have designed, developed, tested, and deployed to production many ETL pipelines across our petabytes-level data warehouse to deliver tested data on a regular schedule to meet stakeholder SLA requirements. In collaboration with product owners, engineering teams, and stakeholders, I build easy-to-understand tables and metrics to meet business needs, support growth, and allow for internal performance transparency.• Leading a proof-of-concept initiative with Confluent Kafka to reduce latency and cost (by 10s of thousands of dollars) of consumer metrics from daily batch processing to near real-time, using Kafka to transport data from MSSQL, transform data in ksqlDB, and send to Rudderstack with Confluent connectors• Developed ETL pipeline to assist in migration of A/B testing framework to Optimizely, using the Optimizely API to download batch A/B testing events, S3 sensors to check when data has landed, and a custom Hive operator to clean data