Data Engineer
Current- Increased Apache Impala data warehouse stability by 70% (measured by number of alerts) and query performance by 40% (measured by 95th percentile query runtimes) by reevaluating node roles to free up existing memory, setting guardrails such as query timeouts, memory and disk usage limits, and implementing efficient data compaction techniques (addressing the “Small Files Problem").- Built a new analytical platform using an open source version of Apache Airflow. The platform components were deployed on an existing Kubernetes cluster and Airflow tasks were run in dedicated pods on AWS spot nodes. This approach reduced costs and enabled greater resource scalability.- Increased customer confidence in the data being delivered by building a custom SQL-based data quality and SLA notification system to detect and respond to data quality issues.- Fostered a culture of open communication by constantly reporting issues and features in a public Slack channel, leading to improved transparency and cross-team collaboration.- Developed and maintained near real-time streaming pipelines ingesting data from Kafka topics and ETL pipelines extracting data from external APIs and operational Postgres-based databases.- Actively maintained and improved a legacy in-house data processing solution written in over 100,000 lines of Python code.