Intermediate Software Engineer
Current- Created a Spark ETL application with Java 11 to load millions of customer configuration records per day into an on-premise Hadoop data warehouse. These records enabled four new report types to be delivered to customers.- Designed and optimized Trino SQL queries used to generate compliance reports for customers. Partitioning, CTEs and execution plan analysis were all used to minimize execution time as each report queries terabytes of data.- Containerized legacy applications using Docker and onboarded them to a company-wide Kubernetes cluster. This allowed for a simplified CI/CD pipeline to be used, reducing deployment time by 80%. - Implemented a low-latency Kafka Connect plugin to deduplicate events read from upstream Kafka topics. Leveraged a NoSQL datastore to efficiently store and query event IDs in a highly concurrent environment.- Created a suite of Grafana dashboards and alerts to monitor production ETL applications. These dashboards reported on application status and resource usage and were used daily by DevOps to detect and debug production issues.