Lead Data Engineer
CurrentAs the only Data Engineer at Deepgram, I have built out a lot in a very short amount of time. Without a team of data engineers helping, I needed to make decisions that allowed me to get up-to-speed quickly, which is why some tools were chosen (ex: Glue vs Spark and/or Flink).* Built out a transactional Data Lakehouse using Apache Hudi with a Medallion architecture pattern* Setup replication for the Bronze-level data from a variety of sources including NATS JetStream, Postgres (Aurora), Salesforce, etc.* Implemented CDC (Change Data Capture) from PostgreSQL to our Data Lakehouse utilizing Debezium Server, and writing to NATS JetStream* Built micro-batched Salesforce exports from NATS JetStream, removing reliance on a 3rd party data management tool, saving ~$50k, annually* Tasked with getting row counts down in our main BI tool from over 200M rows to under 15M to avoid a surcharge; completed the work in under 3 months, saving an estimated $30k jump in annual costs* Completely removed reliance on our main BI tool in favor of a hosted Superset instance, saving ~$70k, annually.* Led research into a replacement visualization tool for our main BI tool, ultimately landing on Apache Superset* Created multiple Gold-level datasets for use in Superset, allowing the Product team to cutover from our main BI tool* Automated hourly aggregated data for our secondary BI tool ingestion, Rill