Data Engineer
Created an auto-tuning program for the company Redshift database. Tuned all the tables (compression, sort/dist keys, etc) – saving 10% of space and improving the average runtime of all workflows by 30-40%. Previously workflows touching on tables > 1TB would often fail to complete at all, after the tuning – some workflows runtimes went from 1h to 10 min or less. Greatly helped the product analytics, bizops teams and product team in the company.• Started implementing materialized views on Redshift and Snowflake where appropriate – this greatly improved certain targeted workflows.• Implemented data lake concept – using Redshift Spectrum and airflow to create external daily updated tables on s3.• Created a daily scheduled program (on ec2 then ported into airflow) to resort/vacuum tables on daily basis in Redshift.• Set up and managed Airflow Scheduler (on Vagrant, then self hosted on ec2, then Google Composer) to schedule and run workflows/pipelines. Created numerous data pipelines to incrementally pull data from 3rd party systems into our databases (Snowflake, Redshift, RDS, Treasure Data) to run on Airflow. Created data pipelines to do complex transforms using python.• Worked on migration from Redshift to Snowflake, created scheduled pipelines to transform data using DBT and/or airflow. For large event tables ( > 1TB) created external tables on snowflake – s3 bucket accepts streaming data then a scheduled snowpipe updates external tables on snowflake, tables sorted by date for faster querying.• Consolidated financial transaction flows into a single schema, architecting the structure for efficient analytics (group effort). About 60m indiv. customer accounts, 16k merchant accounts – bringing data from 3rd party systems, joining and enriching pay data with customer/merchant data, consolidating into final curated tables – all automated, daily. Curated tables are used by bizops/product analytics to create tableau dashboards to calculate company metrics.
