Data Engineer
Architected ETL infrastructure and CI/CD with monitoring, built data warehouses/lakes and pipelines, migrated data to Databricks and Snowflake, created views and dashboards to serve insights to stakeholders.• Architected Airflow infrastructure on kubernetes and created various DAGs for ETL purposes.• Created CI/CD using CircleCI for Airflow development.• Developed cloudformation scripts to have consistent architecture during kubernetes migrations.• Created Kinesis event streaming pipeline to gather KPIs on user behaviors, exported data to s3, migrated to Databricks, separated it to different layers via views, and built Tableau dashboards.• Created EMR jobs to run daily data migrations, collected Spark metrics on DataDog.• Designed a new data warehouse on Redshift and developed ETL jobs to load data. • Created external tables to run Athena queries to leverage NoETL approach as feeding data directly from s3.• Created logical replication slots on RDS and Kafka connectors to consume updated data on the database, migrated pipelines from Fivetran to a Kafka. Created a script for historical resyncing Kafka CDC pipelines.• Developed BE Java microservices and maintained deployments for separate environments on EKS.• Created cron jobs for testing micro service, deployed micro services on kubernetes pods, built VPC peering between clusters, set proper security group inbound access and developed services and new features.• Updated Kubernetes cluster version on EKS worker nodes, updated certs on service configmap.Automated alerting and Slack messaging on failures during Kafka connector and Airflow DAG operations.