Lead Data Engineer
Current• ETL Pipeline:o Designed and built ETL pipelines to ingest, process, and store millions of rows of data from various sources with cost saving and efficiency in mind.o Built Sub/Pub and CRON based pipelines to ingest raw data from sources such as MSSQL, CSV flat files, and external APIs using AWS SnS, AWS Lambda, and AWS DMS replication tasks. o Built pipelines using Python, PySpark, AWS Glue, and AWS Step Functions to process and enrich the ingested raw data.o Used AWS Redshift, AWS S3 (using HIVE Partition), AWS Glue Catalog, and AWS Athena to provide efficient solutions for storing and accessing the enriched data.o Reviewed queries by PowerBI dashboard builders and other data consumers to ensure efficiency and security.o Provided data analysis and insights to business users such as Finance, Accounting, Transportation and Sales.• Held weekly office hours to aid Jr. Engineer with troubleshooting and reviewing architectural designs for future projects. • Supported the Scrum team by providing architectural guidance, conducting code reviews, approving git pull requests, and solving daily issues as a tech lead.