Data Engineer
New York, New York, United States
I drove our internal and investor analytics efforts by means of establishing and managing our data lake in AWS. I ensured an efficient, fast, and robust release process for every data-focused product by:● Reducing monthly data processing costs by 47% by leveraging incremental loads using job bookmarking in AWS Glue and Apache Iceberg format in Athena. ● Saving $1500 per month in database and reporting costs by moving Quicksight dashboards off SQL Server database instance to AWS data lake in S3. ● Saving 8 person-hours weekly by automating BIOps using GitHub Actions and AWS Quicksight API for BI Analysts to release dashboards (I set up a GitHub environment for deployment protection). ● Building 2-way pipeline to push loan status updates to and pull reports from MERSCORP Holdings’ SFTP server using AWS Step Functions, Lambda, and Athena. Also set up alerting using PagerDuty and SNS for the same. ● Implementing row-level security for multi-investor reporting using DynamoDB, Athena, Quicksight, and Glue on client portal.