- Achieved a 97.5% cost improvement by redesigning and implementing an internal compliance service using S3, DynamoDB, Kinesis Streams, Firehose, Redshift, and Lambda, reducing monthly expenses from $200,000 to $5,000, while efficiently processing real-time data in batch.- Significant contribution in developing a CI/CD tool using Python, CFT, SNS, SQS, S3, Lambda, RDS, Redshift, and State Machines to ensure compliance with obligations such as GDPR, COPPA, DSAR, and other privacy laws across the entire organisations data warehouses. Implemented retryable and automatic recovery mechanisms for handling failures, optimizing performance, and capturing relevant metrics in the process for admins and data owners as well.- Designed and built robust data pipelines using S3, Spark, EMR, and StateMachines to handle incoming data upto 1TB on an hourly basis. Implemented highly optimized designs for performing efficient joins and operations across massive datasets, ensuring optimal processing and scalability.- Made significant contributions to the development of data models for RDS, Redshift, and QuickSight, ensuring efficient storage and retrieval of data. Developed complex stored procedures and views to support advanced data processing and analysis requirements.- Built pipelines using SNS, SQS, Lambda, and Glue in Spark to process Redshift metadata organization-wide, enriching an internal metadata discovery tool with valuable insights and enhancing data processing capabilities.

- Developed an Automated ETL recovery tool using Python script on EC2, which effectively restarted or triggered alarms for ETL pipelines in the event of failures, leveraging API calls to internal tools.- Created comprehensive dashboards with integrated alarms to monitor critical ETL pipelines, ensuring visibility for both owners and end users. Developed the dashboards using API calls to internal tools, enabling real-time monitoring of pipeline population and timely alerts for any anomalies or failures.- Successfully executed a large-scale data migration project, transferring data across Redshift clusters for over 100+ tables. Optimized ETL pipelines to enhance efficiency, performance, and data processing capabilities, resulting in improved overall system performance and reduced processing time.