Worked on gathering security (equities, options, derivatives) data from different exchange feeds and storing historical data. Designed and deployed a Kubernetes-based containerized infrastructure for data processing and analytics, leading to a 20% increase in data processing capacity.Automated end-to-end workflows using AWS Step Functions and AWS Lambda, enhancing scheduling reliability and reducing manual intervention by 60%.Implemented Elasticsearch-based search solutions, reducing query response times by 50% and enhancing user experience in real-time analytics applications.Created pipelines in AWS Glue using Glue Jobs, Crawlers, and Databases to extract, transform, and load data from different sources like S3, RDS, Redshift, and DynamoDB.Implemented horizontal scaling for Elasticsearch clusters, ensuring reliability during peak traffic and supporting 1M+ daily transactions.Managed Git repositories for version control, implementing branching strategies for smooth collaboration and code integration, improving codebase integrity and reducing merge conflicts by 20%.Led the migration and warehousing initiative, transforming on-premise SQL-based models to AWS Redshift, optimizing storage through partitioning and indexing, and cutting storage costs by 20%.Architected and deployed AWS serverless infrastructure using Lambda, DynamoDB, and S3 to handle real-time data processing, applying Infrastructure as Code (IaC) principles with AWS CloudFormation to automate resource provisioning and ensure high availability.

Engineered a Data Transformation Pipeline for a Healthcare Analytics Platform on AWS infrastructure using DBT and Redshift, ensuring compliance with HIPAA regulations and improving data processing efficiency by 40%.Architected and implemented a dimensional model on AWS Redshift, structuring healthcare data into optimized fact and dimension tables, which enhanced reporting accuracy and reduced query times by 30%.Reduced storage costs by 30% through the use of S3 storage classes like S3 Glacier and S3 Intelligent-Tiering, optimizing the storage of infrequently accessed data while maintaining quick retrieval times.Developed and maintained ETL processes using Apache Spark integrated with Hadoop and HDFS, enabling the seamless processing and analysis of massive healthcare datasets and improving workflow efficiency.

Engineered a Financial Model Engine for the Credit Risk Platform on Big Data infrastructure using Scala and Spark.Migrated Hive queries to Spark transformations using DataFrames, Spark SQL, SQL Context, and Scala, enhancing processing speed by 40%.Developed and optimized data processing pipelines using Java and Spring Boot, integrating with Apache Kafka for real-time data streaming, which improved data ingestion efficiency by 30%.Implemented shell scripts to automate system backups and log management for Hadoop clusters, ensuring data integrity and reducing system downtime by 20%.