Tech Stack- Python, Java, SQL, Pyspark, NoSQL, AWS, Airflow, GitLab CI, Docker, KubernetesAWS services- S3, EMR, Glue, Step Functions, MWAA, RDS, Redshift, Athena, Lambda, APIGW, ELB, EC2, ECS, Batch, EKS, SageMaker, DynamoDB, NeptuneDB, Cognito, SNS, SQS, Cloudformation, CDK, SDK-> Optimized Pyspark SQL queries and eliminate unnecessary data duplication and resources in Glue ETL process, which reduced overall runtime and cost by 95%-> Designed and implemented solution using Airflow, Step Functions, and EMR to boost historical data processing. Reduced runtime by 70% and costs by 80%-> Designed and launched Tableau Server infrastructure on AWS, using EC2 instance. Ensured secure external access via API Gateway, ALB, and NLB, boosting data analysis capabilities-> Designed and implemented automated scheduling framework with MWAA (Airflow). Enables scheduled execution of data pipeline jobs and dependency checking before it trigger-> Migrated codebase for peak performance on AWS Batch and ECS Fargate. Enabled parallel processing, reducing runtime by 30%-> Developed ETL processes with PySpark and Glue, transforming and transferring data to S3. Optimized SQL queries, cutting runtime and resource usage for cost efficiency-> Experimented a solution to deploy ML models on SageMaker Endpoints. Integrated with AWS Batch for concurrent invocation with retry mechanism using SNS, SQS, and Lambda for endpoint failures-> Explored Spark Spline agent for lineage tracking. Integrated its information with Neptune graph DB via AWS API Gateway and Lambda, improving data insights-> Revamped a Java application, unified interface for users to interact with AWS resources like S3 and RDS. Enabled AWS credential generation and renewal via organization IdP credentials