- Designing and implementing scalable batch and real-time data pipelines on Azure using Python, Spark, and Azure Blob Storage optimizing data flow and enhancing system performance.- Managed daily data migrations of 1.6 TB from bronze to silver to gold layers with Azure Blob Storage, creating fault-tolerant workflows and performing advanced transformations with PySpark in Databricks.- Built CI/CD pipelines using GitHub workflows, and integrating DevOps best practices to support continuous integration and deployment across multi-cloud environments.- Increased processing efficiency by 20% through custom UDFs, caching strategies, and SQL and PySpark optimizations.- Collaborated within Agile and Scrum frameworks, aligning with cross-functional teams to ensure timely project deliverables.

- Developed and presented time-series forecasting models for predicting internal attrition rates in python with xgboost and prophet algorithms.- Reduced the cost of remediating vulnerabilities 10 hours/week by restructuring existing infrastructure for hierarchical data using advanced SQL techniques, like Common Table Expressions.- Saved 3 hours/week by automating the transfer of data between any public Google Sheet (using Google Sheet REST API) and Snowflake with python, to support our business intelligence tool, Tableau.- Mentored junior engineers on setting up python environments and developing REST API’s.

- Developed a python application for streaming big data (+1 TB) between data lakes using AWS EC2, S3, IAM, and Lambda in less than 3 hours.-Managed permissions for datasets with AWS IAM for over 100 clients.- Supported several ETL jobs for Spark/PySpark applications on top of AWS EMR, loading the data into Snowflake and Redshift.*Contract converted to full-time at Capital One*

- Built microservice for collecting consumer credit card data (as part of Google Pay initiative) using MongoDB, Java, and Docker.- Used Postman for testing REST API calls.- Deployed application with Docker through Jenkins pipeline on AWS ECS.

- Reduced the cost 25% of collecting data on IoT devices - American Airlines kiosks – by building data ingestion pipeline with Apache NiFi and forwarding to Splunk.- Used JIRA to discuss tickets with stakeholders, in an Agile environment, to support legacy java codebase.- Used linux, Windows, and macOS command line tools for collecting and analyzing server logs.

- Supported a real-time supply chain network for the U.S Department of Defense- Improved SQL performance with tuning

- Collected speed, volume, turning movement, and geospatial data for traffic signal light timing schedules

- Secured and processed highly sensitive information