Data Engineer
Current• Developed a dbt-powered data cleaning pipeline in Azure Synapse, improving data accuracy by 30% through type conformance, multi-source joins, and advanced Python cleaning scripts, resulting in unified datasets that reduced downstream query errors by 20%.• Built and optimized real-time data pipelines using Azure Data Factory, reducing query times by 40% and increasing analytics throughput by 35% through seamless integration of On-Prem SQL Server with ADLS Gen 2 and Azure Synapse… Show more • Developed a dbt-powered data cleaning pipeline in Azure Synapse, improving data accuracy by 30% through type conformance, multi-source joins, and advanced Python cleaning scripts, resulting in unified datasets that reduced downstream query errors by 20%.• Built and optimized real-time data pipelines using Azure Data Factory, reducing query times by 40% and increasing analytics throughput by 35% through seamless integration of On-Prem SQL Server with ADLS Gen 2 and Azure Synapse Analytics.• Architected and deployed AWS RDS and S3-based data lake, boosting data retrieval speeds by 25% and reducing storage costs by 15%, resulting in more efficient support for analytics projects across teams.• Engineered automated Apache Airflow pipelines, increasing data processing efficiency by 25%, cutting processing times by 50%, and expanding advanced analytics capabilities across the organization.• Led the development of adaptive web scraping solutions with BeautifulSoup4, Scrapy, and Selenium, extracting 100,000+ data points monthly with 99% accuracy, efficiently stored in AWS RDS, enhancing data accessibility by 30%.• Developed a scalable Flask API with Zappa, improving database access speed by 20%, and providing web developers secure, structured access to endpoints, facilitating real-time interactions with a 99.9% uptime.• Collaborated on a CI/CD pipeline built with GitHub Actions, automating testing and deployment processes. Integrated Docker to containerize coed and used Azure DevOps to run tests, ensuring the successful deployment of changes across dev and main branches.• Managed AWS accounts and IAM roles, ensuring secure, compliant, and efficient cloud operations across the organization.• Mentored 5+ new data engineers and data scientists, improving team capability with data ingestion and scraping tools by 30%, and reducing project delivery times by 15%. Show less