I focused on optimizing data infrastructure, using tools like Apache Spark, Scala, and Databricks to cut processing time by 50% for large datasets. I designed scalable data warehousing solutions on AWS, fine-tuning ETL processes with Airflow and Python for faster and more accurate data retrieval. I also led real-time data integration projects using Kafka and Spark, which significantly boosted customer insights and decision-making. Automation was a big part of my work, where I streamlined… Show more I focused on optimizing data infrastructure, using tools like Apache Spark, Scala, and Databricks to cut processing time by 50% for large datasets. I designed scalable data warehousing solutions on AWS, fine-tuning ETL processes with Airflow and Python for faster and more accurate data retrieval. I also led real-time data integration projects using Kafka and Spark, which significantly boosted customer insights and decision-making. Automation was a big part of my work, where I streamlined workflows with Apache Airflow to ensure reliable data delivery. Throughout, I collaborated with cross-functional teams and mentored junior engineers in data engineering best practices. Show less

I worked on optimizing data pipelines for sustainability risk scoring, focusing on automating data ingestion and improving data quality. I led the redesign of an extraction pipeline using Scala and Apache Spark, which reduced inconsistencies by 90%. I also shifted the annual batch processing of risk data into real-time ingestion with Apache NiFi and Kafka, giving clients faster access to updated scores. Additionally, I built and maintained complex data models in Databricks to support physical… Show more I worked on optimizing data pipelines for sustainability risk scoring, focusing on automating data ingestion and improving data quality. I led the redesign of an extraction pipeline using Scala and Apache Spark, which reduced inconsistencies by 90%. I also shifted the annual batch processing of risk data into real-time ingestion with Apache NiFi and Kafka, giving clients faster access to updated scores. Additionally, I built and maintained complex data models in Databricks to support physical risk assessments for assets like Mines and Real Estate. Throughout, I collaborated with cross-functional teams to ensure data integrity and delivered reliable, automated workflows using Apache Airflow. Show less

I led data initiatives that optimized renewable energy analytics and improved customer insights. I revamped the data warehouse using a star schema and automated pipelines with Python, SQL, and Airflow, streamlining data processes. I spearheaded a reverse ETL project, moving 2 TB of data from AWS Redshift to the CRM system, cutting third-party CRM costs by 75%. I also migrated Airflow to AWS EC2, improving error handling and infrastructure management. Additionally, I developed Python algorithms… Show more I led data initiatives that optimized renewable energy analytics and improved customer insights. I revamped the data warehouse using a star schema and automated pipelines with Python, SQL, and Airflow, streamlining data processes. I spearheaded a reverse ETL project, moving 2 TB of data from AWS Redshift to the CRM system, cutting third-party CRM costs by 75%. I also migrated Airflow to AWS EC2, improving error handling and infrastructure management. Additionally, I developed Python algorithms for solar forecast optimization, increasing accuracy by 20%, and built a project management tool to enhance solar installation tracking and governance. Show less