• Implemented Apache Flink streaming applications to handle real-time data processing with low latency, achieving 20% faster results compared to Apache Spark Streaming.• Designed and implemented ETL (Extract, Transform, Load) pipelines using AWS Glue for efficiently moving and transforming data between various sources (S3, RDS, DynamoDB) and target data warehouses (Redshift, Snowflake).• Executed AWS Data Pipeline to orchestrate complex data workflows, reducing manual intervention by 30% and improving data reliability.• Accomplished interactive Power BI reports leveraging advanced DAX and Power Query modeling techniques to unlock deeper data insights and facilitate informed decision-making.• Maintained various Data Build Tool (DBT) models and macros, focusing on incremental model design and modularization, resulting in a 40% increase in data pipeline efficiency and a reduction in pipeline execution time.• Developed and deployed various interactive QuickSight dashboards to deliver actionable insights, boosting data-driven decision-making

• Developed and optimized SQL queries for data extraction, transformation, and loading (ETL) processes, achieving an average 15% reduction in query execution time.• Used Spark libraries like Spark SQL and MLlib to perform complex data analysis tasks, resulting in actionable insights for business stakeholders.• Developed serverless functions using AWS Lambda to process data events in real time, reducing operational costs by 30%.• Enhanced 12 Databricks pipelines to ingest, process, and transform 500GB of data from 3 disparate sources into a data lake, improving data processing efficiency by 35%.• Orchestrated data pipelines with Apache Airflow, automating workflows and reducing manual intervention by 70%.• Optimized SQL queries in AWS Athena to support complex analytical reporting, enabling data-driven decision-making

• Created HiveQL queries that identified a 30% increase in customer churn rate, allowing for targeted customer retention campaigns.• Built and managed complex data pipelines using Azure Data Factory, automating data ingestion, transformation, and delivery processes.• Leveraged PySpark for large-scale data processing and analytics tasks within the Apache Spark ecosystem and utilized DataFrames and SQL functionalities for efficient data manipulation.• Used Kafka to stream clickstream data from a website to a data lake, allowing for near real-time analysis of user behavior and website optimization.• Built and optimized Apache Spark clusters on Azure Databricks to process 500 TB data, accelerating data processing.