• Developed and maintained various DBT models and macros, focusing on incremental model design and modularization, resulting in a 40% increase in data pipeline efficiency and a 25% reduction in pipeline execution time.• Applied AWS Lambda functions triggered by incoming files in S3 to perform data validation and transformation before loading into Amazon Redshift, enhancing data quality and pipeline efficiency.• Used Amazon Kinesis Firehose to stream real-time data into data lakes for near real-time analytics, enabling faster insights.• Developed and deployed Spark jobs using Python and PySpark, achieving a 10% reduction in processing time compared to traditional MapReduce pipelines.• Created complex SQL queries to extract and transform data from relational databases, increasing the availability of data for analytics

• Accomplished 5 UDFs in Snowflake for complex data cleansing and normalization tasks, reducing data movement by 70% and achieving a 30% improvement in data loading times for downstream analytics.• Designed and implemented data pipelines using technology (Apache Airflow, AWS Glue) to ingest data from various sources (relational databases, APIs) into target systems (data warehouse).• Orchestrated a complex data processing workflow consisting of multiple stages (data extraction, transformation, loading) using AWS Pipeline, improving data pipeline reliability and 40% reduction in data processing errors.• Established and deployed a high-performing ETL pipeline in Databricks to process terabytes of data daily, enabling efficient data integration and transformation for the data warehouse.• Designed and implemented efficient HiveQL queries to analyze large datasets stored in HDFS, achieving a 30% reduction in query execution time compared to traditional methods.• Configured Kafka topics and partitions, achieving a 20% increase in message throughput to meet the data ingestion needs of downstream application

• Used HiveQL queries joining data from an average of 5 sources, enabling analysts to complete complex reports in 20% less time.• Utilized PySpark to clean and enrich data by performing operations like data deduplication, and handling missing values.• Accomplished interactive Power BI reports leveraging advanced DAX and Power Query modeling techniques to unlock deeper data insights and facilitate informed decision-making.• Implemented data pipelines using Apache Airflow on Dataproc to automate data processing tasks.• Increased data accessibility by 30% by migrating legacy data warehouses to BigQuery's serverless architecture