Staff Data Engineer
CurrentDesigned and implemented data pipelines for data ingestion, transformation, and loading into data warehouses and data lakes.Developed and maintained streaming data pipelines using technologies like Kafka and Spark Streaming to ensure real-time data availability.Built and optimized data lakehouses on Databricks, enhancing data accessibility and query performance.Automated data processes to reduce manual effort and improve efficiency, freeing up valuable time for more strategic initiatives.Collaborated closely with data analysts and business stakeholders to understand data requirements and translate them into effective data solutions.Successes:Successfully delivered a critical data pipeline project that reduced data processing time by 57%, leading to faster insights, improved decision-making.Implemented a robust data streaming solution that delivers real-time data feeds to downstream applications with 20% faster than existing solution.Automated data quality checks, ETL tasks resulting in a 50% reduction in manual effort, 20% increase in data processing throughput.
