• Designed and implemented ETL pipelines using Apache NiFi and Kafka Connect for real-time data ingestion, transformation, and loading into the data mart.• Orchestrated workflows with Apache Airflow, integrating Kafka/Kafka Connect components for 99.9% data reliability and reducing operational by 90%.• Collaborated with cross-functional teams to understand business requirements, resulting in more accurate data engineering solutions. • Utilized Apache Spark and Scala for large-scale data processing (500GB to 1TB), achieving a 40% improvement in processing efficiency and a 60% reduction in processing time.• Conducted comprehensive data quality assessments and validation processes, ensuring a 99.99% data accuracy and consistency rate.• Collaborated with the DevOps team to streamline and automate deployment processes, employing Docker and Kubernetes for scalable and efficient application deployment.• Support team members to implement APIs/UI for reconciliation tool serving back-office users.