Senior Big Data Engineer
• Engineered and launched a comprehensive big data platform with Scala, Spring Boot, and Apache Spark, facilitating targeted marketing initiatives and processing over 10 terabytes of data daily, resulting in a 25% increase in campaign effectiveness.• Executed data profiling and matching utilizing AWS tools (SageMaker, Lambda, EMR, S3, Kinesis, RDS, EC2) and Elasticsearch, reducing data retrieval times by 40%.• Orchestrated and optimized data pipelines with Apache Spark and Kafka, ensuring seamless data ingestion, processing, and streaming with high efficiency and minimal latency.• Built and sustained a microservices infrastructure using Docker and Kubernetes, achieving 99.99% service availability and reducing infrastructure costs by 15%.• Championed technical standards through BDD and TDD methodologies, enhancing code quality and system reliability.• Upgraded CI/CD processes with GitLab and SonarQube, decreasing deployment times and boosting release frequency.• Led UX and frontend development efforts, collaborating with cross-functional teams to enhance user experience, leading to a 20% increase in customer satisfaction scores.• Headed DevOps projects and managed infrastructure automation with CloudFormation, ensuring scalable and maintainable environments.• Guided and coached a team of developers and contractors, performing code reviews, leading design sessions, and managing daily tasks, promoting continuous learning and improvement.
