Data Engineer
CurrentI have extensive experience in data engineering, focusing on building and optimizing ETL pipelines for efficient data processing using Apache Spark, Kafka, and Flink for both batch and real-time data streaming, while using Kafka Connect for seamless data integration. My workflow orchestration skills include Python and Apache Airflow, enabling automated, scalable ETL processes. I manage and optimize cloud infrastructure on AWS and GCP for scalable, secure solutions.In software engineering, I design APIs for high throughput and reliability, leveraging ZIO in Scala and Spring in Kotlin. My SQL expertise includes AWS Glue, Athena, and complex querying in Amazon Redshift. For observability, I use Prometheus, Grafana, and PagerDuty to ensure real-time monitoring and incident management.My database experience covers PostgreSQL, DynamoDB, and Aerospike for a range of structured and NoSQL data needs. Additionally, I am skilled in Docker, Kubernetes, and CI/CD processes and have explored areas like Machine Learning, Elasticsearch, Snowflake, and Rust through projects and courses.