Senior Data Architect
CurrentMigrate large scale data ecosystems and applications from onprem to Amazon EMR. Helped implement a Cloud Data Strategy and migrate critical services like Hive, Hbase, Druid, Spark, Kafka, Oracle etc to Amazon EMR, Amazon MSK, Glue with DynamicFrames to dynamically infer and compute schema on fly, Lambda, Hudi. Developed a Centralized Logging and Monitoring Observability architecture on Amazon Managed Prometheus and Grafana. Implemented data quality and data governanceDeveloped a Infrastructure as Code using Terraform for implementing large scale MSK cluster with 500k/per second cumulative cluster throughput. Helped migrate and modernize streaming applications. The setup was multi-region having high t/p with requirements to keep cluster's in sync across region using Kafka MirrorMaker2 on Amazon EKS, and implemented LinkedIn CruiseControl for Kafka Cluster Management and Rebalancing. Implemented Amazon MSK with Tiered Storage for cost optimization and savings.