Big Data Architect
CurrentTechnologies used: MapR-FS, Hadoop, Drill, Spark, Hbase, MapR-DB, Python, Java, Yarn, Ansible, Apache NiFi, Apache KafkaVaultLed the engineering effort on the Vault platform (MapR hadoop cluster)Cleaned the tooling used by platform to migrate from legacy mechanisms, like crontab and emails, to automated frameworks like Zapp (Morgan Stanley scheduler), Watchtower (alerting system)Introduced new Hadoop ecosystem tools like Apache Drill, Spark, Hbase and MapR-DB to improve the platform functionalitiesRedesigned the configuration management tooling using AnsibleWorking on the handover of the Operating System support to the Unix Operations teamReal time Ingestion LayerDesigned and implemented a real time ingestion layer using Apache Nifi and Apache KafkaUsed Python as the tool to automate installations and deploymentOnboarded all the IT Security real time feeds to the new platformWorked closely with the application owners consuming the real time data from Apache KafkaScaled the platform to handle large amount of real time dataWorked on the Splunk to Kafka integration