Hadoop Adminstrator
Responsible for architecting Hadoop environment• Responsible for implementation and ongoing administration of Hadoop infrastructure • Installed/Configured/Maintained Apache Hadoop clusters for application development and Hadoop tools like Hive, Pig, HBase, Zookeeper and Sqoop. • 24x7 REMOTE MANAGEMENT through Ganglia • Managing the configuration of the cluster to the meet the needs of analysis whether I/O bound or CPU bound • Having knowledge on documenting processes, server diagrams, preparing server requisition documents • Working on setting up two 5 nodes development clusters and 10 node staging cluster using MapR M5 • Experienced in dealing with MapR support team, bug reporting, resolving configuration and other MapR related issues • Responsible for MapR sales team to discuss MapR architecture and setting up and documenting best practices for Comcast • Working on setting up 100 node production cluster and a 40 node backup cluster at two different data centers • Working on setting up security processes, CADA authentication, user quota management, data management • Involved in setting up hive, hiveserver2, hive authorization and testing the environment • Involved in setting up hbase, oozie, zookeeper • Wrote shell scripts for rolling day-to-day processes and it is automated. • Responsible for onboarding new customers, walking them through the platform • Having knowledge on developing MapReduce scripts with Ruby. • Perform administering Hadoop environment, managing jobs, monitoring cluster performance using MapR control center, Nagios and customized shell scripts, generating reports • Responsible for automatic MapR tasks (adding, deleting volumes, managing users etc.), managing MapFS, NFSMounts through shell scripts Consulting on Hadoop ecosystem: Hadoop, MapReduce, Hbase, Sqoop, Amazon Elastic Map Reduce (EMR)