Responsible for implementation, administration and management of Hadoop infrastructuresEvaluation of Hadoop infrastructure requirements and design/deploy solutions (high availability, big data clusters and involved in cluster monitoring and troubleshooting Hadoop issuesWorked with application teams to install OSs and Hadoop updates, patches, version upgrades as requiredHelped maintain and troubleshoot UNIX and Linux environmentAnalyzed and evaluated system security threats and safeguardsDeveloped Pig program for loading and filtering the streaming data into HDFS using Flume.Experienced in handling data from different datasets, join them and preprocess using Pig join operations.Developed HBase data model on top of HDFS data to perform real time analytics using Java API.Developed different kinds of custom filters and handled predefined filters on HBase data using API.Imported and exported data from Teradata to HDFS and vice-versa.Strong understanding of Hadoop eco system such as HDFS, MapReduce, HBase, Zookeeper, Pig, Hadoop streaming, Sqoop, Oozie and HiveImplemented Secondary sorting to sort reducer output globally in map reduce.Implemented data pipeline by chaining multiple mappers by using Chained Mapper.Created Hive Dynamic partitions to load time series dataHandled different types of joins in Hive like Map joins, bucker map joins,sorted bucket map joins.Created tables, partitions, buckets and performed analytics using Hive ad-hoc queries.Experienced import/export data into HDFS/Hive from relational database and Tera data using Sqoop.Handled continuous streaming data from different sources using Flume and set destination as HDFS.Integrated spring schedulers with Oozie client as beans to handle corn jobs.Experience with CDH distribution and Cloudera Manager to manage and monitor Hadoop clusters

Involved in complete project life cycle starting from design discussion to production deployment Worked closely with the business team to gather their requirements and new support features Involved in running POC's on different use cases of the application and maintained a standard document for best coding practices Developed a 200-node cluster in designing the Data Lake with the Hortonworks distribution Responsible for building scalable distributed data solutions using HadoopInstalled, configured and implemented high availability Hadoop Clusters with required services (HDFS, Hive, HBase, Spark, Zookeeper) Implemented Kerberos for authenticating all the services in Hadoop Cluster Responsible for installation and configuration of Hive, Pig, HBase and Sqoop on the Hadoop cluster and created hive tables to store the processed results in a tabular format. Configured Spark Streaming to receive real time data from the Apache Kafka and store the stream data to HDFS using Scala. Developed the Sqoop scripts to make the interaction between Hive and Vertica Database. Processed data into HDFS by developing solutions and analyzed the data using Map Reduce, PIG, and Hive to produce summary results from Hadoop to downstream systems. Build servers using AWS: Importing volumes, launching EC2, creating security groups, auto-scaling, load balancers, Route 53, SES and SNS in the defined virtual private connection. Written Map Reduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-Hive Integration. Streamed AWS log group into Lambda function to create service now incident. Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries and Pig scripts. Created Managed tables and External tables in Hive and loaded data from HDFS.