• Experiencing in managing teams of data engineers and providing technical guidance and mentorship and leading the team.• Loaded data into the cluster from dynamically generated files using Flume and from relational database management systems using Sqoop.• Knowledgeable in managing data engineering projects, including project planning, project execution, and project monitoring.• Experiencing in providing technical leadership and guidance to data engineering teams and leading the team.• Used Flume to collect, aggregate and store the web log data onto HDFS.• Performed Data Cleansing using Python and loaded into the target tables.• Experienced in setting up cloud infrastructure for CI/CD pipelines using cloud formation templates or Terraform.• Logical implementation and interaction with HBASE.• Used Scala to store streaming data to HDFS and to implement Spark for faster processing of data.• Integrating user data from Cassandra to HDFS. Integrating Cassandra with Storm for real time user attributes look up.• Experienced in automating testing and deployment processes using tools like Selenium, Appium, or TestComplete.• Experienced in implementing data security measures, including data encryption, access control, and authentication of Data Infrastructure.• Performed Sqoop Incremental imports by using Oozie based on every day.• Installed and configured Hadoop MapReduce, HDFS, developed MapReduce jobs in Java for data cleaning and pre-processing.• Involved in using HCATALOG to access Hive table metadata from MapReduce or Pig code.• Created Pig scripts to transform the HDFS data and loaded the data into Hive external table.

• Extract data from multiple sources, integrate disparate data into a common data model, and integrate data into a target database, application, or file using efficient programming processes• Document, and test moderate data systems that bring together data from disparate sources, making it available to data scientists, and other users using scripting and/or programming languages• Write and refine code to ensure performance and reliability of data extraction and processing• Participate in requirements gathering sessions with business and technical staff to distill technical requirement from business requests• Develop SQL queries to extract data for analysis and model construction• Own delivery of moderately sized data engineering projects• Define and implement integrated data models, allowing integration of data from multiple sources• Design and develop scalable, efficient data pipeline processes to handle data ingestion, cleansing, transformation, integration, and validation required to provide access to prepared data sets to analysts and data scientists• Ensure performance and reliability of data processes• Define and implement data stores based on system requirements and consumer requirements• Document and test data processes including performance of through data validation and verification• Collaborate with cross functional team to resolve data quality and operational issues and ensure timely delivery of products• Develop and implement scripts for database and data process maintenance, monitoring, and performance tuning• Analyze and evaluate databases in order to identify and recommend improvements and optimization• Design eye-catching visualizations to convey information to users

Hadoop development and implementation (Environment - HDFS, Hbase, Spark, Kafka, Ozie, Scoop, Flume, Kerberos, Oracle ASO, MySQL)• Loading from disparate data sets using Hadoop stack of ingestion and workflow tools• Pre-processing using Hive and Pig.• Designing, building, installing, configuring and supporting Hadoop.• Translate complex functional and technical requirements into detailed design.• Perform analysis of vast data stores and uncover insights.• Maintain security and data privacy.• Managing and deploying HBase.• Being a part of a POC effort to help build new Hadoop clusters.• Test prototypes and oversee handover to operational teams.• Propose best practices/standards.• Configure and implementation of Data Marts in Hadoop platform• Involved in loading data from Teradata, Oracle database into HDFS using Sqoop queries. • Worked on setting up Kafka for streaming data and monitoring for the Kafka Cluster.• Responsible for importing log files from various sources into HDFS using Flume. • Imported data using Sqoop to load data from MySQL to HDFS on regular basis. • Worked on shell scripting in Linux and the Cluster. Used shell scripts to run hive queries from beeline.• Developed Scripts and automated data management from end to end and sync up between all the clusters. • Worked with Hue GUI in scheduling jobs with ease and File browsing, Job browsing, Metastore management.• Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and processing with Sqoop and Hive.• Developed Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.• Created Partitions, Buckets based on State to further process using Bucket based Hive joins.• Loaded multiple NOSQL databases including MongoDB, PostgreSQL, Couchbase, HBase and Cassandra.

• Involved in loading data from Teradata, Oracle database into HDFS using Sqoop queries. • Responsible for building scalable distributed data solutions using Hadoop. • Developed multiple Map Reduce jobs in java for data cleaning and preprocessing. • Developed Map Reduce pipeline jobs to process the data and create necessary HFiles. • Involved in loading the created HFiles into Hbase for faster access of large customer base without taking Performance hit. • Worked on setting up Kafka for streaming data and monitoring for the Kafka Cluster.• Worked in AWS environment for development and deployment of Custom Hadoop Applications.• Involved in creation and designing of data ingest pipelines using technologies such as Apache Strom and Kafka.• Developed Spark scripts by using Scala shell commands as per the requirement.• Implemented discretization and binning, data wrangling: cleaning, transforming, merging and reshaping data frames using Python.• Created Hbase tables to store various data formats of PII data coming from different portfolios. • Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis • Involved in managing and reviewing Hadoop log files. • Responsible to manage data coming from different sources. • Involved in creating Pig tables, loading with data and writing Pig Latin queries which will run internally in Map Reduce way. • Experienced in Using Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.• Transferred the data using Informatica tool from AWS S3 to AWS Redshift. Involved in file movements between HDFS and AWS S3.• Create a complete processing engine, based on Hortonworks’ distribution, enhanced to performance. • Provide batch processing solution to certain unstructured and large volume of data by using Hadoop Map Reduce framework.

• Responsible to manage data coming from different sources and involved in HDFS maintenance and loading of structured and unstructured data. • Developed data pipeline using Flume, Sqoop, Pigand MapReduce to ingest behavioral data into HDFS for analysis. • Responsible for importing log files from various sources into HDFS using Flume. • Imported data using Sqoop to load data from MySQL to HDFS on regular basis. • Extracted files from MongoDB through Sqoop and placed in HDFS and processed. • Created customized BI tool for manager team that perform Query analytics using HiveQL.• Created Partitions, Buckets based on State to further process using Bucket based Hive joins. • Experienced in using Kafka as a data pipeline between JMS and Spark Streaming Applications.• Created storage with Amazon S3 for storing data. Worked on transferring data from Kafka topic into AWS S3 storage.• Worked on python files to load the data from csv, json, mysql, hive files to Neo4j Graphical database.• Estimated the hardware requirements for Name Node and Data Nodes & planning the cluster.• Created Hive Generic UDF's, UDAF's, UDTF's in java to process business logic that varies based on policy. • Moved Relational Database data using Sqoop into Hive Dynamic partition tables using staging tables. • Consolidating customer data from Lending, Insurance, Trading and Billing systems into data warehouse and mart subsequently for business intelligence reporting.• Optimizing the Hive queries using Partitioning and Bucketing techniques, for controlling the data distribution. • Experienced on Loading streaming data into HDFS using Kafka messaging system.• Used the Spark -Cassandra Connector to load data to and from Cassandra.• Worked with NoSQL database Hbase to create tables and store data.• Proficient in querying Hbase using Impala.• Worked on custom Pig Loaders and storage classes to work with variety of data formats such as JSON and XML file formats.

• Analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase and Sqoop. • Creating multiple MapReduce jobs in Pig and Hive for data cleaning and pre-processing. • Successfully loading files to Hive and HDFS from Oracle, SQL Server using Sqoop. • Writing Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data. • Creating Hive tables, loading with data and writing Hive queries. • Involved in Spark for fast processing of data. Defining job flows. • Using Hive to analyze the partitioned data and compute various metrics for reporting. • Moved data from HDFS to Cassandra using MapReduce and BulkOutputFormat class.• Managing and reviewing theHadoop log files. • Using Pig as ETL tool to do Transformations, even joins and some pre-aggregations. • Unit testing and delivered Unit test plans and results documents. • Exporting data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose. • Worked on Oozie workflow engine for job scheduling.

• Involved in the process Design, Coding and Testing phases of the software development cycle.• Designed use-case, sequence and class diagram (UML).• Developed rich web user interfaces using JavaScript (pre-developed library).• Created modules in Java and C++, python.• Developed JSP pages with Struts framework, Custom tags and JSTL.• Developed Servlets, JSP pages, Beans, JavaScript and worked on integration.• Developed SOAP/WSDL interface to exchange usage and Image and terrain information from Geomaps.• Developed Unit test cases for the classes using JUnit.• Developed stored procedures to extract data from Oracle database.• Developed and maintained Ant Scripts for the build purposes on testing and production environments.• Designed and developed user interface components using AJAX, JQuery, JSON, JSP, JSTL & Custom Tag library.• Involved in building and parsing XML documents using SAX parser.• Application developed with strict adherence to J2EE best practices.

• Involved in the design and development of the entire application.• Created UML diagrams (use case, class, sequence, and collaboration) based on the business requirements • Involved in designing and developing dynamic web pages using HTML and JSP with Struts tag libraries • Used JavaScript for client-side validation and Struts Validator Framework for form validations • Implemented PL/SQL queries and used Oracle stored procedures, and built-in functions to retrieve and update data from the databases • Used Hibernate Framework for object relational mapping and persistence. • Interfaced with the Oracle back-end using Hibernate Framework and XML config files. • Used Eclipse for the development environment• Wrote JUnit test cases to test the functionality of each method in the DAO layer.• Used CVS for version control.• Configured and deployed the web application using WebSphere Application Server • Used Rational ClearQuest.• Prepared technical reports and documentation manuals for efficient program development..