• Developed custom data Ingestion adapters to extract the log data and click stream data from external systems and load into HDFS by making use of Spark/Scala.• Developed the Pysprk code for AWS Glue jobs and for EMR.• Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution.• Developed spark program using scala API’s to compare the performance of spark with hql.• Implemented spark using Scala and SparkSql.• Creating Hive tables, loading data and writing hive queries for building Analytical Datasets.• Worked on real time data ingestion and processing using Spark Streaming, and HBase.• Used Spark and Spark-SQL to read the data and create the tables in hive using the Scala API.• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.• Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.• Worked with Play framework and Akka parallel processing.• Work with spark eco system using Spark SQL and Scala queries on different formats like Text file, CSV file.• Developed Kafka producer and Spark Streaming consumer to read the stream of events as per business rules.• Designed and developed Job flows using TWS.• Developed Sqoop commands to pull the data from Teradata.• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.• Used AVRO, Parquet File formats and Snappy compression through the project.• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.• Configured Spark streaming to get ongoing information from the source and store the stream information to HDFS.

• Worked on AWS pipelines for data migration to create and monitor multiple instances such as EC2, S3, AWS lambda, step function, EMR, spark, and Hadoop Development.• Design, development, and implementation of ETL solutions on AWS and in the Big Data environment, as well as migration of existing objects from on-premises, Redshift, and HDFS to S3 Data Lake and Snowflake.• Developed reusable framework to be leveraged for future migrations that automates ETL from RDBMS systems to the Data Lake utilizing Spark Data Sources and Hive data objects.• Used SQL, NumPy, Pandas, Boto3, and Hive for data analysis and model building• Created Python scripts in Spark for data aggregation, queries, and writing data back into OLTP (Online Transactional processing) system using Data frames/SQL/Data sets and RDD/MapReduce.• Involved in design and analysis of the issues and providing solutions and workarounds to the users and end-clients.• Used Python to create multiple Spark Streaming and Spark SQL jobs on AWS.• Created a job monitoring tool to monitor jobs scheduled using Data Pipeline, Step Functions, and EMR.• Created a Data Pipeline utilizing Processor Groups and numerous processors in Apache Nifi for Flat File, RDBMS as part of a Proof of Concept (POC) on Amazon EC2.• Provisioning of EC2 instances, as well as transient and long-running EMRs (Elastic Map Reduce) to handle petabytes of data• Configure access to RDS DB services, DynamoDB tables, and EBS volumes for inbound and outbound traffic to set alarms for notifications or automated actions on AWS.• Developed a data pipeline for various events such as ingestion, aggregation, and loading consumer response data from an AWS S3 bucket into Hive external tables in HDFS to feed tableau dashboards.• Wrote scripts to collect high-frequency log data from various sources and integrate it into AWS using Kinesis, staging data in the Data Lake for later analysis.

• Developed custom data Ingestion adapters to extract the log data and click stream data from external systems and load into HDFS by making use of Spark/Scala.• Developed the Pysprk code for AWS Glue jobs and for EMR.• Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution.• Developed spark program using scala API’s to compare the performance of spark with hql.• Implemented spark using Scala and SparkSql.• Creating Hive tables, loading data and writing hive queries for building Analytical Datasets.• Worked on real time data ingestion and processing using Spark Streaming, and HBase.• Used Spark and Spark-SQL to read the data and create the tables in hive using the Scala API.• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.• Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.• Worked with Play framework and Akka parallel processing.• Work with spark eco system using Spark SQL and Scala queries on different formats like Text file, CSV file.• Developed Kafka producer and Spark Streaming consumer to read the stream of events as per business rules.• Designed and developed Job flows using TWS.• Developed Sqoop commands to pull the data from Teradata.• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.• Used AVRO, Parquet File formats and Snappy compression through the project.• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.• Configured Spark streaming to get ongoing information from the source and store the stream information to HDFS.• Used various spark Transformations and Actions for cleansing the input data.• Developed shell scripts to generate the hive create statements from the data and load the data into the table.• Optimized Hive QL/ pig scripts by using execution engine like Tez, Spark.

• Involved in the entire project life cycle, from design discussions to production deployment.• Good Knowledge of monitoring, managing, and reviewing Hadoop clusters using Cloudera Manager.• Expertise in Cluster Analysis using various big data analytic tools such as MapReduce and Hive• Supported the Hadoop Architect team in developing a Database Design in HDFS using HBase Architecture Design.• Used Scala to create Spark applications to ease the transition to Hadoop.• Developed scripts and batch jobs for scheduling various Hadoop programs and was involved in the maintenance and review of Hadoop log files.• Used Sqoop for data import and export from RDBMS to HDFS• Extracted the required data from the server into HDFS and Bulk Loaded cleaned data into HBase.• Built a data pipeline for a compliance report proof-of-principle consisting of Sqoop, Hadoop (HDFS), SparkSQL, Scala, Elasticsearch, and Kibana.• The open-source Python web scraping framework was used to crawl and extract data from web pages, and the conversion was performed using Hadoop, Hive, and MapReduce.• Created Airflow Scheduling scripts in Python.• Used Apache Spark to ingest Kafka data. Loading and transforming large volumes of structured, semi-structured, and unstructured data was done.• Expert in Oozie, and Data pipeline operational services to coordinate clusters and plan workflows.• Used Cloudera Manager to continuously monitor and manage the Hadoop cluster.• Mappings done with reusable components such as worklets and mapplets, as well as other transformations.

• Installed and configured Hadoop MapReduce HDFS Developed multiple MapReduce jobs in java for data cleaning and preprocessing.• Experience in installing configuring and using Hadoop ecosystem components.• Importing and exporting data into HDFS and Hive using Sqoop.• Experienced in defining job flows.• Knowledge in performance troubleshooting and tuning Hadoop clusters.• Experienced in managing and reviewing Hadoop log files.• Participated in development/implementation of Cloudera Hadoop environment.• Load and transform large sets of structured semi structured and unstructured data.• Responsible to manage data coming from different sources.• Got good experience with NOSQL database.• Supported Map Reduce Programs those are running on the cluster.• Involved in loading data from UNIX file system to HDFS.• Installed and configured Hive and also written Hive UDFs.• Involved in creating Hive tables loading with data and writing hive queries which will run internally in map reduce way.• Worked on installing cluster commissioning decommissioning of datanode namenode recovery capacity planning and slots configuration.• Created HBase tables to store variable data formats of PII data coming from different portfolios.• Implemented best income logic using Pig scripts.• Load and transform large sets of structured semi structured and unstructured data• Cluster coordination services through Zookeeper.• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.• Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.• Managing the version control for the deliverables by streamlining and re-basing the development streams of the SVN.Environment: Hadoop, MapReduce, HDFS, Hive, Java, SQL, PIG, Zookeeper, Python and Sqoop, Java/JDK, XML, Web Services, SVN, JUnit, Log4J, Windows, Oracle.