• Installed, configured and maintained Apache Hadoop cluster. • Using the modern cloud and big data architecture, design and develop highly available features.• By utilizing Spark, Hive, Hadoop, Kafka and other supporting technologies built big data solutions.• Used big data tools Spark (Pyspark, SparkSQL) to conduct real-time analysis of the transaction. • Used Sqoop to import data into HDFS/Hive from multiple relational databases.• Generated Tableau dashboards and worksheets for large datasets. • Implemented custom interceptors for Flume to filter data, and defined channel selectors to multiplex the data into different sinks. • Implemented security on Hadoop cluster using Kerberos. • Regulates based on agile planning and designed, implemented and performed testing of solutions.

• Designed and developed ETL solution followed by system and integration testing and defining standards.• Designed and developed Security Framework to provide fine grained access to objects in AWS S3 using AWS Lambda, DynamoDB. • Designed Data Quality Framework to perform schema validation and data profiling on Spark (PySpark).Performed end- to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3.• Worked on MongoDB by using CRUD, Indexing, Replication and Shading features. • Worked on SQL queries and performed data analysis and data profiling using complex SQL queries.• Overall responsibility for the data integration deliverables including data flow diagrams.

• Processed data into HDFS by developing solutions, analyzed the data using MapReduce, Pig, Hive and produce results from Hadoop to downstream systems.• Worked on ingesting data from RDBMS sources like - Oracle, SQL Server and Teradata into HDFS using Sqoop. • Developed real time data ingestion/ analysis using Kafka / Spark-streaming. • Involved in loading data from UNIX file system to HDFS. • Configured Hive and written Hive UDF's and UDAF's. Also, created Static and Dynamic with bucketing as required.• Worked on writing Scala programs using Spark on Yarn for analyzing data. • Worked on BI tools as Tableau to create dashboards and published them to HDFS cluster. • Monitor System health and logs and respond accordingly to any warning or failure conditions.

• Creating end to end Spark applications using Scala to perform various data cleansing, validation, transformation and summarization activities on user behavioral data. • Developed custom Input Adaptor utilizing the HDFS File system API to ingest click stream log files from FTP server to HDFS. • Handled importing other enterprise data from different data sources into HDFS using Sqoop and performing transformations using Hive, Map Reduce and then loading data into HBase tables. • Exported the analyzed data to the relational databases using Sqoop, to further visualize and generate reports for the BI team. • Developed end-to-end data pipeline using FTP Adaptor, Spark, Hive and Impala.• Developed interactive shell scripts for scheduling various data cleansing and data loading process.• Used Oozie and its coordinators to deploy end-to-end data processing pipelines and scheduling the workflows.