Sunil S Email & Phone Number

• Involved in importing and exporting data between Hadoop Data Lake and Relational Systems like Oracle, MySQL using Sqoop.• Involved in developing spark applications to perform ELT kind of operations on the data.• Modified existing MapReduce jobs to Spark transformations and actions by utilizing Spark RDDs, Data frames, and Spark SQL API’s• Create and maintain optimal data pipeline architecture in cloud Microsoft Azure using Data Factory and Azure Databricks• Create an Architectural solutions that leverages the best Azure analytics tools to solve our specific need in Chevron use case• Design and present technical solutions to end users in a way the is easy to understand and buy into• Educating client/business users on the pros and cons of various Azure PaaS and SaaS solutions ensuring the most cost-effective approaches are taken into consideration.• Written Templates for Azure Infrastructure as Code using Terraform to build staging and production environments.• Utilized Hive partitioning, Bucketing and performed various kinds of joins on Hive tables• Involved in creating Hive external tables to perform ETL on data that is produced on daily basis.• Design and developed services to persist and rea data from Hadoop, HDFC, hive and writing Java based MapReduce batch jobs using Hortonworks Hadoop data platform.• Designed and deployed data pipelines using Data Lake, Databricks, and Apache Airflow.• Install and configure Apache Airflow for S3 bucket and snowflake data warehouse and created dags to run the Airflow.• Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.• User Profile and other unstructured data storage using Java and MongoDB

Us

• Developed Spark Applications by using Python and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.• Installed Kafka on Hadoop Cluster and Configured producer and consumer in java to establish connection from source to HDFS with Popular hash tags.• Worked with Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD's, Spark YARN.• Performed tuning of Spark Applications to set batch interval time and correct level of Parallelism and memory tuning. • Used Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data from Kafka in real time and persist it to Cassandra.• Scheduling Spark/Scala jobs using Oozie workflow in Hadoop Cluster and generated detailed design documentation for the source-to-target transformations.• Developed Kafka consumer’s API in python for consuming data from Kafka topics.• Developer user defined functions in Java if required for hive queries.• Used Kafka to consume XML messages and Spark Streaming to process the XML file to capture UI updates.• Valuable experience on practical implementation of cloud-specific technologies including IAM, Amazon Cloud Services like Elastic Compute Cloud (EC2), ElastiCache, Simple Storage Services (S3), Cloud Formation, Virtual Private Cloud (VPC), Route 53, Lambda, Glue, EMR.

• Involved in importing data from Microsoft SQLserver, MySQL, Teradata. into HDFS using Sqoop. • Installed and configured Hadoop Map reduce, HDFS, developer multiple Map Reduce jobs in Java and Scala for data cleaning and preprocessing.• Developed workflow in Oozie to automate the tasks of loading the data into HDFS. • Used Hive to analyze the partitioned and bucked data to compute various metrics of reporting. • Involved in creating Hive tables loading data, and writing queries that will run internally in MapReduce• Involved in creating Hive External tables for HDFS data. • Solved performance issues in Hive and Pig Scripts with understanding of Joins, Group and Aggregation and perform the MapReduce jobs. • Used Spark for transformations, event joins and some aggregations before storing the data into HDFS.• Troubleshoot and resolve data quality issues and maintain elevated level of data accuracy in the data being reported. • Analyze the large amount of data sets to determine optimal way to aggregate. • Worked on the Oozie workflow to run multiple Hive and Pig jobs. • Worked on creating Custom Hive UDF's. • Developed automated shell script to execute Hive Queries.