Analyzed SQL scripts and designed the solution to implement using PySpark.Worked in Spark streaming to get ongoing information from Kafka and store thestream information to HDFS.Developed and Configured Kafka brokers to pipeline server logs data into SparkStreaming.Worked on MySQL database to retrieve information from storage using Python.Experienced in implementing and working on the python code using shell scripting.Developed Python code to gather the data from HBase (Cornerstone) and designs thesolution to implement using PySpark.Installed and configured Hadoop MapReduce, HDFS, HIVE, PIG, SQOOP, Flume,Oozie on the Hadoop cluster.Developed and analyzed the SQL scripts and designed the solution to implement usingPyspark.Leveraged cloud and GPU computing technologies for automated machine learning andanalytics pipelines, such as AWS, and GCP.Developed and deployed data pipeline in clouds such as AWS and GCPUsing Hive join queries to join multiple tables of a source system and load them to Elasticsearch tables.

Involved in installation, configuration, and maintenance of Hadoop clusters for applicationdevelopment with Cloudera distribution.Developed Kafka consumer’s API in Scala for consuming data from Kafka topics.Developed end-to-end scalable distributed data pipelines which receive data using distributedmessaging systems Kafka through the persistence of data into HDFS with Apache Spark usingScala.Experienced in query data using Spark SQL on Spark to implement Spark RDD’S in Scala.Experienced in working with different scripting technologies like Python, and UNIX shellscripts.Worked on Partitioning, Bucketing, Parallel execution, Map side Joins for optimization ofnecessary hive queries.Performed Hive QL to create Hive tables and to write Hive queries to perform the dataanalysis.Experience in implementing Spark RDD transformations, actions, data frames, case classes torequired data by using Spark core.Migrated the computational code in HQL to PySpark.Completed data extraction, aggregation, and analysis in HDFS by using PySpark and store thedata needed in Hive.

Developed Hive Scripts, Hive UDFs, and Python Scripting and used Spark (Spark-SQL,Spark-shell) to process data in Hortonworks.Performed advanced procedures like text analytics and processing using the in-memorycomputing capabilities of Spark.Designed and Developed Scala code for data pull from cloud-based systems and applyingtransformations on it.Usage of Sqoop to import data into HDFS from MySQL database and vice-versa.Implemented optimized joins to perform analysis on different data sets usingMapReduce programs.Implemented Partitioning, Dynamic Partitions, and Buckets in HIVE & Impala forefficient data access.Worked in an Agile environment, and used rally tool to maintain the user stories and tasks.Extensively worked on HiveQL, join operations, writing custom UDF's and having goodexperience in optimizing Hive Queries.Experienced in running queries using Impala and used BI tools and reporting tool(tableau) to run ad-hoc queries directly on Hadoop.Developed Scala & Python scripts, UDF's using both Data frames/SQL andRDD/MapReduce in Spark-SQL for Data Aggregation queries and writing data backinto RDBMS through Sqoop.