Implemented Restful web service to interact with Redis Cache framework. Intake happens through Sqoop, and Ingestion happens through Map Reduce, HBASE. Extensively worked on Spark Streaming and Apache Kafka to fetch live stream data. Interacting with other data scientists and architected custom solutions for data visualization using tools like Tableau, packages in R.Developed predictive models using Python & R to predict customers churn and classification of customers. Documenting the best practices and target approach for CI/CD pipelineConstructed product-usage SDK data and data aggregations by using PySpark, Scala, Spark SQL and Hive context in partitioned Hive external tables maintained in AWS S3 location for reporting, data science dashboarding, and ad-hoc analyses. Involved in data processing using an ETL pipeline orchestrated by AWS Data Pipeline using Hive. Installed Kafka manager for consumer lags and for monitoring Kafka Metrics also this has been used for adding topics, Partitions etc.

Performed ETL on data from different formats like JSON, Parquet.Analyzed the data by performing Hive queries (HiveQL) and running Pig Scripts (Pig Latin). Used Oozie Scheduler systems to automate the pipeline workflow and orchestrate the Spark jobs.Worked in Loading and transforming large sets of structured, semi structured, and unstructured data. Involved in collecting, aggregating, and moving data from servers to HDFS using Flume. Collecting data from various Flume agents that are imported on various servers using Multi-hop Flow. Created Hive UDFs and UDAFs using python scripts & Java code based on the given requirement Automated all the jobs to pull the data and load into Hive tables, using Oozie workflows Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior. Knowledge on microservices architecture in spring Boot integrating with various restful webservices. Created and maintained technical documentation for launching Hadoop Clusters and for executing Pig Scripts. Developed a Python Script to load the CSV files into the S3 buckets and created AWS S3 buckets, performed folder management in each bucket, managed logs and objects within each bucket.

Designed end to end scalable architecture to solve business problems using various Azure Components like HDInsight, Data Factory, Data Lake, Storage and Machine Learning Studio.Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the SQL Activity.Developed Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.Written multiple Hive UDFS using Core Java and OOP concepts and spark functions within Python programs.Wrote Spark applications for Data validation, cleansing, transformations, and custom aggregations. Imported data from various sources into Spark RDD for processing. Developed custom aggregate functions using Spark SQL and performed interactive querying. Worked on installing cluster, commissioning & decommissioning of Data node, Name node high availability, capacity planning, and slots configuration.Cleaned and congruous data was then streamed using Kafka into Spark and manipulations were performed on real time data with Python and Scala.