Abhinav G Email and Phone Number

Involved in all phases of Software Development Life Cycle (SDLC) such as requirements gathering, modelling, analysis, design, development, and testing. Developed and deployed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple le formats for analyzing & transforming the data to uncover insights into the customer usage patterns Developed HIVE UDFs to incorporate external business logic into Hive script and developed join data set scripts using HIVE join operations. Created various Hive external tables, staging tables and joined the tables as per the requirement. Implemented static Partitioning, Dynamic partitioning and Bucketing. Worked with the Spark for optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, PySpark, Pair RDD's, and Spark YARN. Involved in migrating the client Datawarehouse architecture from on-premises into MS Azure cloud. Used Spark for interactive queries, processing of streaming data and integration with popular NoSQL database for huge volume of data. Worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi-structured data coming from various sources. Created pipelines in ADF using linked services to extract, transform and load data from multiple sources like Azure SQL, Blob storage and Azure SQL Data warehouse. Implemented Cluster for NoSQL tool HBase as a part of POC to address HBase limitations. Used Spark Data Frames Operations to perform required Validations in the data and to perform analytics on the Hive data. Developed Apache Spark applications by using Spark for data processing from various streaming sources. Design and implement end-to-end data solutions (storage, integration, processing, and visualization) in Azure. Worked on architecture and components of Spark, and efficient in working with Spark Core, Spark SQL.

Analyzed stories, participated in grooming sessions and point estimation for development according to agile methodology. Developed Spark jobs using Scala and Python on top of Yarn/ MRv2 for interactive and Batch Analysis. Worked with the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, PySpark, Pair RDD's, Spark YARN, and Spark MLLib. Involved in implementation of data movements from on-premises to cloud in MS Azure. Evaluated the performance of Apache Spark in analyzing genomic data. Developed a NIFI Workflow to pick up the data from SFTP server and send that to Kafka broker. Used Python - NumPy, SciPy, Pandas, NLTK, Matplotlib, Beautiful Soup, and TextBlob to finish the ETL process of clinical data for future NLP analysis. Migrated data from traditional database systems to Azure SQL databases. Used different Machine Learning modules for ruining the spark jobs on daily and weekly basis. Migrated complex Map reduce programs into Spark RDD transformations, actions. Designed and developed Oozie workflows to orchestrate Hive scripts, Sqoop. Involved in developing the Spark Framework to provide structure to data on the fly and process the data using Spark core API's, Data Frame, Spark-SQL and Scala Evaluated and improved application performance with Spark. Recreated existing application logic and functionality in the Azure Data Lake, Data Factory, and Azure SQL Database. Developed Spark scripts to import large files from Azure and imported the data from different sources like HDFS/HBase into Spark RDD. Extensively worked on Python and build the custom ingest framework. Design, build and deliver the operational and management tools, frameworks and processes for the Azure Data Lake and drive the implementations into the Azure Data Lake Cloud Operations team.

Administered, maintained, provisioned, patched, and maintained Cloudera Hadoop clusters on Linux. Worked on analyzing Hadoop stack and different big data analytic tools including Pig and Hive, HBase database, and Sqoop. Utilized Python matplotlib and SciKit-Learn modules to generate the basic prototype visualization restored using visualization tools such as Tableau. Written multiple MapReduce programs to extract data for extraction, transformation, and aggregation from more than 20 sources having multiple file-formats including XML, JSON, CSV & other compressed file formats. Worked in AWS environment for development and deployment of Custom Hadoop Applications. Created Oozie workflows for Hadoop-based jobs including Sqoop, Hive, and Pig. Involved in file movements between HDFS and AWS S3. Created Hive External tables and loaded the data into tables and query data using HQL. Performed data validation on the data ingested using MapReduce by building a custom model to filter all the invalid data and cleanse the data. Handled the importing of data from various data sources, performed transformations using Hive, and Map-Reduce, loaded data into HDFS, and extracted data from MySQL into HDFS using Sqoop. Transferred the data using the Informatica tool from AWS S3 to AWS Redshift. Wrote HiveQL queries by configuring a number of reducers and mappers in the query needed for the output. Transferred data between Pig Scripts and Hive using HCatalog, transferred relational database using Sqoop. Involved in Extract Transfer and Load (ETL) process using SSIS and generated reports using SSRS. Analyzed the data by performing Hive queries (HiveQL) and running Pig Scripts (Pig Latin). Cluster coordination services through Zookeeper. Installed and configured Hive and written Hive UDFs.

Worked with BI team in gathering the report requirements and Sqoop to export data into HDFS and Hive. Involved in the below phases of Analytics using R, Python, and Jupyter Notebook. Data collection and treatment: Analyzed existing internal data and external data, worked on entry errors, classification errors and defined criteria for missing values. Data Mining: Used cluster analysis for identifying customer segments, Decision trees used for profitable and non-profitable customers, Market Basket Analysis used for customer purchasing behavior and part/ product association. Developed multiple Map Reduce jobs in Java for data cleaning and pre-processing. Assisted with data capacity planning and node forecasting. Installed, Configured and managed Flume Infrastructure. Administrator for Pig, Hive and HBase installing updates patches and upgrades. Worked closely with the claims processing team to obtain patterns in filing of fraudulent claims. Developed Map Reduce programs to extract and transform the data sets and results exported back to RDBMS using Sqoop. Patterns observed in fraudulent claims using text mining in R and Hive. Exported the data required information to RDBMS using Sqoop to make the data available for the claims processing team to assist in processing a claim based on the data. Developed Map Reduce programs to parse the raw data, populate staging tables and store the refined data in partitioned tables in the EDW. Created tables in Hive and loaded the structured (resulted from Map Reduce jobs) data Using Hive QL developed many queries and extracted the required information. Created Hive queries that helped market analysts spot emerging trends by comparing fresh data with EDW reference tables and historical metrics. Responsible for importing the data (mostly log files) from various sources into HDFS using Flume.