Gautham R Email and Phone Number

• Extensively used Apache Kafka, Apache Spark, HDFS and Apache Impala to build a near real time data pipelines that get, transform, store and analyze click stream data to provide a better personalized user experience.• Primarily involved in Data Migration using SQL, SQL Azure, Azure Storage, and Azure Data Factory, SSIS, PowerShell.• Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling• Implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB).• Performed data extraction, transformation, loading, and integration in data warehouse, operational data stores and master data management• Experienced in ETL concepts, building ETL solutions and Data modeling• Worked on architecting the ETL transformation layers and writing spark jobs to do the processing.• Aggregated daily sales team updates to send report to executives and to organize jobs running on Spark clusters• Used Pyspark for data frames, ETL, Data Mapping, Transformation and Loading in complex and high-volume environment• Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines• Create Spark code to process streaming data from Kafka cluster and load the data to staging area for processing.• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team Using Tableau.• Implemented business use case in Hadoop/Hive and visualized in Tableau• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.• Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.

• Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions• Created monitors, alarms, notifications and logs for Lambda functions, Glue Jobs, EC2 hosts using Cloudwatch.• Worked on EMR clusters of AWS for processing Big Data across a Hadoop Cluster of virtual servers.• Developed a python script to transfer data, REST API’s and extract data from on-premises to AWS S3. Implemented Micro Services based Cloud Architecture using Spring Boot.• Worked on Docker containers snapshots, attaching to a running container, removing images, managing Directory structures and managing containers.• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS.• Used Apache NiFi to copy data from local file system to HDP.• Designed both 3NF Data models and dimensional Data models using Star and Snowflake Schemas Handling message streaming data through Kafka to s3.• Implementing python script for creating the AWS Cloud Formation template to build EMR cluster with instance types.• Experience with deploying Hadoop in a VM and AWS Cloud as well as physical server environment• Monitor Hadoop cluster connectivity and security and File system management.• Developed Mappings using Transformations like Expression, Filter, Joiner and Lookups for better data messaging and to migrate clean and consistent data.• After the transformation of data is done, this transformed data is then moved to Spark cluster where the data is set to go live on to the application using Spark streaming and kafka.• Developed highly complex Python and Scala code, which is maintainable, easy to use, and satisfies application requirements, data processing and analytics using inbuilt libraries.

• Designing and building multi-terabyte, full end-to-end Data Warehouse infrastructure from the ground up on Confidential Redshift for large scale data handling Millions of records every day• Developed SSRS reports, SSIS packages to Extract, Transform and Load data from various source systems• Having experience in developing a data pipeline using Kafka to store data into HDFS.• Worked on Big data on AWS cloud services i.e. EC2, S3, EMR and DynamoDB• Created Entity Relationship Diagrams (ERD), Functional diagrams, Data flow diagrams and enforced referential integrity constraints and created logical and physical models using Erwin.• Created ad hoc queries and reports to support business decisions SQL Server Reporting Services (SSRS).• Strong understanding of AWS components such as EC2 and S3• Used Hive SQL, Presto SQL and Spark SQL for ETL jobs and using the right technology for the job to get done.• Measured Efficiency of Hadoop/Hive environment ensuring SLA is met• Managed security groups on AWS, focusing on high-availability, fault-tolerance, and auto scaling using Terraform templates. Along with Continuous Integration and Continuous Deployment with AWS Lambda and AWS code pipeline.• Wrote various data normalization jobs for new data ingested into Redshift.• Defined facts, dimensions and designed the data marts using the Ralph Kimball's Dimensional Data Mart modeling methodology using Erwin.• Created various complex SSIS/ETL packages to Extract, Transform and Load data• Defined and deployed monitoring, metrics, and logging systems on AWS.• Connected to Amazon Redshift through Tableau to extract live data for real time analysis.• Optimizing and tuning the Redshift environment, enabling queries to perform up to 100x faster for Tableau and SAS Visual Analytics. Environment: AWS, EC2, S3, DynamoDB, Redshift, Kafka, SQL Server, Erwin, Oracle 10g/11g, Informatica, RDS, NOSQL, Snow Flake Schema, MySQL, PostgreSQL, Tableau, Git Hub.

• Created HBase tables to load large sets of structured data.• Managed and reviewed Hadoop log files.• Used AWS Glue for the data transformation, validate and data cleansing.• Used Sqoop widely in order to import data from various systems/sources (like MySQL) into HDFS.• Created components like Hive UDFs for missing functionality in HIVE for analytics.• Cluster co-ordination services through Zookeeper.• Worked extensively with HIVE DDLs and Hive Query language (HQLs).• Developed data pipeline using flume, Sqoop and pig to extract the data from weblogs and store in HDFS.• Used Sqoop to import and export data from HDFS to RDBMS and vice-versa.• Exported the analyzed data to the relational database MySQL using Sqoop for visualization and to generate reports.• Developed UDF, UDAF, UDTF functions and implemented it in HIVE Queries.• Implemented SQOOP for large dataset transfer between Hadoop and RDBMs.• Processed data into HDFS by developing solutions.• Created Map Reduce Jobs to convert the periodic of XML messages into a partition avro Data.• Assisted in creating and maintaining Technical documentation to launching HADOOP Clusters and even for executing Hive queries and Pig Scripts.Environment: Hadoop, HDFS, Map Reduce, AWS, Hive, Pig, Sqoop, HBase, Shell Scripting, Oozie, Oracle 11g.

• Loaded data from MySQL server to the Hadoop clusters using the data ingestion tool Sqoop.• Extensively worked with PySpark / Spark SQL for data cleansing and generating Data Frames and RDDs.• Involved in creating Hive tables, loading with data and writing hive queries on top of data present in HDFS.• Worked on tuning the performance Pig queries. Involved in Developing the Pig scripts for processing data.• Written Hive queries to transform the data into tabular format and process the results using Hive Query Language.• By using Apache Flume loaded real time unstructured data like xml data, log files into HDFS.• Processed large amount both structured and unstructured data using MapReduce framework.• Designed solution to perform ETL tasks like data acquisition, data transformation, data cleaning and efficient data storage on HDFS• Developed Spark code using Scala and Spark Streaming for faster testing and processing of data.• Store the resultant processed data back into Hadoop Distributed File System.• Applied machine learning algorithms (K- nearest Neighbors, random forest) using Spark MLib on top of HDFS data and compare the accuracy between the models.• Used Tableau to get the visualizations on data outcome from the ML algorithms.Environment: Apache Sqoop, Apache Flume, Hadoop, MapReduce, Spark, Hive, pig, Spark MLib, Tableau