Sai Kumar Dyavathi Email & Phone Number

California, United States

Indianapolis, Indiana, United States

• Automated resulting scripts and workflow using Apache Airflow and shell scripting to ensure daily execution in production.• Evaluating client needs and translating their business requirement to functional specifications thereby onboarding them onto Hadoop ecosystem. • Installed application on AWS EC2 instances and configured the storage on S3 buckets. • Used the AWS-CLI to suspend an AWS Lambda function. Used AWS CLI to automate backups of ephemeral data-stores to S3 buckets, EBS.• Developed Java Map Reduce programs for the analysis of sample log file stored in cluster.• Developed custom Kafka producer and consumer for different publishing and subscribing to Kafka topics. • Worked on setting up and configuring AWS's EMR Clusters and Used Amazon IAM to grant fine-grained access to AWS resources to users• Very good implementation experience of Object-Oriented concepts, Multithreading and Java/Scala• Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark data bricks cluster• Migrated Map reduce jobs to Spark jobs to achieve better performance.• Used AWS Data Pipeline to schedule an Amazon EMR cluster to clean and process web server logs stored in Amazon S3 bucket.• Written the Map Reduce programs, Hive UDFs in Java• Extracted and updated the data into HDFS using Sqoop import and export. • Developed a Spark job in Java which indexes data into ElasticSearch from external Hive tables which are in HDFS.• Implemented Data Quality in ETL Tool Talend and having good knowledge in Data Warehousing• Working on designing the Map Reduce and Yarn flow and writing Map Reduce scripts, performance tuning and debugging. • Created Airflow Scheduling scripts in Python

Grapevine, Tx

• Installed Kafka Producer on different severs and Scheduled to produce data for every 10 seconds Implemented Data Quality in ETL Tool Talend and having good knowledge in Data Warehousing Developed Apache Spark applications by using spark for data processing from various streaming sources.• Participates in the development improvement and maintenance of snowflake database applications.• Create custom logging framework for ELT pipeline logging using Append variables in Data factory• Enabling monitoring and azure log analytics to alert support team on usage and stats of the daily runs• Took proof of concept projects ideas from business, lead, developed and created production pipelines that deliver business value using Azure Data Factory• Developed and designed data integration and migration solutions in Azure.• Configured Oozie workflow to run multiple Hive and Pig jobs which run independently with time and data availability.• Delivered de normalized data for Power BI consumers for modeling and visualization from the produced layer in Data lake• Worked with PowerShell and UNIX scripts for file transfer, emailing and other file related tasks.• Kept our data separated and secure across national boundaries through multiple data centers and regions.• Build the Logical and Physical data model for snowflake as per the changes required.• Implement Continuous integration/continuous development best practice using Azure DevOps, ensuring code versioning• Used Databricks for encrypting data using server-side encryption.• Used Airflow to monitor and schedule the work• Involved in creating HDInsight cluster in Microsoft Azure Portal also created Events hub and Azure SQL Databases.• Responsible to manage data coming from different sources through Kafka.

Mahwah, New Jersey, United States

• Responsible for loading the data from BDW Oracle database, Teradata into HDFS using Sqoop. • Implemented AJAX, JSON, and Java script to create interactive web screens. • Created Session Beans and controller Servlets for handling HTTP requests from Talend• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE. • Perform DB activities such as indexing, performance tuning, and backup and restore. • Expertise in writing Hadoop Jobs for analyzing data using Hive QL (Queries), Pig Latin (Data flow language), and custom MapReduce programs in Java. • Developed HIVE UDFs to incorporate external business logic into Hive script and Developed join data set scripts using HIVE join operations. • Performed Data Visualization and Designed Dashboards with Tableau and generated complex reports including chars, summaries, and graphs to interpret the findings to the team and stakeholders. • Developed spring boot applications to read data from Kafka in an event-based manner. These applications were developed to run as micro-services that deals with parts• Implemented Bucketing and Partitioning using hive to assist the users with data analysis. • Used Oozie scripts for deployment of the application and perforce as the secure versioning software.• Connected to AWS Redshift through Tableau to extract live data for real time analysis• Performed statistical analysis using SQL, Python, R Programming and Excel.• Used Python& SAS to extract, transform & load source data from transaction systems, generated reports, insights, and key conclusions.• Developed story telling dashboards in Tableau Desktop and published them on to Tableau Server which allowed end users to understand the data on the fly with the usage of quick filters for on demand needed information.

San Francisco, California, United States

• Developed data pipeline using flume, Sqoop and pig to extract the data from weblogs and store in HDFS.• Created HBase tables to load large sets of structured data. • Worked extensively with HIVE DDLs and Hive Query language (HQLs).• Processed data into HDFS by developing solutions.• Analyzed the data using Map Reduce, Pig, Hive and produce summary results from Hadoop to downstream systems.• Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.• Implemented Sqoop for large dataset transfer between Hadoop and RDBMs.• Created components like Hive UDFs for missing functionality in HIVE for analytics.• Developing Scripts and Batch Job to schedule a bundle (group of coordinators) which consists of various.• Cluster co-ordination services through Zookeeper.• Assisted in creating and maintaining Technical documentation to launching HADOOP Clusters and even for executing Hive queries and Pig Scripts.

India

• Utilized Agile and Scrum methodology for team and project management. • Involved in integration of Hadoop cluster with spark engine to perform BATCH and GRAPHX operations.• Achieved business intelligence by creating and analyzing an application service layer in Hive containing internal tables of the data which are also integrated with Hbase• Generated report on predictive analytics using Python and Tableau including visualizing model performance and prediction results. • Utilized Sqoop to import structured data from MySQL, SQL Server, PostgreSQL, and a semi-structured csv file dataset into HDFS data lake.• Troubleshot RSA SSH keys in Linux for authorization purposes.• Partnered with ETL developers to ensure that data is well cleaned and the data warehouse is up-to-date for reporting purpose by Pig. • Experience creating and organizing HDFS over a staging area.