Sai Kumar Dyavathi Email & Phone Number

California, United States

Indianapolis, Indiana, United States

• Automated resulting scripts and workflow using Apache Airflow and shell scripting to ensure daily execution in production.
• Evaluating client needs and translating their business requirement to functional specifications thereby onboarding them onto Hadoop ecosystem.
• Installed application on AWS EC2 instances and configured the storage on S3 buckets.
• Used the AWS-CLI to suspend an AWS Lambda function. Used AWS CLI to automate backups of ephemeral data-stores to S3 buckets, EBS.
• Developed Java Map Reduce programs for the analysis of sample log file stored in cluster.
• Developed custom Kafka producer and consumer for different publishing and subscribing to Kafka topics.

Grapevine, TX

• Installed Kafka Producer on different severs and Scheduled to produce data for every 10 seconds Implemented Data Quality in ETL Tool Talend and having good knowledge in Data Warehousing Developed Apache Spark.
• Participates in the development improvement and maintenance of snowflake database applications.
• Create custom logging framework for ELT pipeline logging using Append variables in Data factory
• Enabling monitoring and azure log analytics to alert support team on usage and stats of the daily runs
• Took proof of concept projects ideas from business, lead, developed and created production pipelines that deliver business value using Azure Data Factory
• Developed and designed data integration and migration solutions in Azure.

Mahwah, New Jersey, United States

• Responsible for loading the data from BDW Oracle database, Teradata into HDFS using Sqoop.
• Implemented AJAX, JSON, and Java script to create interactive web screens.
• Created Session Beans and controller Servlets for handling HTTP requests from Talend
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Perform DB activities such as indexing, performance tuning, and backup and restore.
• Expertise in writing Hadoop Jobs for analyzing data using Hive QL (Queries), Pig Latin (Data flow language), and custom MapReduce programs in Java.

San Francisco, California, United States

• Developed data pipeline using flume, Sqoop and pig to extract the data from weblogs and store in HDFS.
• Created HBase tables to load large sets of structured data.
• Worked extensively with HIVE DDLs and Hive Query language (HQLs).
• Processed data into HDFS by developing solutions.
• Analyzed the data using Map Reduce, Pig, Hive and produce summary results from Hadoop to downstream systems.
• Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.

India

• Utilized Agile and Scrum methodology for team and project management.
• Involved in integration of Hadoop cluster with spark engine to perform BATCH and GRAPHX operations.
• Achieved business intelligence by creating and analyzing an application service layer in Hive containing internal tables of the data which are also integrated with Hbase
• Generated report on predictive analytics using Python and Tableau including visualizing model performance and prediction results.
• Utilized Sqoop to import structured data from MySQL, SQL Server, PostgreSQL, and a semi-structured csv file dataset into HDFS data lake.
• Troubleshot RSA SSH keys in Linux for authorization purposes.