Kalyan P Email and Phone Number

• Used Hive to perform partitioning and Bucketing the data present in Hive for getting and drawing the conclusions from the correlated data.• Developed automation system using PowerShell scripts and JSON templates to remediate the Azure services.• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.• Migrate databases to cloud platform SQL Azure and as well the performance tuning.• Used Spark Data Frame API to get the analysis fast using Hive Context and handover the data to the machine learning analytics team based on the requirement.• Used Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala and NoSql databases such as HBase and Cassandra.HDFS, Spark, Kafka, Azure, Power BI, Azure Data Lake, Data Factory, Data Storage, Data Bricks, Map reduce, Scala, Python, Spark, Hive, HBase, Pig, Zookeeper, Oozie, Sqoop, PL/SQL, Oracle 12c, MS SQL, Mongo DB, JSP, Git Hub

• Performed data analysis and developed analytic solutions. Data investigation to discover correlations / trends and the ability to explain them.• Worked with Data Engineers, Data Architects, to define back-end requirements for data products (aggregations, materialized views, tables – visualization)• Worked on AWS services like EC2, S3, EMR, FSx, Lambda, Cloud watch, RDS, Auto scaling, Cloud Formation, SQS, ECS, EFS, DynamoDB, Route53, Glue etc. • Working in relational SQL and NoSQL databases, including Oracle, Hive, Sqoop and HBase.• Experience in job workflow scheduling and monitoring tools like Oozie and good knowledge on Zookeeper to coordinate the servers in clusters and to maintain the data consistency.• Developed and executed custom MapReduce programs, PigLatinscripts and HQLqueries. • Created a Lambda Deployment function, and configured it to receive events from S3 buckets• Writing UNIX shell scripts to automate the jobs and scheduling cron jobs for job automation using commands with Crontab.• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS.• Schedule nightly batch jobs using Oozie to perform schema validation and IVP transformation at larger scale to take the advantage of the power of Hadoop.• Developed Spark/Scala,Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources.HDFS, Spark, Kafka, AWS (EC2, Lambda, S3, IAM, Cloud Watch, Cloud Formation, Redshift), Map reduce, Sqoop, Scala, Python, Spark, Hadoop, Hive, Impala, Hbase, Pig, Zookeeper, Oozie, PL/SQL, Oracle 12c, Mongo DB, T-SQL, Git

• Worked on Apache Flume to stream data from Oracle to Apache Kafka topics.• Managed docker images using Quay.• Installed Oozie workflow engine to run multiple Hive and Pig Jobs.• Created hive managed and external tables.• Designed AWS Glue pipelines to ingest, process, and store data interacting with different services in AWS.• Populated HDFS and Cassandra with huge amounts of data using Apache Kafka. • Monitoring YARN applications. Troubleshoot and resolve cluster related system problems.• Upgrading the Hadoop Cluster from CDH3 to CDH4, setting up High Availability Cluster and integrating HIVE with existing applications. • Developing UDFs in java for hive and pig and worked on reading multiple data formats on HDFS using Scala.• Imported Legacy data from SQL Server and Teradata into Amazon S3.• Monitoring jobs using Hadoop resource manager and Ambari Views.• Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis. • Partnered with ETL developers to ensure that data is well cleaned and the data warehouse is up-to-date for reporting purpose by Pig. • Experienced on Hadoop/Hive on AWS, using both EMR and non-EMR-Hadoop in EC2.• Developed and designed system to collect data from multiple portal using kafka and then process it using spark.• Used Sqoop tool to extract data from a relational database into Hadoop.• Worked in AWS environment for development and deployment of Custom Hadoop Applications.• Worked on ETL using Spark, Hive, HBase, and Oozie on Hadoop. • Processing of incoming files using Spark native API, using Spark scriptsKafka, AWS, Python, Spark, Hadoop, Pig, Hive, Yarn, Zookeeper, Hbase, Oozie, Java, Scala, My SQL, Cassandra, MapReduce, Linux.Kafka, AWS, Python, Spark, Hadoop, Pig, Hive, Yarn, Zookeeper, Hbase, Oozie, Java, Scala, My SQL, Cassandra, MapReduce, Linux.

• Collaborated with Business Analysts, SMEsacross departments to gather business requirements, and identify workable items for further development. • Developed a python script to transfer data from on-premises to AWS S3• Validated the Map reduce, Pig, Hive Scripts by pulling the data from the Hadoop and validating it with the data in the files and reports.• Developed a python script to hit REST API’s and extract data to AWS S3• Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions• In charge of PySpark code, creating dataframes from tables in data service layer and writing them to a Hive data warehouse• Expertized in implementing Spark using Scala and Spark SQL for faster testing and processing of data responsible to manage data from different sources.• Implemented Spark using Scala and Spark SQL for faster testing and processing of data• Collaborated with the team using GIT, GitHub, SourceTree version control platform.• Installed Airflow and created a database in PostgreSQL to store metadata from Airflow.• Configured documents which allow Airflow to communicate to its PostgreSQL database.• Developed Airflow DAGs in python by importing the Airflow libraries.• Worked with SCRUM team in delivering agreed user stories on time for every Sprint. • Worked on analyzing and resolving the production job failures in several scenarios.• Implemented UNIX scripts to define the use case workflow and to process the data files and automate the jobs.• Knowledge on implementing the JILs to automate the jobs in production cluster. Spark, Redshift, Python, HDFS, Hive, Pig, Scala, Kafka, Shell scripting, Linux, Jenkins, Eclipse, Git, Oozie, Talend, Agile Methodology.

• Installed/Configured/Maintained Apache Hadoop clusters for application development and Hadoop tools like Hive, Pig, HBase, Flume, Oozie Zookeeper and Sqoop. • Generated Custom SQL to verify the dependency for the daily, Weekly, Monthly jobs.• Using Nebula Metadata, registered Business and Technical Datasets for corresponding SQL scripts• Expert in creating Hive UDFs using Java to analyze the data efficiently. • Wrote MapReduce jobs using Java API and Pig Latin. • Wrote Pig scripts to run ETL jobs on the data in HDFS and further do testing. • Created consumption views on top of metrics to reduce the running time for complex queries• Built APIs that will allow customer service representatives to access the data and answer queries.• Designed changes to transform current Hadoop jobs to HBase. • Design and implement MapReduce jobs to support distributed processing using java, Hive and Apache Pig. • Monitored and tuned database resources and activities for SQL Server databases.• Create Hive external tables on the MapReduce output before partitioning, bucketing is applied on it. • Handled fixing of defects efficiently and worked with the QA and BA team for clarifications. • Evaluated model accuracy by dividing data into training and test datasets and computing metrics using evaluators.• Tuned training hyper-parameters by integrating cross-validation into pipelines.• Wrote data ingestion systems to pull data from traditional RDBMS platforms such as Oracle and Teradata and store it in NoSQL databases such as MongoDB. Hadoop, HDFS, Hive, Hbase. Flume. Zookeeper, Sqoop, Pig, Java, Map Reduce, Oracle 9i, SQL Server, Mongo DB, Windows.