Amruth P Email & Phone Number

United States

 Used Sqoop to import data into HDFS/Hive from multiple relational databases, performed operations, and exported the results back. Got involved in migrating the on-prem Hadoop system to using GCP (Google Cloud Platform).Demonstrated expertise in leveraging GCP services such as Compute Engine, Kubernetes Engine, Cloud Storage, BigQuery, and Cloud SQL for seamless migration and efficient operation of workloads. Extensively used Spark Streaming to perform the analysis of credit department data on real-time regular window time intervals coming from sources like Kafka. Performed Spark join optimizations, troubleshooting, monitored, and wrote efficient codes using Scala. Used big data tools Spark (Pyspark, SparkSQL) to conduct real-time analysis for banking transactions. Performed Spark transformations and actions on large datasets. Implemented Spark SQL to perform complex data manipulations, and to work with large amounts of structured and semi-structured data stored in a cluster using Data Frames/Datasets. Working with big data tools: Hadoop, Spark, Kafka, etc. Support existing GCP Data Management implementations. Created GCP Big Query authorized views for row-level security or exposing the data to other teams. Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark SQL, Data Frame, pair RDDs, and Spark YARN. Experience with terraform scripts which automate the step execution in EMR to load the data to Scylla DB. De-normalizing the data as part of transformation which is coming from Netezza and loading it to No Sql Databases and MySQL.  Loading data into snowflake tables from the internal stage using SnowSQL. Used import and export from the internal stage (snowflake). Implemented metadata and documentation management practices for Erwin modeling and data cataloging, ensuring accurate and up-to-date documentation of the data environment

United States

 Built finance performance metrics applications built on Hadoop, spark, and Bigdata.  Worked on analyzing data on the Hadoop cluster using different big data analytics tools including Spark (Spark SQL, Spark-Shell), Hive Data Warehouse, and Impala.  Implemented Spark using Scala, utilizing Data frames, Spark-SQL API, and RESTful APIs for faster processing of data.  Automated feature engineering mechanisms using Python scripts and deployed on BigQuery in Google Cloud Platform (GCP). Prepared Data models and schema on GCP for different projects based on star and snowflake schema designs. Hands-on experience installing, configuring, and using Hadoop ecosystem components like Hadoop HDFS, Yarn, Zookeeper, MapReduce, and Hive. Created a program using Python to archive raw files in a GCS bucket. Created a program using Scala for spark transformation in Dataproc. Used spark for fast In-Memory data processing and performed joins (Broadcast hash, Sort Merge Join), pivot (data transpose), and complex transformations on terabytes of data. Developed POCs on Spark Streaming to ingest flat files automatically when file landed in Edge Node landing Zone. Maintained and developed Docker images for a tech stack including Cassandra, Kafka, Apache, and several in-house written Java services running in Google Cloud Platform (GCP) on Kubernetes. Used rest API with Python to ingest Data from different sources to BigQuery for analysis. Created data catalog tables that provide an overview of the origin of data and its destination. Followed Agile Scrum Methodology to analyze, design, and implement applications to support functional and business requirements

Houston, Texas, United States

 Developed simple and complex spark jobs in Python for Data Analysis on different health data formats. Developed upgrade and downgrade scripts in SQL that filter bad and unnecessary records and find out unique records based on different criteria. Developed Secondary sorting implementation to get sorted values at the reduced side to improve spark job performance. Configure and launch AWS EC2 instance along with S3 bucket for ingestion of data and responsible for constructing identifiable and unidentifiable layers in RedShift spectrum for data masking in AWS. Automated CI/CD process using Jenkins, build-pipeline-plugin, maven, and GIT.  Implemented custom Data Types, Input Format, Record Reader, Output Format, and Record Writer for Spark job computations to handle custom business requirements. Implemented Pyspark scripts to classify data organizations into different classifieds based on different types of records. Strong programming skills in SCALA Worked on Parquet files, CSV files, Map side joins, bucketing, and partitioning for hive performance enhancement and storage improvement. Implemented Daily Cron jobs that automate parallel tasks of loading the data into SQL tables and Spark-Redis database. Performed extensive data validation using SQL. Worked with AWS services like S3, Glue, EMR, SNS, SQS, Lambda, EC2, RDS, and Athena to process data for downstream customers. Experience in creating microservices using Scala programming. Design and build GCP data-driven solutions for enterprise data warehouses and data lakes. Knowledge of handling Hive queries using Spark SQL that integrates the Spark environment. Implemented test scripts to support test-driven development and continuous integration. Junit framework was used to perform unit and integration testing. Configured build scripts for multi-module projects with Maven and Jenkins CI.

 Developed ETL data pipelines using Sqoop, Spark, Spark SQL, Scala, and Oozie. Used Spark for interactive queries, processing of streaming data, and integrated with popular NoSQL databases. Experience with AWS Cloud IAM, Data Pipeline, EMR, S3, EC2. Developed the batch scripts to fetch the data from AWS S3 storage and do required transformations.  Developed Spark code using Scala and Spark-SQL for faster processing of data. Created Oozie workflow engine to run multiple Spark jobs. Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark SQL, Data Frame, pair RDDs, and Spark YARN. Experience with terraform scripts which automate the step execution in EMR to load the data to Scylla DB. De-normalizing the data as part of transformation which is coming from Netezza and loading it to No SQL Databases and MySQL.  Very good understanding of Partitions and bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance. Solved performance issues in Hive and Pig scripts with an understanding of Joins, Group, and aggregation and how it translates to MapReduce jobs. Developed UDFs in Java as and when necessary to use in PIG and HIVE queries.