Varshith Reddy Email & Phone Number

United States

United States

San Antonio, Texas, United States

• Created Glue ETL jobs which reads the data from S3 and process the data using transformation techniques and stores the data in s3.• Created Glue Crawlers which crawl the data present in s3 and create the Glue tables.• Used Athena service to query the above tables.• Implemented a serverless architecture using API Gateway, Lambda and DynamoDB.• Created lambda functions and configured it to receive events from s3 bucket.• Designed data models that are used in AWS lambda applications which are aimed to do complex analysis creating analytical reports for end-to-end traceability, lineage, definition ok key Business elements from Aurora.• Writing code that optimizes performance of AWS services used by application teams and provide Code level security for clients.(IAM roles, credentials, encryption etc...)• creating code, scripts, and data pipelines that make use of data that is both organized and unstructured and is integrated from several sources.• Designing and refining Spark SQL queries, Data frames, importing data from data sources, performing transformations, read/write operations, and saving the outcomes to an output directory in S3 are all part of the process.• Database cloning was designed and put into practice using Python, and shell scripts were used to build backend support for applications.• Utilizing the Scala and Java APIs, Spark programs were created to perform transformations and operations on RDDs.• creating apps for interactive analysis, batch processing, and stream processing using PySpark, Spark Java, and Scala.• Created pods and controlled using Kubernetes to deploy customized Python scripts. Used Jenkins pipelines to push all micro-service builds out to the Docker registry.• Provided all daily GIT assistance required for various projects. Design and upkeep of GIT repositories, as well as access control measures, fall within this responsibility.

Liberty, New York, United States

• Analyzed SQL scripts and designed the solution to implement using Scala.• Developed analytical component using Scala, Spark, and Spark Stream.• Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.• Developed Spark code to using Scala and Spark-SQL for faster processing and testing.• Designed scalable (Scala) Web Architecture hosting reports for the entire application.• Wrote coding highly flexible, scalable, & distributed applications using Scala.• Constructed and implemented multiple-table links requiring complex join statements, including Outer-Joins and Self-Joins.• Coordinated with team and Developed framework to generate Daily adhoc reports and Extracts from enterprise data from BigQuery.• Worked with NoSQL databases like Cassandra and HBase. To identify, define, compile, document, and communicate the requirements for data migration, work with the project team and subject matter experts.• To improve performance, Map migrates jobs and converts them to Spark jobs.• Created the RDDs and Data Frames for the necessary input data and used PySpark to conduct the data manipulations.• Involvement in the analysis, design, coding, and implementation of requirements. Data is transferred into ADLS and dBLOB storage via Linked Services that are connected to Teradata, SQL Server, and both.• Continuous Delivery pipeline was put in place using Docker and GIT Hub.

Delhi, India

• Spent a lot of time working with Hive's metadata imports and migrating existing tables and programs to run on Hive and the AWS cloud.• Created a standard framework to export data using Sqoop to external data bases and import data from external data bases into HDFS.• Worked on integrating EMR with Spark 2, S3, and Snowflake on Amazon Web Service (AWS)..• Accountable for operating the Elastic MapReduce cluster through the AWS UI and continuously monitoring it.• Used Spark Data Frame Operations to perform required Validations in the data and to perform analytics on the Hive data• Developed PySpark scripts, using both Data Frames/SQL and RDD in Spark for data aggregation and queries.• Designed and implemented a fully operational production grade largescale data solution on Snowflake.• Hands-on experience on developing SQL Scripts for automation purpose. Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).

Mumbai, Maharashtra, India

• Accountable for developing Hadoop-based scalable distributed data solutions. Hadoop/HBase clusters were set up and benchmarked for internal use.• Java programming language was used to create simple to complicated Map/Reduce jobs, which were then implemented using Hive and Pig.• Participated in all phases of data mining, data collection, data cleaning, developing models, validation, visualization, performed Gap analysis to provide feedback to the business team to improve the software delivery.• Creating reusable Python scripts to ensure data integrity between the source (Teradata/Oracle) and target system.• Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.• Developed programs in Spark to use on the application for faster data processing than standard MapReduce programs• Worked on the ETL process to clean and load massive amounts of data (JSON/CSV files) that were extracted from a number of websites to the SQL server.• Enhancing Map/Reduce tasks to utilize HDFS effectively using a variety of compression techniques