S Deepthi Email & Phone Number

Des Moines, Iowa, United States

Developing new data streaming patterns leveraging an enterprise platform to support system to system data exchange. Formulates and defines the system scope and objectives for the assigned projects. Prepares detailed specifications to write programs. Designs, codes, tests, debugs, documents, and maintains those programs. Has good understanding of business and functional units for which the application is designed. Responsible for project completion and end-user satisfaction.

Philadelphia, Pennsylvania, United States

St Louis, Missouri, United States

• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.• Authoring Python (PySpark) Scripts for custom UDF’s for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks. Migrate data from on-premises to AWS storage buckets.• Developed frameworks and processes to analyze unstructured information. Assisted in Azure Power BI architecture design.• Created a Python process hosted on Elastic Beanstalk to load the Redshift database daily from several source.• Developed SQL scripts to Upload, Retrieve, Manipulate and handle sensitive data (National Provider Identifier Data I.e. Name, Address, SSN, Phone No) in Teradata, SQL Server Management Studio and Snowflake Databases for the Project.• Experienced in running query using Impala and used BI tools to run ad-hoc queries directly on Hadoop.• Experienced in working with various kinds of data sources such as Teradata and Oracle. Successfully loaded files to HDFS from Teradata, and load loaded from hdfs to hive and impala• Worked on to retrieve the data from FS to S3 using spark commands• Built S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup on AWS.• Provided access to data necessary to perform analysis on scheduling, pricing, bus bunching and performance. Queries in the Redshift environment performed 100-1000x faster than in legacy environments.• Transform and analyze the data using Pyspark, HIVE, based on ETL mappings• Developed Unix Shell Scripts and automated them using CRON job scheduler.• Experienced with machine learning algorithm such as logistic regression, random forest, XGboost, KNN, SVM, neural network, linear regression, lasso regression and k - means• Implemented Statistical model and Deep Learning Model (Logistic Regression, XGboost, Random Forest, SVM, RNN, and CNN).

Englewood, Colorado, United States

• Expertise in using Docker to run and deploy the applications in multiple containers like Docker Swarm and Docker Wave.• Developed complex Talend ETL jobs to migrate the data from flat files to database. Pulled files from mainframe into Talend execution server using multiple ftp components.• Developed complex Talend ETL jobs to migrate the data from flat files to database. Developed Talend ESB services and deployed them on ESB servers on different instances.• Architect and design server less application CI/CD by using AWS Server less (Lambda) application model.• Developed stored procedures/views in Snowfpyth authoring, scheduling and monitoring Data Pipelines• Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines• Performed data extraction, transformation, loading, and integration in data warehouse, operational data stores and master data management.• I have been a lead and organized the learnings and have given the presentation for the new people to learn about the process that they have to do in the real time project.• I have guided/lead the group of new people for the application process.• Strong understanding of AWS components such as EC2 and S3• Performed Data Migration to GCP• Used Sqoop to channel data from different sources of HDFS and RDBMS.• Performing Database Testing and Unit Testing.• Performed UAT testing before and liased with development team.• Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra.

St. Louis City County, Missouri, United States

• Migrating data from FS to Snowflake within the organization• Created consumption views on top of metrics to reduce the running time for complex queries.• Exported Data into Snowflake by creating Staging Tables to load Data of different files from Amazon S3.• Compare the data in a leaf level process from various databases when data transformation or data loading takes place. I need to analyze and look into the data quality when these types of loads are done (To look for any data loss, data corruption).• As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to snowflake.• Created Metric tables, End user views in Snowflake to feed data for Tableau refresh.• Generated Custom SQL to verify the dependency for the daily, Weekly, Monthly jobs.• Using Nebula Metadata, registered Business and Technical Datasets for corresponding SQL scripts• Experienced in working with spark ecosystem using Spark SQL and Scala queries on different formats like text file, CSV file.• Developed spark code and spark-SQL/streaming for faster testing and processing of data.• Closely involved in scheduling Daily, Monthly jobs with Precondition/Post condition based on the requirement.• Worked on data processing and transformations and actions in spark by using Python (Pyspark) language.• Monitor the Daily, Weekly, Monthly jobs and provide support in case of failures/issues.• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.• Worked on analyzing Hadoop Cluster and different big data analytic tools including Pig, Hive. • Working experience with data streaming process with Kafka, Apache Spark, Hive.

Hyderabad, Telangana, India

• Developed and Configured Kafka brokers to pipeline server logs data into Spark streaming.• Defined job work flows as per their dependencies in Oozie.• Successfully Generated consumer group lags from Kafka using their API• Deployed an Apache Solr/Lucene search engine server to help speed up the search of financial documents.• Wrote queries Using DataStax Cassandra CQL to create, alter, insert and delete elements.• Written the Map Reduce programs, Hive UDFs in Java.• Used Map Reduce JUnit for unit testing.• Developed Automation Regressing Scripts for validation of ETL process between multiple databases like AWS Redshift, Oracle, Mongo DB, T-SQL, and SQL Server using Python. • Installed and configured Hive, Pig, Sqoop, Flume and Oozie on the Hadoop cluster.• Estimated the Software & Hardware requirements for the Name Node and Data Node & planning the cluster.• Extracted the needed data from the server into HDFS and Bulk Loaded the cleaned data into Hbase.• Supporting Continuous storage in AWS using Elastic Block Storage, S3, Glacier. Created Volumes and configured Snapshots for EC2 instances• Developed Airflow DAGs in python by importing the Airflow libraries.• Worked on a python script to extract data from Netezza databases and transfer it to AWS S3• Developed Lambda functions and assigned IAM roles to run python scripts along with various triggers (SQS, EventBridge, SNS)

Hyderabad, Telangana, India

• The custom File System plugin allows Hadoop MapReduce programs, HBase, Pig and Hive to work unmodified and access files directly.• Performed and automated SQL Server version upgrades, patch installs and maintained relational databases.• Developed complex parameterized reports which were used for making current and future business decisions.• Analyzed requirement and impact by participating in Joint Application Development sessions with business client online.• Created HBase tables and used HBase sinks and loaded data into them to perform analytics using Tableau.• Used Spark-SQL to Load JSON data and create Schema RDD and loaded it into Hive Tables and handled structured data using Spark SQL.• Configured, monitored, and optimized Flume agent to capture web logs from the VPN server to be put into Hadoop Data Lake.