Sunitha N Email & Phone Number

South Brunswick Township, Nj, Us

United States

Rochester, Minnesota, United States

1.Developing Data Pipelines: Developed a data pipeline using Kafka and Spark to store data into HDFS. Worked on reading and writing multiple data formats like JSON, ORC, Parquet on HDFS using PySpark. Designed and mechanized custom-constructed input connectors utilizing Spark, Sqoop, and Oozie to ingest and break down informational data from RDBMS to Azure Data Lake.2. Data Analysis and Processing:Extensively utilized Databricks notebooks for interactive analysis utilizing Spark APIs.Worked with data science group to do pre-processing and include feature engineering, helping machine learning algorithms in production.Developed spark jobs using Scala on top of YARN for batch analysis.Developed spark applications in Python (PySpark) on a distributed environment to load a huge number of CSV files with different schemas into Hive ORC tables.3.Database and SQL Development:Broad involvement in working with SQL, with profound knowledge of T-SQL (MS SQL Server).Developer SQL stored procedures to support new development projects.Provide guidance to the development team working on PySpark as an ETL platform.4.Cloud Platform and Architecture:Used Azure Databricks for a fast, easy, and collaborative spark-based platform on Azure.Used Databricks to integrate easily with the whole Microsoft stack.Involved in building an Enterprise Data Lake utilizing Data Factory and Blob storage, enabling different groups to work with more complex situations and ML solutions.Involvement in working with Azure cloud platforms (HDInsight, Databricks, Data Lake, Blob, Data Factory, Synapse, SQL DB, and SQL DWH).5.Collaboration and Team Support:Provide guidance to the development team working on PySpark as an ETL platform.Collaborate with the data science group and various teams to support their needs in data preprocessing and feature engineering to ensure smooth integration and operation of the data pipeline and analysis tasks.

St Louis, Missouri, United States

1.CI/CD Pipeline Development: Programmatically create CI/CD pipelines in Jenkins using Groovy scripts and Jenkinsfile. Integrate enterprise tools and testing frameworks into Jenkins for fully automated pipelines from Dev Workstations to Prod environment.2.Cloud Infrastructure and Migration: Work with AWS services like EC2, S3, RDS, ELB, EBS, VPC, Route53, auto scaling groups, CloudWatch, CloudFront, IAM. Build configurations and troubleshoot server migration from physical to cloud infrastructure on Amazon Web Services.3.Big Data Processing and Analytics: Implement Spark using PySpark libraries for faster testing and data processing. Perform capacity planning for cloud infrastructure with AWS EMR. Use Delta Lake for reliable data storage in data lakes. Write Kafka producers to stream data from external REST APIs to Kafka topics. Create Hive tables, load and analyze data using Hive scripts. Implement partitioning, dynamic partitions, and buckets in Hive.4.Data Warehouse Migration and Integration: Migrate data from Teradata to Snowflake cloud data warehouse. Develop Scala-based Spark applications for data cleansing, event enrichment, aggregation, denormalization, and preparation for machine learning and reporting teams.5.ETL Development and Integration: Use SSIS to create interfaces between front-end applications and SQL Server databases. Migrate data between legacy databases and SQL Server Database.6.Documentation and Collaboration: Document data engineering processes, workflows, and configurations. Collaborate with cross-functional teams, developers, data scientists, and reporting teams to ensure smooth integration and utilization of data engineering solutions.7.Performance Optimization and Troubleshooting: Optimize data processing workflows and pipelines for improved performance. Troubleshoot and debug issues related to data processing, ETL, and pipeline failures.Automated tasks and processes and Implement scalable solutions to handle large data volumes.

Tampa, Florida, United States

Data Cleaning and Warehouse Management:Partner with ETL developers to ensure data cleanliness and keep the data warehouse up-to-date for reporting using Pig.Select and generate data, store it in CSV files, and upload them to AWS S3 using AWS EC2. Structure and store the data in AWS Redshift.Reporting and Visualization:Work with Tableau to integrate Hive, create Tableau Desktop reports, and publish them to Tableau Server.Collaborate on designing visualizations and dashboards for effective data analysis.Technology Stack Management:Install and configure various technologies like Hadoop, MySQL, PostgreSQL, SQL Server, Sqoop, Hive, and HBase.Utilize Oozie Operational Services for batch processing and dynamic workflow scheduling.Infrastructure Automation:Develop bashrc files and XML configurations to automate the deployment of Hadoop VMs on AWS EMR.Set up and debug Logstash to send Apache logs to AWS Elasticsearch.Data Collection and Processing:Design and implement a system to collect data from multiple portals using Kafka and process it using Spark.Create and organize HDFS for efficient data storage and processing.Data Transformation and Cleansing:Develop PySpark scripts to merge static and dynamic files and perform data cleansing.Data Integration and Database Management:Use Spark to insert data from multiple CSV files into MySQL, SQL Server, and PostgreSQL databases.Create a data service layer in Hive with internal tables for data manipulation and organization.Business Intelligence and Analytics:Achieve business intelligence by creating and analyzing an application service layer in Hive containing integrated internal tables with HBase.Cluster Coordination and Data Consistency:Utilize ZooKeeper to coordinate servers in clusters and maintain data consistency.

Wheeling, West Virginia, United States

Requirements Analysis:Review functional and non-functional requirements to understand project goals and deliverables.Cloud Services and Infrastructure:Gain proficiency in AWS cloud services to leverage cloud-based solutions for data engineering tasks.Parallel Computing and Distributed Systems:Design and implement a MapReduce-based large-scale parallel relation-learning system to process and analyze vast amounts of data efficiently.Hadoop Configuration and Development:Install and configure Hadoop MapReduce and HDFS for distributed data processing.Develop custom File System plugins to enable seamless integration of Hadoop ecosystem components such as HBase, Pig, and Hive.Middleware Optimization:Rewrite middle-tier on the WebLogic application server to enhance performance and scalability.Quality Assurance and Testing:Actively participate in code reviews, ensure adherence to coding standards, and perform unit testing and integration testing.Data Modeling and Querying:Create Hive tables and work with Hive QL for data modeling and querying tasks.Job Flow Management:Define and manage job flows to orchestrate data processing workflows efficiently.Data Integration:Import and export data between HDFS and Oracle Database using Sqoop for seamless data integration.

Hyderabad ,Atlanta, Georgia, United States

❖ Development and Maintenance of STI Platform application and Channel Link Application - solutionto integrate many distribution channels and support the application integration for the entirescope of financial products.❖ Create and update application documentations and Perform quality audits to adhere to IBM❖ Quality standards.❖ Developed complex PL/SQL functions and triggers to automate data workflows and ensure data integrity within the database.❖ Associated with coding phases of the project.❖ Involved in Unit Testing of the application. ❖ Generated reports as per the client requests.❖ Involved in the production support to handle different incidents.❖ Monitoring of daily jobs.❖ Resolving the tickets❖ Involved in code changes for releases and continuous development.

Hyderabad, Telangana, India

❖ Responsible for development of detailed design document.❖ Responsible for Designing the Forms and validating various fields in the entire application.❖ Developed complex SQL queries, PL/SQL stored procedures, and functions in DB2 and Oracle for data retrieval, manipulation, and reporting purposes.❖ Implemented User controls for User data input.❖ Responsible for coding unit testing.❖ Involved in maintenance and enhancements.❖ Responsible for development and bug fixing.❖ Participated in reviews❖ Responsible for configuration management activities.❖ Involved in process review audits.❖ Responsible for preparing test cases.