Divya Nimmala Email & Phone Number

Livonia, Mi, Us

Baltimore, Md, Us

Detroit, Michigan, United States

In my recent project, I led the development of a comprehensive data integration and warehousing solution using Snowflake, AWS Glue, Airflow, and Python. I began by analyzing our source databases and tables to understand the data landscape thoroughly. This analysis allowed me to develop precise SQL scripts to create essential database structures in Snowflake, including tables, views, functions, and stored procedures. I also created detailed mapping documents to define the connections between our existing Hadoop-based data warehouse (HIVE) and Snowflake, facilitating seamless data migration.I designed and implemented integration objects using AWS Glue to transfer data from HIVE to AWS S3, and then used Snowflake Task objects to load this data into Snowflake's Raw Layer. By orchestrating data movement from the Raw Layer to the Consumption Layer, I ensured that data was transformed and prepared for business intelligence (BI) reports. Additionally, I utilized DBT for debugging complex queries and optimizing ETL processes for better performance. To manage the entire ETL process, I scheduled jobs using Snowflake Task objects with CRON job patterns and developed comprehensive schedules in Airflow, ensuring efficient job execution and monitoring.Automation was a key aspect of the project. I wrote Python scripts to automate the extraction of data from Snowflake tables and generate Excel reports, which were then distributed via email to business users. Furthermore, I integrated automated tests into our CI/CD pipeline, enabling early detection of issues and rapid feedback on code changes. This project streamlined our data processes, enhanced reporting capabilities, and provided the business with timely, accurate insights, ultimately improving decision-making and operational efficiency.

Covington, Kentucky, United States

In this project, I led the creation of a strong data setup using Azure tools like Data Lake Storage Gen2 and Databricks clusters. By using Databricks clusters, we processed and transformed these large datasets, making them accurate and easily accessible. With Airflow DAGs and Azure Data Factory, we orchestrated ETL workflows, ensuring smooth data extraction, transformation, and loading processes. Collaboration with data providers and consumers ensured data accuracy and consistency. We optimized performance by reviewing error patterns regularly and implementing encryption and security measures to comply with industry standards. Integration of JSON processing and REST API interactions into SSIS projects showcased our expertise in handling diverse data formats. Automation of CI/CD pipelines using tools like Jenkins and Terraform ensured smooth deployment processes. Visualizing insights in Tableau from processed datasets facilitated data-driven decision-making, while Azure DevOps pipelines ensured continuous deployment. We organized training sessions for effective collaboration and standardized documentation practices for consistency and ease of use across the team. I managed ETL workflows with Airflow and Azure Data Factory for smooth data processing. Working closely with data sources and users, I ensured data accuracy and consistency. I also optimized data querying and implemented strict security measures. My skills in handling various data formats and integrating APIs were evident. Using DevOps methods, I automated deployment pipelines. I visualized data with Tableau for stakeholders and organized training for team collaboration. Additionally, I standardized documentation for better project consistency.

Hyderabad, Telangana, India

In this project, I oversaw Hadoop clusters to ensure they ran smoothly, focusing on availability, security, and performance. I also developed applications using MapReduce, Hive queries, and Pig scripts for data processing. Real-time data streaming applications were built with Kafka, and I managed topics to ensure data reliability. I utilized AWS Glue, Databricks, and RedShift Analytics for complex data solutions, including Spark applications for both batch and real-time processing. ETL procedures were fully integrated to maintain data integrity across various sources to Amazon S3 and SQL Data Warehouse. Sqoop was used to move data between Hadoop and relational databases, and SQL queries were written for Spark to analyze structured data. Hive queries were optimized for Hadoop data warehousing and analytics, and large-scale data processing was managed with Databricks and PySpark for effective transformations. AWS Glue simplified complex activities, and I designed and optimized SQL queries for maximum performance. Scala was used to develop apps, often in conjunction with Spark, and Pig Latin scripts were written for data transformations and ETL on Hadoop. I managed workflows for Hadoop job coordination and scheduling, as well as HBase clusters for storing and retrieving NoSQL data. Tableau was used to create interactive data visuals for analysis and reporting, while Python was employed for creating applications, scripts, and data analysis tools. Additionally, I designed and maintained data pipelines using AWS services such as Glue, Lambda, Step Functions, Code Build, Event Bridge, and Athena for various data processing tasks, and managed shell scripts for server administration, automation, and system maintenance.

Hyderabad, Telangana, India

In this project, I worked as a Python Developer, focusing on both backend and frontend development using the Django framework. I was responsible for designing and building REST APIs with Flask and Django to integrate data from diverse sources, such as Google Analytics, Salesforce, and AWS S3. Additionally, I managed the ingestion of streaming and transactional data across seven primary sources using PySpark, Redshift, and AWS services, ensuring a 99.9% data pipeline uptime. I implemented Apache Airflow for scheduling and monitoring data pipelines and handled infrastructure automation using Terraform to manage AWS resources like EC2, VPC, and ELB.I also performed data validation and created SQL and Python scripts to ensure accuracy across different Redshift metrics. On the frontend, I developed user interfaces using JavaScript, Bootstrap, Cassandra with MySQL, and HTML5/CSS. My role also involved working closely with business stakeholders to gather requirements and present insights, reports, and analysis. Furthermore, I showcased proficiency in managing Hadoop clusters and used JIRA for tracking issues, ensuring smooth project delivery.