Nagendra Babu Email and Phone Number

Led the design, development, and implementation of robust and scalable data pipelines, leveraging technologies such as AWS Glue, AWS Data Pipeline, Apache Spark, and Apache Hadoop to process large volumes of data efficiently.Architected and managed cloud infrastructure on AWS, including EC2 instances, S3 buckets, RDS databases, Lambda functions, and IAM roles, ensuring high availability, reliability, and security.Utilized AWS services such as SQS and SNS for asynchronous messaging and… Show more Led the design, development, and implementation of robust and scalable data pipelines, leveraging technologies such as AWS Glue, AWS Data Pipeline, Apache Spark, and Apache Hadoop to process large volumes of data efficiently.Architected and managed cloud infrastructure on AWS, including EC2 instances, S3 buckets, RDS databases, Lambda functions, and IAM roles, ensuring high availability, reliability, and security.Utilized AWS services such as SQS and SNS for asynchronous messaging and event-driven architectures, enhancing system decoupling and fault tolerance.Developed ETL (Extract, Transform, Load) workflows employing AWS Glue, Apache Spark, and SQL to extract, transform, and load data from diverse sources into data warehousing solutions like Redshift and MySQL.Developed and maintained data processing workflows using AWS Step Functions, orchestrating complex tasks across different AWS services.Monitored and optimized system performance using CloudWatch metrics and alarms, identifying and resolving issues to ensure optimal data processing and resource utilization.Applied advanced data analysis techniques with Python libraries like Pandas and NumPy to derive actionable insights from structured and unstructured data sets.Designed and enforced data governance policies, including schema validation using XML, XSD, and XSLT, to ensure data quality and compliance with industry standards.Managed relational and NoSQL databases such as MySQL and DynamoDB, optimizing schema designs and query performance for efficient data retrieval and storage.Configured and managed containerized applications using Docker and orchestration tools like Kubernetes, ensuring seamless deployment and scalability.Implemented Ansible for automating infrastructure provisioning and managing configuration, optimizing deployment procedures, and maintaining uniformity across different environments. Show less

Designed and implemented high-performance data pipelines using Apache Spark for large-scale data processing on Azure Databricks.Azure Data Factory (ADF) was used to orchestrate data movement and ETL workflows across various data sources.Leveraged Impala for querying data stored in Apache HDFS within the Hadoop ecosystem.Designed and managed relational databases on SQL Server for structured data storage and analysis.Implemented data pipelines for interaction with NoSQL databases… Show more Designed and implemented high-performance data pipelines using Apache Spark for large-scale data processing on Azure Databricks.Azure Data Factory (ADF) was used to orchestrate data movement and ETL workflows across various data sources.Leveraged Impala for querying data stored in Apache HDFS within the Hadoop ecosystem.Designed and managed relational databases on SQL Server for structured data storage and analysis.Implemented data pipelines for interaction with NoSQL databases like MongoDB and Snowflake for flexible data storage.Utilized Pandas and NumPy libraries in Python for data manipulation, analysis, and preparation for machine learning models built with PyTorch.Developed and optimized MapReduce jobs on Hadoop clusters for large-scale data processing tasks.Utilized Git for version control and collaboration on data engineering projects within the team.Managed the software development lifecycle using Azure DevOps for continuous data pipeline integration and deployment (CI/CD).Automated cloud infrastructure provisioning and management in Azure using Terraform (IaC).Integrated Apache Kafka for real-time data streaming and ingestion into data pipelines.Created data visualizations and dashboards using Power BI to communicate data insights effectively.Implemented and monitored the ELK Stack (Elasticsearch, Logstash, Kibana) for log management, analysis, and visualization.Followed Agile methodologies like Scrum, collaborating effectively with cross-functional teams to deliver data-driven solutions.Utilized JIRA for issue tracking and project management within the data engineering team.Presented data findings and technical solutions to stakeholders clearly and concisely. Show less

Designed and implemented data pipelines for data ingestion, transformation, and loading (ETL) on Google Cloud Platform (GCP) using tools like Cloud Dataprep and Dataflow.Built and managed big data solutions on platforms including Apache Hadoop (CDH), Apache Spark, and PySpark for scalable data processing.Utilized Sqoop and Pub/Sub for secure data transfer between various data sources and cloud storage.Developed and optimized data models for data warehouses like PostgreSQL using… Show more Designed and implemented data pipelines for data ingestion, transformation, and loading (ETL) on Google Cloud Platform (GCP) using tools like Cloud Dataprep and Dataflow.Built and managed big data solutions on platforms including Apache Hadoop (CDH), Apache Spark, and PySpark for scalable data processing.Utilized Sqoop and Pub/Sub for secure data transfer between various data sources and cloud storage.Developed and optimized data models for data warehouses like PostgreSQL using expertise in relational databases.Utilized Python libraries such as NumPy, Pandas, and Scikit-Learn to conduct data manipulation analysis and perform various machine-learning tasks.Engineered data pipelines using Scala and PySpark to handle complex data transformations and computations within the Apache Spark ecosystem.Implemented data integration solutions using tools like Informatica for seamless data flow between heterogeneous systems.Developed and maintained RESTful APIs for data access and integration with external applications.Created data visualizations and dashboards using Tableau to communicate data insights effectively.Performed data exploration and analysis using Splunk for log management and security investigations.Followed Agile methodologies like Scrum, collaborating effectively with cross-functional teams to deliver data-driven solutions.Utilized GitLab CI for continuous data pipeline integration and deployment (CI/CD), ensuring automated testing and delivery.Managed project dependencies and builds using Maven for efficient development workflows.Documented data pipelines, architectures, and procedures with clarity for maintainability.Presented data findings and technical solutions to stakeholders clearly and concisely.Continuously monitored and optimized data pipelines for performance and scalability on the cloud platform.Utilized Confluence for knowledge sharing and collaboration within the data engineering team. Show less

Designed and implemented high-performance data pipelines using SQL, PL/SQL, Python, and SSIS for data ingestion, transformation, and loading (ETL) processes.Built and maintained scalable big data architectures on platforms like Hadoop, Spark, and Hive for efficient data storage and processing.Leveraged Sqoop to securely transfer data between relational databases and Hadoop Distributed File System (HDFS).Developed and optimized data models for data warehouses like Oracle using… Show more Designed and implemented high-performance data pipelines using SQL, PL/SQL, Python, and SSIS for data ingestion, transformation, and loading (ETL) processes.Built and maintained scalable big data architectures on platforms like Hadoop, Spark, and Hive for efficient data storage and processing.Leveraged Sqoop to securely transfer data between relational databases and Hadoop Distributed File System (HDFS).Developed and optimized data models for data warehouses like Oracle using knowledge of relational database concepts.Utilized NumPy and Pandas libraries in Python for data manipulation, analysis, and visualization.Engineered data pipelines on the AWS cloud platform, leveraging services like EC2 for compute instances, S3 for object storage, RDS for managed databases, and Lambda for serverless functions.Implemented robust data quality checks and cleansing routines to ensure data accuracy and consistency.Employed Git for version control and collaboration on data engineering projects.Logged and tracked issues using Bugzilla to resolve data pipeline problems efficiently.Worked effectively within Agile and Kanban methodologies, collaborating with data scientists and analysts to deliver data-driven solutions.Utilized SharePoint for document management and collaboration within the data engineering team.Performed data analysis and exploration using Excel for specific tasks and reporting needs.Documented data pipelines and architectures with clarity and detail.Presented data insights and technical solutions to stakeholders effectively.Continuously monitored and optimized data pipelines for performance and scalability. Show less