Shiva S Email & Phone Number

• Designed and built ETL pipelines using PySpark and SQL on AWS, processing large datasets for business intelligence. • Implemented CI/CD pipelines for automated testing and deployment of data workflows, improving system efficiency by 30%. • Collaborated with data scientists and business stakeholders to translate data requirements into scalable data solutions. • Optimized Apache Spark jobs for performance, reducing processing times by 25%. • Managed data quality and consistency across distributed systems, utilizing Databricks Delta Lake.

• Engineered and optimized data pipelines, resulting in a 30% improvement in data processing efficiency.• Implemented ETL processes using Python and Apache Airflow, reducing data latency by 25%.• Designed and maintained scalable data architectures on AWS, leading to a 20% reduction in storage costs.• Collaborated with cross-functional teams to integrate data from diverse sources, enhancing data accessibility by 40%.• Developed and managed data models in Snowflake, improving query performance by 50%.• Utilized SQL and NoSQL databases to support advanced analytics, increasing data retrieval speeds by 35%.Job Profile:• Developed and optimized data pipelines using Python, Java, and Scala to process large-scale datasets efficiently.• Deployed and managed data solutions on cloud platforms including AWS, Azure, and Google Cloud Platform (GCP) to ensure scalable and reliable data infrastructure.• Utilized Apache Spark, Hadoop, and Kafka for real-time data processing and analysis, enabling swift and informed decision- making.• Created automated workflows with Airflow and Autosys, integrating data from Snowflake, Redshift, and Teradata SQL Assistant to streamline data operations.• Implemented data storage and retrieval solutions using HDFS, Hive, Pig, and Sqoop within Hadoop distributions such as Hortonworks and Cloudera Hadoop.• Leveraged Spark, Spark SQL, and Kafka for efficient real-time data processing and analytics across Linux, Unix, ZOS, and Windows operating systems.• Designed and executed ETL workflows with IBM Infosphere Information Server, ensuring seamless data integration and transformation across various platforms.

• Designed and implemented data pipelines and ETL processes to handle over 5 TB of data daily, improving data processing efficiency by 20%.• Utilized Python and SQL to develop and maintain data integration solutions, ensuring seamless data flow between various systems.• Collaborated with cross-functional teams to gather and analyze requirements, resulting in the successful deployment of 15+ data projects.• Developed and maintained data models and schemas, enhancing data quality and consistency across multiple business units.• Managed and optimized data storage solutions using AWS services, reducing storage costs by 25% through effective data lifecycle management.Job Profile:• Designed and implemented data pipelines and ETL processes using Python, Scala, and SQL to handle large-scale data integration and processing.• Developed and maintained data integration solutions with Python and PL/SQL, ensuring seamless data flow and integration between diverse systems.• Collaborated with cross-functional teams to gather and analyze requirements, utilizing Java and C++ to build robust data solutions.• Managed and optimized data storage solutions using SQL and UNIX Shell Scripts to ensure efficient data retrieval and storage management.• Developed and maintained data models and schemas with PL/SQL, enhancing data quality and consistency across business units.• Implemented data validation and error-handling mechanisms using Python, reducing data discrepancies and ensuring data accuracy.

• Developed and maintained data warehouse solutions, ensuring data integrity and availability for over 1 million records daily.• Collaborated with cross-functional teams to gather and analyze business requirements, delivering data-driven insights that supported strategic decision-making.• Implemented Apache Spark for real-time data processing, enhancing data analytics capabilities across multiple projects.• Managed data pipelines using Airflow, ensuring smooth workflow orchestration and scheduling of data workflows.• Utilized Kafka for real-time data streaming, enabling seamless integration of data sources and applications.• Designed and optimized data models in Snowflake and Redshift, facilitating efficient querying and reporting.• Developed ETL workflows in Databricks, leveraging cluster computing for scalable data processing tasks.• Collaborated with cross-functional teams using GitHub, JIRA, Confluence, Trello, and Slack to ensure effective project management and streamlined communication.• Utilized Scala and Java for big data processing, optimizing performance and reducing data retrieval time by 25%.• Conducted data quality assessments and implemented data validation processes, leading to a 20% reduction in data discrepancies.• Created and maintained complex SQL queries and stored procedures to support various data extraction and transformation tasks.• Configured and managed Kubernetes clusters for container orchestration, ensuring high availability and scalability of microservices applications.• Implemented Terraform for infrastructure as code (IaC), automating the deployment and management of cloud resources across AWS and Azure environment

• Managed ETL processes, processing an average of 1.5 TB of data weekly from various sources into the data warehouse.• Developed and maintained data pipelines using Apache Spark, processing over 10 million records daily for analytics and reporting purposes.• Conducted performance tuning and troubleshooting of Hadoop ecosystem components (HDFS, MapReduce, Hive, HBase).• Utilized Python scripting for data manipulation and transformation tasks, enhancing data processing efficiency.• Participated in on-call rotation for production support, maintaining high availability and reliability of data systems.• Optimized SQL queries and database performance, achieving a 30% reduction in query execution time.• Collaborated with cross-functional teams to gather and analyze business requirements, resulting in improved data quality and accuracy.• Collaborated closely with data scientists to integrate machine learning models into data pipelines for predictive analytics.• Conducted data profiling and analysis to identify data quality issues and implement corrective actions.• Developed and maintained documentation for data processes, schemas, and workflows to ensure knowledge transfer and continuity.• Participated in Agile sprint planning and retrospectives, contributing to continuous improvement in team efficiency and delivery.• Provided technical guidance and mentoring to junior team members, fostering their professional growth and development.• Implemented data governance policies and procedures, ensuring compliance with industry standards and regulations.• Designed and implemented data models in PostgreSQL, supporting scalable and efficient data storage solutions.• Automated data validation and monitoring processes, reducing manual effort by 40% and improving data reliability.

Master Of Science - Ms, Information Technology, 4.0

Bachelor Of Technology, Computer Science, 74%