Shiva Kumar Email & Phone Number

Ohio, United States

In my role as a data engineer, I've focused on automating processes using Python scripting, particularly for script generation, and leveraging Azure Data bricks for data curation. I've built and managed ADF pipelines to extract and manipulate data from various cloud sources like Azure Blob Storage, Data Lake, Cosmos DB, and SQL Server.My expertise includes using Azure Data Lake and PolyBase for data retrieval, performing ETL processes with Azure Data bricks, and migrating Oracle ETL to Azure Synapse Analytics. I've utilized Matillion for Snowflake data integration and maintained robust ETL processes with Informatica.I've optimized data processing with Spark scripts and MapReduce on Hadoop, explored PySpark on Azure Data bricks for algorithm optimization, and designed scalable workflows for efficient data processing on Hadoop. Additionally, I've integrated Cassandra for NoSQL data analysis and managed metadata with Nebula Metadata.I've ensured timely data availability by loading diverse sources into Snowflake and developed and maintained data models in Power BI. Integration with Tableau facilitated comprehensive data analysis. Moreover, I've managed version control with GitHub and task tracking with Jira for collaboration and accountability.

San Antonio, Texas Metropolitan Area

I have extensive experience in designing and deploying AWS applications, utilizing services such as EC2, Route53, S3, RDS, Dynamo DB, SNS, SQS, and IAM. My focus includes ensuring high availability, fault tolerance, and auto-scaling through AWS Cloud Formation. I've been actively involved in migrating applications from traditional data centers to AWS, installing applications on EC2 instances, and configuring storage on S3 buckets.In terms of access management, I've managed IAM policies and contributed to refining workflows. I possess a strong understanding of ETL concepts and have used AWS Redshift, S3, Spectrum, and Athena to query large datasets without extensive ETL processes, creating a Virtual Data Lake.My expertise extends to optimizing Informatica workflows, Spark Streaming for real-time data, and developing PySpark applications for text data analysis. In the Hadoop ecosystem, I've optimized cluster performance, worked with Spark SQL and Scala queries, and fine-tuned configurations.I have experience with relational SQL and NoSQL databases, including Oracle, Hive, Sqoop, and HBase. Additionally, I've implemented and optimized data transformations within Snowflake.In the visualization space, I've developed efficient ETL processes for Power BI and maintained data preparation workflows in Tableau. For collaboration and version control, I've implemented branching strategies in GitHub. I've actively participated in sprint planning sessions in Jira, prioritizing and allocating resources for data engineering activities based on project requirements.

Mumbai, Maharashtra, India

I have hands-on experience working on AWS Data Pipeline, configuring data loads from S3 into Redshift, and managing data movement across various AWS services. Utilizing AWS Glue, I designed ETL processes to relocate Campaign information from diverse sources into AWS Redshift. My expertise includes Data Warehousing, Business Intelligence, and ETL processes using tools like Informatica and AWS Glue for large dataset analysis.I implemented best practices for data quality and governance in Informatica, ensuring compliance with industry standards. Using Spark scripts in Hadoop environments, I optimized data processing speed and maintained data security. I developed ETL pipelines in Python and Snowflake's Snow SQL, handling SQL queries against Snowflake.In terms of analysis, I built products and delivered insights using SQL in Snowflake and Amazon Redshift, along with visualization tools like Periscope and Tableau. I ensured data security within Snowflake, enforced access controls, and maintained confidentiality.I implemented data governance practices in Power BI, ensuring quality and compliance, and set up secure measures in Tableau for data integrity. For efficient testing and deployment, I established CI/CD pipelines in GitHub. Additionally, I maintained project documentation in Jira for effective project management.

Bengaluru, Karnataka, India

I have hands-on experience building Azure Web Jobs for data extraction and loading into Azure Data Warehouse, connecting with various APIs and sources. Additionally, I've built pipelines to integrate Azure Cloud with AWS S3 for seamless data transfer to Azure Database. My role involves researching and identifying data sources for ETL solutions, creating Azure Data Factory pipelines, and scheduling ETL jobs using SQL Agent.Collaborating closely with stakeholders, I've designed and implemented effective data integration solutions using Informatica and PowerCenter. I've also utilized Spark with Python and Spark SQL for faster testing and processing of data. My contributions extend to building scalable distributed data solutions using PySpark on Cloudera Hadoop through Azure Data Factory pipelines.In terms of data management, I've handled datasets using Panda data frames and MySQL, querying MySQL databases from Python. I've optimized Snowflake queries and workloads, as well as Power BI reports, dashboards, and Tableau queries for enhanced performance.Actively participating in code reviews on GitHub, I ensure code quality and consistency in data engineering practices. My collaboration within Jira is instrumental in identifying and resolving issues during the development process.