Sai P. Email & Phone Number

United States

• Leveraged Azure Data Factory to seamlessly integrate on-premises databases (MS SQL, Cassandra) which are SQL, No SQL and cloud-based storage (Blob storage), applying advanced transformations by using Databricks, and loading data into Snowflake.• Developed custom Python, Spark, and Bash scripts to facilitate data transformation and loading across both on-premises and cloud platforms.• Developed and optimized ETL processes using PySpark for processing large datasets, enabling efficient data ingestion, transformation, and storage.Integrated Spark workflows with Azure Data Lake, Blob Storage, and Azure Synapse for scalable data processing, storage, and analysis solutions.• Created data pipelines in Azure Data Factory (ADF) using Linked Services, and Datasets to efficiently extract, transform, and load data from various sources including Azure SQL, Blob storage, Azure SQL Data Warehouse, and write-back tool.• Managed the seamless import and export of databases using SQL Server Integration Services (SSIS) and Data Transformation Services (DTS Packages).• Utilized Azure DevOps for managing source code, builds, releases, and infrastructure as code (IaC), leveraging tools such as ARM templates or Terraform.• Successfully migrated applications from Cassandra DB to Azure Data Lake Storage Gen A1 using Azure Data Factory.• Designed and implemented end-to-end IoT solutions on the Azure platform including Azure IoT Hub, Azure IoT Edge, Azure Time Series Insights, and Azure Stream Analytics, to enable seamless device-to-cloud and cloud-to-edge communication, encompassing Device Connectivity, Data Processing, storage, and insights generation.• Developed and implemented best practices, guidelines, and standards for Data Governance, Data Quality, and Data Integration.• Utilized networking concepts such as VPN, and firewalls to ensure secure and reliable communication in IoT environments.

California, United States

Highly skilled in developing ELT/ETL pipelines using Python and Snowflake Snow SQL. Expertise in creating ETL transformations and validations through Spark-SQL/Spark Data Frames, utilizing Azure Databricks and Azure Data Factory. Experienced in creating and optimizing complex SQL queries in Azure Synapse Serverless SQL Pools, providing meaningful insights to stakeholders. Created data pipelines in Azure Data Factory (ADF) using Linked Services, Datasets, and Pipelines to efficiently extract, transform, and load data from various sources including Azure SQL, Blob storage, Azure SQL Data Warehouse, and write-back tool. Skilled in utilizing Databricks Notebooks to develop and execute complex data transformation, machine learning, and advanced analytics tasks. In-depth, understanding of Apache sparks job execution components like DAG, lineage graph, DAG Scheduler, Task scheduler, and Stages and worked on relational, NoSQL databases.

California, United States

Designed and maintained data systems and databases, this includes fixing coding errors and other data-related problems. Utilized MongoDB for storing the data received from the trained ML modules. Worked on Data Mining, DB Maintenance, Data Preparation, Quality Assurance, calibrating with other teams, Report generation and preparation, Troubleshooting and Maintaining confidentiality of data. Prepared final analysis reports for the stakeholders to understand the data-analysis steps, enabling them to make important decisions based on various facts and trends.

India

• Leveraged Azure Data Factory to seamlessly integrate on-premises databases (MySQL, MS SQL, Cassandra) which are SQL and NO SQL and cloud-based storage (Blob storage, Azure SQL DB), applying advanced transformations, and loading data into Snowflake.• Created data pipelines in Azure Data Factory (ADF) using Linked Services, Datasets, and Pipelines to efficiently extract, transform, and load data from various sources including Azure SQL, Blob storage, Azure SQL Data Warehouse, and write-back tool.• Optimized code in Azure Functions for efficient data extraction, transformation, and loading from diverse sources, including databases, APIs, and file systems. Developed event-based triggers using Azure Event Grid for real-time processing and event-driven workflows and implemented tumbling window triggers with Azure Stream Analytics for time-based aggregations and insights.• Leveraged Azure HDInsight's and Azure data lake storage Gen2 for processing massive volumes of data.• Demonstrated experience in using Scala with Apache Spark for distributed data processing.• Leveraged Delta Live Tables in conjunction with Apache Kafka to perform streaming analytics, enabling real-time monitoring of key performance indicators.• Enhanced Azure Functions code to efficiently extract, transform, and load data from various sources such as databases, APIs, and file systems.• Orchestrated smooth data movement into SQL databases using Data Factory's robust data pipelines.• Implemented data warehousing techniques, including data cleansing, Slowly Changing Dimension (SCD) handling, surrogate key assignment, and change data capture, to enhance Snowflake modelling.• Designed and executed scalable data ingestion pipelines using Apache Kafka, Apache Flume, and Apache Nifi, effectively handling large volumes of data from various sources.• Utilized Azure Data Factory to create end-to-end ETL pipelines and streamlined data transformation and cleansing through Alteryx processes.

India

Implemented efficient solutions for harmonizing data from diverse sources, including databases, APIs, and file systems, utilizing tools such as Apache Kafka, and Apache Ni-Fi. Engineered solutions designed for effortless scalability, harnessing the elasticity of Serverless SQL Pools to adapt to varying demands. Conducted comprehensive testing of HDFS, Hive, Pig, and MapReduce to extend cluster access to new users. Employed Spark SQL for both Scala and Python interfaces, automating the conversion of RDD case classes to schema RDD. Implemented performance optimization strategies, including distributed cache usage for small datasets, efficient partitioning, bucketing in Hive, and map-side joins. Demonstrated in-depth knowledge of Spark platform parameters, including memory management, core utilization, and executor configuration. Pioneered the development of a reusable framework, automating ETL processes from RDBMS systems to the Data Lake by harnessing Spark Data Sources and Hive data objects. Managed the seamless import and export of databases using SSIS and DTS Packages. Managed the loading and transformation of structured, semi-structured, and unstructured data .Created and improved scalable ETL/ELT pipelines for processing and transforming massive datasets across several cloud environments utilizing GCP (e.g., Dataflow, Dataproc, BigQuery).Used AWS EMR Spark cluster and Cloud Dataflow on GCP to compare the efficiency of a POC on a developed pipeline.Created and improved scalable ETL/ELT pipelines for processing and transforming massive datasets across several cloud environments utilizing Google Cloud products (e.g., Dataflow, Dataproc, BigQuery). Used AWS EMR Spark cluster and Cloud Dataflow on GCP to compare the efficiency of a POC on a developed pipeline.Deployed containerized apps with Google Kubernetes Engine (GKE) that support multi clouds, enabling portable and scalable solutions.