Krishna M Email & Phone Number

Minnesota, United States

• Gathering data and business requirements from end users and management. Designed and built data solutions to migrate existing source data in Data Warehouse to Atlas Data Lake (Big Data)• Performed data migrations from on-prem to Azure Data Factory using ETL by using Azure Dataflow which is an integral component of Azure Datafactory. • Used activities such as DELETE, METADATA, COPY, IF-CONDITION, and LOOKUP for building ingestion pipelines on Azure Datafactory.• Used the bulk load… Show more • Gathering data and business requirements from end users and management. Designed and built data solutions to migrate existing source data in Data Warehouse to Atlas Data Lake (Big Data)• Performed data migrations from on-prem to Azure Data Factory using ETL by using Azure Dataflow which is an integral component of Azure Datafactory. • Used activities such as DELETE, METADATA, COPY, IF-CONDITION, and LOOKUP for building ingestion pipelines on Azure Datafactory.• Used the bulk load feature to download entire historical data into Azure blob using Azure Datafactory. Built Databricks external tables on top of it.• Inserted records into Salesforce using Azure datafactory, used UPSERT feature to update and insert records on Salesforce through ADF pipelines. The pipelines are metadata driven using metadata tables in SQLserver.• Used Azure DevOps pipelines for CICD to deploy code from Dev to QA and then to prod. Used Azure test plans tab to implement unit-testing at DEV level and integration testing on QA and Prod. Implemented custom test configuration for regression and functional testing.• Ingested data from SFTP and API’s through Azure Datafactory and ingested the data into blob storage.• Implemented parallel copy of all tables from Source database using ADF and metadata tables and created external Databricks Deltalake tables on top of it.• Used Azure Data Factory Pipelines to move data from Hadoop system to Azure Data Lake Storage. Handled Azure Resource Manager (ARM) in Deploying, Updating, and Deleting all the resources and even managing after the Deployment.• Analyzed huge volumes of data Devised simple and complex HIVE, and SQL scripts to validate Dataflow in various applications. Performed Cognos report validation. Made use of MHUB for validating Data Profiling & Data Lineage. Show less

California, United States

• Analyze, design, and build Modern data solutions using Azure PaaS service to support visualization of data. Understand the current Production state of the application and determine the impact of new implementation on existing business processes.• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake,… Show more • Analyze, design, and build Modern data solutions using Azure PaaS service to support visualization of data. Understand the current Production state of the application and determine the impact of new implementation on existing business processes.• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.• Implemented Proof of concepts for SOAP & REST APIs and REST APIs to retrieve analytics data from different data feeds.• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and backward.• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into customer usage patterns.• Responsible for estimating the cluster size, and monitoring and troubleshooting of the Spark data bricks cluster.• Experienced in performance tuning of Spark Applications for setting the right Batch Interval time, the correct level of Parallelism, and memory tuning.• Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the SQL Activity.• Hands-on experience in developing SQL Scripts for automation purposes.• Created Build and Release for multiple projects (modules) in a production environment using Visual Studio Team Services (VSTS). Show less