• Design Data Lake storage solution for Data science Project using Azure Data factory Pipelines.• Build and develop data strategy and road map for Analytics stream.• Designed and implemented the Azure Data factory framework (V2) with Error logging to populate data in the Azure SQL Data warehouse from Azure Blob storage and Azure data lake store.• Implemented Automation for Azure services using azure runbooks, Azure logic apps.• Design and implemented Data sync between the… Show more • Design Data Lake storage solution for Data science Project using Azure Data factory Pipelines.• Build and develop data strategy and road map for Analytics stream.• Designed and implemented the Azure Data factory framework (V2) with Error logging to populate data in the Azure SQL Data warehouse from Azure Blob storage and Azure data lake store.• Implemented Automation for Azure services using azure runbooks, Azure logic apps.• Design and implemented Data sync between the On-premises SQL server database and Azure SQL databases to deliver Hybrid Data analytic solution.• Designed and built the Azure Data factory framework to ingest data from Office 365, Dynamics 365 data in Data Lake and modelled them as per requirement for the Data science project.• Implemented Azure DevOps (VSTS/TFS) branching and merging processes for team development in the Azure analysis service Tabular cubes, Azure SQL Database and Azure Data factory pipelines for Continuous Integration (CI).• Created pipelines, data flows and complex data transformations and manipulations using Azure Data Factory (ADF) and PySpark with Databricks.• Performed ETL using Azure Data Bricks. Migrated on-premises Oracle ETL process to Azure Synapse Analytics.• Involved in migration of large amount of data from OLTP to OLAP by using ETL Packages.• Configured and implemented the Azure Data Factory Triggers and scheduled the Pipelines, monitored the scheduled Azure Data Factory pipelines, and configured the alerts to get notification of failure pipelines. • Extensively used ETL to load data from flat file and from the relation database. Show less

• Developed data pipeline using Spark, Hive and HBase to ingest customer behavioral data and financial histories into Hadoop cluster for analysis.• Created database tables and stored procedures as required for reporting and ETL needs.• Databricks job configuration, Refactoring of ETL Databricks notebooks• Worked on managing the Spark Databrick on Azure.• Implemented data ingestion from various source systems using Sqoop and PySpark.• Created Key-Vaults with Secrets to… Show more • Developed data pipeline using Spark, Hive and HBase to ingest customer behavioral data and financial histories into Hadoop cluster for analysis.• Created database tables and stored procedures as required for reporting and ETL needs.• Databricks job configuration, Refactoring of ETL Databricks notebooks• Worked on managing the Spark Databrick on Azure.• Implemented data ingestion from various source systems using Sqoop and PySpark.• Created Key-Vaults with Secrets to maintain Passwords and changed Authentication method to connect to SQL DB from SQL Authentication to Service Principal.• Involved in extraction, transformation and loading of data directly from different source systems (flat files/Excel/Oracle/SQL) using SAS/SQL, SAS/macros.• Generated PL/SQL scripts for data manipulation, validation, and materialized views for remote instances.• Validation of data transformations and perform end-to-end data validation for ETL and BI systems.• Involved in development of test strategy, test plans and designs, execute test cases for ETL and BI systems.• Created partitioned tables in Hive, also designed a data warehouse using Hive external tables and created hive queries for analysis.• Good experience working on analysis tools like Tableau, Splunk for regression analysis, pie charts and bar graphs.• Created and modified several database objects such as Tables, Views, Indexes, Constraints, Stored procedures, Packages, Functions and Triggers using SQL and PL/SQL. Show less

• Created Linked services to connect the external resources to ADF.• Worked with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.• Used Azure DevOps and Jenkins pipelines to build and deploy different resources (Code and Infrastructure) in Azure.• Ensure the developed solutions are formally documented and signed off by business.• Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue… Show more • Created Linked services to connect the external resources to ADF.• Worked with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.• Used Azure DevOps and Jenkins pipelines to build and deploy different resources (Code and Infrastructure) in Azure.• Ensure the developed solutions are formally documented and signed off by business.• Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.• Worked on the cost estimation, billing, and implementation of services on the cloud.• Experience managing Azure Data Lakes (ADLS) and Data Lake Analytics and an understanding of how to integrate with other Azure Services.• Migration of on-premises data (Oracle/ Teradata) to Azure Data Lake Store (ADLS) using Azure Data Factory (ADF V1/V2).• Work closely across teams (Support, Solution Architecture) and peers to establish and follow best practices while solving customer problems.• Worked on Azure Data Factory (ADF) to build and execute data pipelines.• Worked on all types of transformations that are available in Power BI query editor.• Implement of custom database solutions that will extract large data sets from multiple sources. Integrate and stage data.• Prepare Customized reports based on Business requirements.• Troubleshoot and resolve bugs/issues with applications enhancing security and sustainability.• Perform quality checks on reports and extracts to ensure exceptional quality.• Involved in designing, developing, and testing of the ETL (Extract, Transformation and Load) strategy to populate the data from various source systems feeds using SSIS. Show less

• Designed and implemented end-to-end data pipelines for batch and real-time processing, integrating data from various sources such as databases, APIs, and logs.• Developed ETL workflows using tools like Apache NiFi, Apache Airflow, or custom scripts to extract, transform, and load data into target systems.• Automated data pipeline orchestration and monitoring using workflow management tools to ensure data integrity and timely processing.• Designing and creating Data Marts… Show more • Designed and implemented end-to-end data pipelines for batch and real-time processing, integrating data from various sources such as databases, APIs, and logs.• Developed ETL workflows using tools like Apache NiFi, Apache Airflow, or custom scripts to extract, transform, and load data into target systems.• Automated data pipeline orchestration and monitoring using workflow management tools to ensure data integrity and timely processing.• Designing and creating Data Marts, Databases, Indexes, Views, Aggregations, Stored Procedures, Partitions and Data Integrity.• Migrating Data from On-Perm SQL Server to Cloud databases in Azure.• Performing research to identify the source and nature of data required for ETL solutions.• Conducted performance tuning of Spark jobs and Hive queries, optimizing resource utilization and query execution times for improved efficiency.• Implemented data partitioning and clustering strategies to distribute data evenly across nodes and enhance parallel processing capabilities.• Migrate the entire CRM database present on the IBM DB2 servers, Informix to cloud based data warehouse called Snowflake using ETL DataStage.• Performing Data Validations of the DW using Power BI.• Identified data quality issues and design processes and procedures to improve/maintain quality.• Performance tuning of SQL queries, Data Pipelines, Tableau and Power BI Dashboards.• Performing Code release from one environment to another environment using release management in various environments. Show less