Created numerous pipelines in Azure using ADFv2 to get the data from different source databases by using different Azure Activities like Move & Transform, Copy, filter, for each, Databricks etc. Orchestrated variety of solution in Azure Data factory by chaining required components (activities that use different computing platforms) that fulfill the desired requirement. Migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQLDB). Created, provisioned different Databricks clusters needed for batch and continuous streaming data processing and installed the required libraries for the clusters. Created Linked services to connect the external resources to ADF and monitoring ADF pipelines, creating incidents based on severity. Architect & implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/ Databricks, NoSQL DB). Involved in converting Hive/SQL queries into Spark transformations using Spark RDD and Pyspark concepts. Building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in Azure. Implemented big data technologies such as Hadoop, Map Reduce Frameworks, HBase, and Hive for ingesting data from diverse sources and processing Data-at-Rest. Ingested huge volume and variety of data from disparate source systems into Azure Data Lake Gen2 using Azure Data Factory V2. Configuring and managing Azure AD Connect, Azure AD Connect health, Microsoft Azure Active Directory. Enhancements to conventional data warehouses based on the STAR schema, data model updates, Tableau data analytics and reporting. Architect and implement ETL and data movement solutions using Azure Data Factory, SSIS create and run SSIS Package ADF V2 Azure-SSIS IR.

Perform ongoing monitoring, automation, and refinement of data engineering solutions, prepare complex SQL views, stored procs in azure SQL DW and Hyper scale. Designed and developed a new solution to process the NRT data by using Azure stream analytics, Azure EventHub, and Service Bus Queue. Created Linked service to land the data from SFTP location to Azure Data Lake. Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards. Extract Transform and Load (ETL) data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Design and developed Batch processing and real-time processing solutions using ADF, Databricks clusters and stream Analytics. Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different Azure Activities like Move & Transform, Copy, filter, for each, Databricks etc. Built the data pipeline using Azure Service like Data Factory to load the data from Legacy SQL server to Azure Data Base using Data Factories, API Gateway Services, SSIS Packages, Talend Jobs, custom .Net and Python codes. Migrating on-prem ETLs from MS SQL server to Azure Cloud using Azure Data Factory and Databricks. Designing and deployment of Hadoop cluster and different big data Analytic tools including pig, hive, HBase, Oozle, Zookeeper, SQOOP, flume, Spark, Impala, Cassandra with Horton work Distribution. Created Pyspark data frames in Azure Databricks to read the data from Data Lake and use Spark SQL context for transformation. Developed facts and dimensions tables using star and/or snowflake schema. Designed one-time load strategy for moving large databases to Azure SQL DWH.

Building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation. Data Extraction, aggregations, and consolidation of Adobe data within AWS Glue using PySpark. Analyzed large and critical datasets using HDFS, HBase, Hive, HQL, PIG, Sqoop and Zookeeper. Developing Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats. Building a Scala and spark based configurable framework to connect common Data sources like MYSQL, Oracle, Postgres, SQL Server, Salesforce, Big query and load it in big query. Creating, maintaining, support, repair, customizing System & Splunk applications, search queries and dashboards. Worked with SSIS tool suite, designed, and created mapping using various SSIS transformations like OLEDB command, Conditional Split, Lookup, Aggregator, Multicast and Derived Column. Design and develop new Power BI solutions and migration reports from SSRS. Creating rules Engine in Apache Strom to categorize the alarm into Detection, Interrogation &, Association types before processing of alarms. Designed and Implement test environment on AWS and Implemented AWS solutions using EC2, S3, EMR, Redshift, RDS, Elastic Load Balancer, Athena etc. Performed Data Aggregation, Validation and on Azure HDInsight using spark scripts written in Python. Analyzed source data coming from different sources like SQL Server tables, XML files and Flat files then transformed according to business rules using Informatica and loaded the data in to target tables. Created various types of reports such as complex drill-down reports, drill through reports, parameterized reports, matrix reports, Sub reports, non-parameterized reports and charts using reporting services based on relational and OLAP databases.

Creation, manipulation and supporting tech SQL Server databases. Involved in tech Data modelling, Physical and Logical Design of Database. Helped in integration of tech front end Wi-Fi tech SQL Server backend. Created Stored Procedures, Triggers, Indexes, User defined Functions, Constraints etc. on various database objects to obtain tech required results. Import & Export of data from one server to other servers using tools like Data Transformation Services (DTS). Wrote T-SQL statements for retrieval of data and involved in performance tuning of TSQL queries. Transferred data from various data sources/business systems including MS Excel, MS Access, Flat Files etc. to SQL Server using SSIS/DTS using various features like data conversion etc. Also Created derived columns from tech present columns for tech given requirements. Supported team in resolving SQL Reporting services and T-SQL related issues and Proficiency in creating different types of reports such as Crosstab, Conditional, Drill-down, Top N, Summary, Form, OLAP and Sub reports, and formatting them. Created logging for ETL load at package level and task level to log number of records processed by each package and each task in a package using SSIS. Developed, monitored, and deployed SSIS packages. Created different Parameterized Reports (SSRS 2005/2008) which consist of report Criteria in various reports to minimize tech report execution time and to limit tech no of records required. Worked on all types of report types like tables, matrix, charts, sub reports etc. Created Linked reports, Ad-hoc reports etc. based on tech requirement. Linked reports are created in tech Report Server to reduce tech repetition tech reports.

Database development experience with Microsoft SQL Server in OLTP/OLAP environments using integration services (SSIS) for ETL (Extraction, Transformation and Loading). Created ETL Projects using SSIS 2012 in Package Deployment Model as well as Project Deployment Model, also efficiently used. SSIS catalog for ETL Monitoring. Developed SSIS packages that fetches files from FTP, did transformation on those data base on business need before I loaded to destination. Creating Metadata tables to log the activity of the packages, errors and change of the variable. Various techniques such as. CDC, SCD and Hash bytes were used to capture the change of the data and execute incremental loading of the dimension tables. Responsible for Deploying, Scheduling Jobs, Alerting and Maintaining SSIS packages. Implementing and managing Event Handlers, Package Configurations, Logging, System and User-defined Variables, Check. Points and Expressions for SSIS Packages. Automating process by creating jobs and error reporting using Alerts, SQL Mail Agent, FTP and SMTP. Developed, tested, and deployed all the SSIS packages using project deployment model in the 2016 environment by configuring the DEV, TEST and PROD Environments. Created SQL Server Agent Jobs for all the migrated packages in SQL Server 2016 to run as they were running in the 2014 version. Created shared dimension tables, measures, hierarchies, levels, cubes, and aggregations on MS OLAP/ Analysis Server (SSAS). Created a cube using multiple dimensions and modified the Relationship between a Measure Group and a Dimension, created calculated members and KPI' s, using SSAS. Created aggregations, partitions, KPI', s and perspectives for a cube as per business requirements. Involved in developing ASP.NET Web API Application that required integrating the Applications with Service Now using SNOW.API for creating Incidents and deploying in Azure.