Rakesh C Email & Phone Number

United States

Contributed to the development of PySpark Data Frames in Azure Data bricks to read data from Data Lake or Blob storage and utilize Spark SQL context for transformation.Developed ETL pipelines in and out of data warehouse using a combination of Python and Snowflake. Used Snow SQL to write SQL queries against Snowflake.Worked on an Azure copy to load data from an on-premises SQL server to an Azure SQL Data warehouse.Integration of data storage solutions in spark - especially with Azure Data Lake storage and Blob snowflake storage.Developed data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQL.Designed and implemented a real-time data streaming solution using Azure EventHub.Developed Spark Streaming applications to process real-time data from various sources, such as Kafka and Azure Event Hubs.Built streaming ETL pipelines using Spark Streaming to extract data from various sources, transform it in real- time, and load it into a data warehouse like Azure Synapse Analytics.Developed Spark APIs to import data into HDFS from Teradata and created Hive tables.Loaded data into Parquet Hive tables from Avro Hive tables.Demonstrated proficiency in programming languages like Python and Scala.Performed various types of joins on Hive tables and implemented Hive SerDe like Avro, JSON.Developed Spark core and Spark SQL scripts using Scala for faster data processing.Worked on the Hadoop ecosystem in PySpark on HDInsight and Databricks.Created reports in Power BI with reference to wire frame documentation.Implemented DAX expressions for MTD and YTD based on slicer selection.Configuration of RLS (Row Level Security) for specified users with the help of specific user selection in Power BI service.Configuration of Power BI service account after publishing reports Extensively utilized Spark core, Spark SQL, and Spark Streaming for real-time data processing.Conducted data profiling and transformation on raw data using Python.

Texas, United States

Used Python scripts to update content in the database and manipulate files.Created data mapping, data governance, transformation and cleansing rules involving OLTP and OLAP.Developed logical data model based on the requirements utilizing Erwin.Automated Landing Zone and processed zone layers on Azure storage accounts by orchestrating Azure data factory pipelines.Identified the dimensions along with the measures and fact on the top of OLTP source.Developed data mapping, data governance, transformation and cleansing rules involving OLTP and OLAP.Authored multitude of ETL packages to extract data from heterogenous OLTP sources such as flat files, excel files, SQL Server tables to populate dimensional data mart.Developed pipelines in Azure data factory for migrating data from SQL server to synapse tables.Worked on different load strategies like SCD 1, SCD2 and SCD 0 for loading data from different sources to synapse tables.Handled different types of files like JSON, fixed width, text files with header and without header, excel files and different types of delimited files in Azure data factory.Implemented notebooks in Azure data bricks for transforming complex files which can’t be handled in Azure Data Factory.Orchestration of master pipeline by utilizing parameters and metadata driven solutions using control tables from Azure SQL DB where details of files will be stored to control the pipeline.Loading audit details like pipeline parameters, run id, timestamp, and pipeline details into Audit tables by using pipeline system variables and activity outputs in ADF.Handling code in Azure data bricks platform by utilizing spark SQL and pyspark functionalities.Deploying ADF code and SQL scripts from one environment to another with the help of Azure devops platform.Developed and implemented data cleansing, data security, data profiling and data monitoring processes.Created several views on top of delta lake files in serverless SQL Pool Synapse.

Norfolk, Virginia, United States

Created pipeline in Azure Data Factory (ADF) for migrating history and incremental data from File Share on premises system to ADLS Gen 2 storage account by creating pipelines in Azure data factory which is called as Bronze Layer Created metadata driven pipeline for loading data from File Share system which is completely parameterized.Utilized different functionalities of Azure data factory like parameters, expressions in Azure data factory, and variables while creating pipelines.Created notebooks in azure data bricks for migrating parquet files generated by the above pipeline to the form of delta files using SPARK SQL and Pyspark scripts in Data Bricks which is called as Silver LayerCreated SQL scripts and stored procedures to insert and update data in Audit table which is holding all the pipeline details and load status.After completing Silver Layer developed several Data Bricks notebooks based on business requirements which will load data using different load strategies like SCD 1 and SCD2.This final layer is called as gold layerCreated views on top of delta files. These views will be used as source in Power BIValidating and unit testing data from source to target based on the mapping sheet by creating complex SQL queries using cte’s, analytical functions and Aggregate functions in SQL.Involved in creating data model along with data architect.Created mounting path code to access data from ADLS Gen2 and connections in Azure data Factory to call notebooks wherever necessary.Fixing issues related to Azure data bricks notebooks and debugging the solutions.After Gold layer started involving in several Power BI report creation to get clear visuals which will be utilized by end user.Designed ETL strategies to populate the data warehouse/data-mart with facts and dimensions.Deployed a code using Azure Devops by merging feature branch to master, raising PR, and executing CI CD pipelines.

Irving, Texas, United States

Created Pipelines in ADF using Linked Services, Datasets and Pipeline to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Datawarehouse, write-back tool, and backward. Extracted, Transformed and Loaded data from Sources Systems to Azure Data Storage services using a combination of ADF, Spark SQL, and U-SQL Azure Data Lake Analytics.Data is Ingested to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. Worked on Azure Services like IaaS, PaaS and worked on storage like Blob (Page and Block), SQL Azure.Implemented OLAP multi-dimensional functionality using Azure SQL Data Warehouse. Retrieved data using Azure SQL and Azure ML which is used to build, test, and predict the data. Worked on Cloud databases such as Azure SQL Database, SQL managed instance, SQL Elastic pool on Azure, and SQL server. Architect & implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB).Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.Involved in Migrating Objects from Teradata to Snowflake and created Snow pipe for continuous data load. Increased consumption of solutions including Azure SQL Databases, Azure Cosmos DB and Azure SQL.Created (CI/CD) pipeline on Azure that helps to automate steps in the software delivery process. Deploying and managing applications in Datacenter, Virtual environment, and Azure platform as well. Involved in converting Hive/SQL queries into Spark transformations using Spark RDD's and PySpark. Handled importing of data from various data sources, performed transformations using Hive, and loaded data into HDFS. Design, development, and implementation of performant ETL pipelines using PySpark and Azure Data Factory.

India

Worked on new Data warehouse design for ETL and Reporting projects using SSIS and SSRS.Implemented indexes such as clustered index, non-clustered index, covering index appropriately on data structures to achieve faster data retrieval.Worked with Data Governance tools and extract-transform-load (ETL) processing tool for data mining, data warehousing, and data cleaning using SQL. Optimized SQL performance, integrity, and security of the project’s databases/schemas. Performed ETL operations to support incremental, historical data loads and transformations using SSIS.Created SSIS packages to extract data from OLTP to OLAP systems and scheduled jobs to call the packages. Implemented Event Handlers and Error Handling in SSIS packages and notified process results to various user communities. Designed SSIS packages to import data from multiple sources to control upstream and downstream of data into SQL database. Used various advanced SSIS functionalities like complex joins, conditional splitting, column conversions for better performance during package execution. Developed impactful reports using SSRS, MS Excel, Pivot tables and Tableau to solve the business requirements. Maintained the physical database by monitoring performance, integrity and optimize SQL queries for maximum efficiency using SQL Profile.Worked on formatting SSRS reports using the Global variables and expressions.Created data maintenance plan for backup and restore and update statistics and rebuild indexes.Extracting data from multiple sources like Excel Files, SQL Server and Oracle transforming, cleansing, validating data and loading into final Destination. Designed packages Using Control Flow Tasks Sequential Container, Execute SQL Task and Data Flow Task and Send Mail Task.Extracted data from sources and transformed the data using different transformations like data conversion, derived columns, look up, Conditional Split, Union all, merge join and multi cast transformations.

India

Involved in analysis, design, implementation, and testing phases of Software Development Life Cycle (SDLC) using Agile methodology.Hands on experience in Extract, Transform & Load (ETL) development using SQL Server Integration Services (SSIS).Designed and developing SSIS packages using different transformations like Conditional Split, Multicast, Union-All, Merge, Merge Join and Derived Column.Experience on scheduling ETL jobs through SQL server agent.Experience on working with different databases like SQL server, Oracle. Handling various kinds of data from heterogeneous sources (SQL server, Oracle, Flat File, Excel).Hands on experience development and testing on SSIS packages.Worked in creation of Sequences for automatic generation of students registering.Created indexes on tables for optimizing stored procedure queries.Participated in Performance Tuning of SQL queries using Explain Plan to improve the performance of the application.Worked extensively on exception handling for handling errors using system defined exceptions and user defined exceptions. Developed number of Reports like parameterized reports, cross tab reports, and sub reports according to the client requirement using Reports 6i.Documented all the technical work for future references and customizations.