Creating and maintaining the ETL pipelines using ADF, Databricks, Cosmos DB, SQL DB, Azure Function apps etc.Performing full data load using ADF and incremental data load using Azure Functions into multiple targets like Delta Tables, Cosmos DB etc. based on the requirementsDeveloping multiple APIs using python and deploying the APIs in the ACA(Azure Container App) by following the microservices architecture.

Majorly worked on alternative data and geospatial data using various tools and technologies such as AWS S3, AWS RDS, AWS ECR, AWS Lambda Function, AWS Cloud Watch, Cron Jobs, MongoDB, PostgreSQL, Selenium, Beautiful Soup, APIs, python multiprocessing etc. for the day to day Data Engineering work.Created an end-to-end automated pipeline for an Indian Government Organization from the scratch.Implemented geo-hash, MongoDB geo-queries, log rotate for MongoDB logs and partitioning the data… Show more Majorly worked on alternative data and geospatial data using various tools and technologies such as AWS S3, AWS RDS, AWS ECR, AWS Lambda Function, AWS Cloud Watch, Cron Jobs, MongoDB, PostgreSQL, Selenium, Beautiful Soup, APIs, python multiprocessing etc. for the day to day Data Engineering work.Created an end-to-end automated pipeline for an Indian Government Organization from the scratch.Implemented geo-hash, MongoDB geo-queries, log rotate for MongoDB logs and partitioning the data on various levels based on the type of data.Worked on all types of data like structured, semi-structed and unstructured.Created an automated script to test all the DMS(Data Management System - Internal Data Warehouse) APIs using python decorators.Created/updated a middleware-cli and local python package so that users can use the DMS APIs easily and effectively.Helped to create an impact across the top public listed enterprises in India in BFSI,FMCG & CPG and it played a key role in enabling financial inclusion across the country Show less

I was working as an standalone Azure Data engineer.I had a hands on knowledge in ADF, databricks, Azure SQL DB, ADLS etc.I collaborate with the business stakeholders to understand the business requirements and based on that I create transformation scripts in pyspark/scala.I also had a good experience in working in agile methodology.I created many ETL pipelines in ADF, write transformation scripts in databricks using pyspark or Scala or spark SQL and I can also write SQL queries… Show more I was working as an standalone Azure Data engineer.I had a hands on knowledge in ADF, databricks, Azure SQL DB, ADLS etc.I collaborate with the business stakeholders to understand the business requirements and based on that I create transformation scripts in pyspark/scala.I also had a good experience in working in agile methodology.I created many ETL pipelines in ADF, write transformation scripts in databricks using pyspark or Scala or spark SQL and I can also write SQL queries efficiently to store transformed data into SQL DB. Later, the Power BI was connected to the Table where transformed data is kept for visualisation.I used to extract data from various sources like On-prem DBs using self hosted Integration Runtime, flat files, using APIs etc. Show less