Engineering Manager
CurrentResponsible for assessing client’s requirements & source applications, defining a design, technical architecture, implementation plan for the proposed design and delivering the solution in public cloud providers like Azure and GCP or in on-prem systems.Designed a Azure based Data Lake solution for one of the oil & gas companies in UK which includes services like ADLS-Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Databricks and Azure SQL Database.Worked on building a Delta Lake solution which is based on Lakehouse Architecture using Azure DatabricksDesigned a Governance solution which uses Microsoft Purview for creating Data Catalog for the Enterprise Data which helps client to get up-to-date map of enterprise data landscape with automated data discovery, getting sensitive data classification and getting end-to-end data lineageDesigned and developed business reports in Power BIWorked on implementing a Data Warehouse Solution for one of the payment gateway companies in US on their on-prem systems using Hive, Spark and UC4 scheduler. The same was then migrated to GCP based suing services like GCS, Big Query, Dataproc and Airflow.Worked on developing generic frameworks for Extraction, Ingestion and Transformation of the data. These frameworks are built in pyspark, python and shell scripts for both on-prem systems as well as cloud systems like Azure and GCPWorked on creating an Enterprise Data Model using Dimensional Modeling (Dimensions and Facts)Worked on multiple Automations to reduce the manual effortsWorked on creating and scheduling jobs in Schedulers (Airflow, UC4, Tivoli etc)Worked on Azure Monitoring which is used for collecting and analysing data about various Azure resources and the infrastructure on which these resources are runExperience in tuning and optimising the jobs that run over hadoop, hive, spark and cloud services for both Azure and SparkExperience on writing complex SQL scripts for various data transformation use cases