Worked as Data Engineer II for following: -• Created AWS Database Migration Service (DMS) based generic ingestion pipeline for Finance Data Lake project. Created EMR based framework for further data cleanup, deduplication, and transformations. Implemented auto reconciliation and alerting for the same. Orchestrated the entire logic via Apache Airflow.• Developed a Lambda-based framework to dynamically enable data sharing across three AWS Redshift databases, encompassing over 350 tables. This initiative also involved configuring custom CloudWatch metrics, alarms, and alerts to ensure seamless operation.• Led the initiative to attain GDPR compliance for Vendor payment related data mart. Analysed over 200 dimension and fact tables to find PII/PCI columns and created dynamic SQL based framework to implement data retention and deletion strategy. Co-ordinated with over 30 teams to ensure compliance among all data consumers.• Migrated Redshift based SQL scripts to AWS EMR based Spark SQL code to decrease overall processing time from 8 hours to 30 minutes at 60% lower cost.

Role – Cloud Data Warehouse Architect, Domain: Subscription/Usage analytics• Led the migration of an on-premises usage data warehouse to Azure Cloud as a Senior Data Engineer.• Implemented near real-time ELT pipelines utilizing Kafka, Event Hub, Azure Functions (Python), and Azure Data Factory to ingest streaming data into Synapse at hourly intervals. Migrated 150+ tables from on-prem SQL server based DW to Synapse SQL DW.• Designed and optimized Synapse table structures, enhancing data load and retrieval performance through distribution and partition strategies.• Modernised Azure Functions-based solutions into Spark-based solutions to efficiently handle high data volumes, significantly reducing operational costs.• Introduced data virtualization in Azure using third party tool Tibco (TDV) in conjunction with Synapse and Azure SQL server.• Conducted multiple proof-of-concept evaluations for technologies including TDV, Kyligence, Prefect, Airflow, Azure Batch, Event Grid, Logic Apps, etc., contributing to the technical architecture design of Factset's Usage Cloud Data Platform.• Mentored a team of four members in various tasks related to platform development.

Project - Enterprise Data PlatformRole – Manager, Senior Data Engineer, Domain: Retail (Sales, Inventory, Customer data)• Proactively participated in the development of Enterprise Data Platform for Gap, which emulates as a source for most of the BI projects in cloud using Azure Data Lake, Kafka, Azure Data bricks to handle all complex transformations, ADF Pipelines, ADW, Azure SQL DB etc.• Automated ADF pipelines to optimize the existing resources and to retire few of the complex SQL SP's by building it in Databricks.• Created Azure functions based hydration tool to pick up and convert streaming thin events from event hub into thick events.• Designed data model of Raw, Transformation and Consolidation layers of Enterprise data warehouse for online and stores Sales, Inventory and Forecast data.• Created Power BI dashboards on top of cloud datasets for Enterprise reporting.Project - Stores Reporting Mobile App – MyInsightsRole – Manager, Senior Data Engineer, Domain: Stores Sales reporting (Sales, Inventory, Traffic, Labor data etc.)• Led team of 6 to develop and maintain React JS based mobile application for Stores Reporting. Handled backend data pipelines development as Senior Data Engineer and created high level and low-level designs for the entire backend architecture.• Co-ordinated with different teams and business partners for requirement gathering and project implementation. Performed detailed milestone and sprint plans for various releases.• Built ETL pipelines to source data from On Prem Teradata to Azure SQL DB. Performed Batch Automation of pipelines using Shell and Python scripts in CAWA. Implemented overall orchestration in Azure Data Factory using a generic framework.• Created custom dynamic SQL logic to process different metrics at various levels of company hierarchy.• Designed REST API structure for the App and modelled data to provide compatibility with APIs.

MSBI Developer - SSIS, SSAS, SSRS & PPS

Analytical Instrumentation