Lead Solutions Architect
CurrentData Management Tool Integration: Built a web-based data management tool for visualizing and reporting data, integrating it with Tableau dashboards and multiple data layers (raw, processed, aggregate).Development of Automated Data Pipeline: Designed and implemented a Python-based data pipeline to process data from 100+ open-source data sources, incorporating data profiling, quality checks, and automated rectification.Creation of Data Quality Framework: Developed a comprehensive data quality framework, including unit tests, data profiling, and quality control mechanisms to ensure the accuracy and integrity of data layers.Data Governance and Remediation Techniques: Implemented dynamic data masking, data minimization, and governance rules to eliminate dark, dirty, and ROT (redundant, outdated, trivial) data, enhancing data security and compliance.Advanced Data Modeling and Segmentation: Developed data similarity and segmentation models to identify patterns and anomalies, enabling deeper insights across multiple data sources.Data Warehouse Design and Migration: Architected and managed a data warehouse on AWS Redshift, migrating data from SAP HANA, Retek Systems, LS Retail, and other sources using AWS Lake Formation, Glue, DMS, and Lambda.Legacy ETL Pipeline Modernization: Modernized legacy ETL pipelines by migrating from Informatica and Tibco to an AWS-managed stack, reducing licensing costs by 30% and improving overall system efficiency.Big Data Workload Optimization: Migrated on-premise big data workloads to AWS EMR, enhancing runtime performance by 70% through auto-scaling and parallel processing.
