Big Data Engineer
Current1.Spearheaded a transformative project at Celebal Technologies within one of the foremost financial services groups, overseeing the migration of over 30 ETL jobs to PySpark and SQL. Executed a seamless transition, enhancing efficiency by 30% and reducing processing time by 25%, contributing significantly to the company's data engineering capabilities.2. Contributed to the R&D team driving strategic integration of Microsoft Fabric analytics suite. Aligned features with industry needs and… Show more 1.Spearheaded a transformative project at Celebal Technologies within one of the foremost financial services groups, overseeing the migration of over 30 ETL jobs to PySpark and SQL. Executed a seamless transition, enhancing efficiency by 30% and reducing processing time by 25%, contributing significantly to the company's data engineering capabilities.2. Contributed to the R&D team driving strategic integration of Microsoft Fabric analytics suite. Aligned features with industry needs and streamlined governance processes by implementing a new decision-making framework, resulting in a 40% reduction in approval time for project initiatives and improved cross-functional collaboration.3. Provided valuable expertise in Slowly Changing Dimensions and Change Data Capture. Enhanced team efficiency with optimized Spark code and Autoloader for scalable data processing.4. Played a pivotal role in a Proof of Concept (PoC) and prepared Unity Catalog to build a business glossary. This catalog standardized business terminology during the SAP to Databricks transition, promoting robust data governance, improving communication, and streamlining data asset management, ultimately boosting overall operational efficiency.5. Leveraged the Medallion pattern in a project, harnessing Azure Databricks, Delta Lake, and Data Factory to ingest, transform, and deliver enriched data for advanced analytics and reporting. Show less
