Senior Data Analytics Developer
Current• Microsoft Synapse & Microsoft Fabric: Proficient in leveraging Microsoft's premier analytics services to streamline and enhance big data processing workflows.• MSSQL to PySpark Framework: Created an innovative MSSQL2PySpark Migration framework using Large Language Models (LLM), Python, PySpark, Langchain, Ollama/Azure OpenAI to translate MS SQL queries into PySpark SQL, allowing for fast migration, smoother transitions and improved efficiency.• SSIS to PySpark Migration: Successfully migrated SSIS packages to PySpark, dramatically reducing query run times from over 10 minutes to a mere 25 seconds, thereby boosting performance and productivity.• PySpark Pipeline Optimization: Dedicated to continuous improvement of PySpark pipelines, enhancing their reliability and performance through meticulous tuning and optimization.• PySpark & ETL: Skilled in developing robust ETL pipelines using PySpark, ensuring seamless data integration and transformation using Azure Data Lake Storage Gen2 and OneLake to meet business needs.• AI-Driven Development: Leveraging AI and LLM to minimize coding effort, I develop solutions that are not only efficient but also cutting-edge, keeping pace with the latest advancements in technology.• ChatGPT Custom Portal Development: Implemented a custom ChatGPT portal using SharePoint documentation and added a backend database to keep history for future analysis, enhancing user interaction and access to information.