Software Engineer
CurrentProject-1:Technology Stack: Pandas, PySpark, Dataiku DSS, Postgres - Implemented a migration from Visual Recipes to Code Recipes, optimizing processing efficiency by achieving a notable 30% reduction in overall processing time.Project-2: Migration of ETL Pipelines from Dataiku to DatabricksTechnology Stack: Pandas, Databricks - Worked with the team and designed the ETL Architecture on Databricks. - Migrated existing ETL pipelines, scripts, and data workflows to Databricks, ensuring compatibility and optimization for Databricks' runtime. - Refactored and optimized ETL processes to leverage Databricks' distributed computing capabilities for improved performance. - Deployed the migrated ETL pipelines into production environments, establishing robust monitoring and logging mechanisms to ensure continuous management and effective troubleshooting.