Sr. Software Engineer
Current● Created automated cloud optimization script to efficiently detect and delete unused resources on regular schedule, saving costs through metadata scanning APIs in Databricks and Azure Policy.● Developed flexible big data transformation pipelines using standardized, reusable Databricks notebooks with SQL, Scala, and Python.● Designed new Azure data architecture for large-scale daily integrations from critical systems; delivered pipeline with resilience to failures—coordinated builds across Synapse and Databricks.● Pioneered first data migration across Azure subscribers to expand business continuity infrastructure after partnering with Microsoft to develop custom solutions.● Introduced versioning in the SpringBoot REST API for the team to be able to deploy and release new features in waves with backward compatibility. ● Extended data ingestion capabilities to support Avro and other formats providing flexibility to handle growing unstructured data beyond existing Parquet pipelines.● Diagnosed cloud container connectivity issues and high transient load causing frequent system failures, reduced by optimizing cluster configurations and fault tolerance.