Designed and implemented scalable ETL pipelines using Azure Data Factory to automate data ingestion from various sources, including sales systems and external APIs. Developed complex data transformations in Databricks with PySpark and Spark SQL, enabling faster insights for sales analytics and agricultural research.Optimized Delta Lake architecture for efficient data retrieval and enhanced performance through advanced Spark techniques like caching and partitioning. Implemented RBAC policies with Azure Active Directory to improve security and access management.Managed cloud infrastructure using Azure Monitor and Log Analytics, proactively identifying performance bottlenecks. Led the implementation of Unity Catalog for centralized data governance and ensured compliance with corporate security policies.

Led the architecture and implementation of end-to-end data pipelines using Spark and Python, ensuring efficient processing of large datasets. Automated data ingestion workflows with Azure Data Factory, significantly enhancing processing efficiency. Managed and optimized data storage solutions in Azure Data Lake Storage (ADLS) for various data formats, including XML, JSON, AVRO, Parquet, and CSV.Leveraged Databricks for advanced analytics and machine learning, implementing data governance practices to ensure secure data access. Administered Azure cloud infrastructure, including Virtual Networks, VPN Gateway, ADLS, and ExpressRoute, for secure communication between cloud services and on-premise systems. Developed CI/CD pipelines for data projects, managed Azure Key Vault for sensitive data, and configured Azure Monitor for comprehensive system monitoring.

In this role, I designed and scheduled Spark jobs to source data from various Azure-based systems, applying transformations and loading datasets into target storage systems. I led multiple data migration projects, orchestrating the transfer of data from Azure SQL Database to Azure Synapse Analytics (formerly Azure SQL Data Warehouse), and optimized data warehousing solutions by implementing best practices to enhance performance. Additionally, I extracted and transformed data using Azure Data Factory, migrating it from transactional storage systems to Azure Data Lake Store Gen2 to support downstream analytics and business intelligence applications.I built efficient data pipelines for ingestion, aggregation, and loading, and used CI/CD practices within Azure DevOps to streamline deployment processes. I also configured and managed Azure Data Factory to automate workflows, improving data processing efficiencies, and implemented robust security protocols, including Azure Key Vault and network security groups, to ensure compliance with regulatory requirements. Furthermore, I performed data validation and quality checks to ensure the accuracy of ingested data, and implemented disaster recovery strategies using Azure Site Recovery to ensure business continuity.

In this role, I developed and optimized SQL Server applications to improve data processing and performance for client projects. I collaborated with data scientists to analyze high-velocity transaction data, supporting advanced analytics. I contributed to SDLC processes, designing and implementing scalable data solutions tailored to business needs. Additionally, I authored data quality scripts using Python and PostgreSQL to ensure data consistency across platforms. I also performed complex SQL transformations and tuning of Hadoop jobs for optimal performance. My responsibilities included managing Azure Infrastructure, deploying workloads on virtual machines (VMs), and establishing secure cloud connectivity. I developed ETL processes for data migration and transformation, facilitating the seamless transition from legacy systems to modern platforms.