Designed and developed a scalable real-time data pipeline using Apache Kafka and Spark Streaming to ingest and process data from multiple sources.Implemented a robust ETL framework in Python to transform raw data into analytical datasets, improving data processing speed by 35%.Created a dynamic analytics dashboard using Power BI to visualize key performance metrics, enabling clients to make data-driven decisions in real-time.Integrated RESTful APIs to facilitate seamless data access and integration with existing systems.Ensured high data quality and consistency by implementing automated validation and monitoring processes.

Led and executed the migration of on-premises data systems to Azure Cloud, designing, implementing ETL pipelines using Azure Data Factory to extract, transform, and load data into Azure Data Lake Storage Gen2. Implemented data transformation logic with Azure Databricks notebooks and designed ETL pipelines to cleanse, aggregate, and enrich raw data, converting it into structured and optimized formats for comprehensive analysis and reporting. Designed and implemented ETL pipelines using Azure Data Factory (ADF) and SQL Server Integration Services (SSIS), improving data integration and transformation processes and improved query performance by 70% Integrated Apache Spark with Azure Databricks to streamline data pipeline constructions, reducing system latency by 25%. Used Azure Data Factory pipelines to pull data from variety of semi-structured data sources, both on-premises and in the cloud. Ingested data from file-based locations containing CSV or JSON files. Designed and implemented scalable data warehousing solutions using Snowflake, leveraging its cloud-native architecture to handle large volumes of data efficiently. Successfully delivered large-scale data management initiatives, covering planning, design, building, and deployment phases, leveraging different delivery methodologies including Agile. Applied knowledge of continuous integration, static code analysis, and test-driven development to improve code quality and deployment processes. Automated report generation using SQL queries, reducing manual effort, and increasing process efficiency. Worked with cross-functional teams to define data requirements and data solution architecture for new projects. Utilized Azure Synapse Analytics for data warehousing and querying large datasets, facilitating real-time analytics and business decision-making. Managed scheduling, creating snapshots, and subscriptions for reports using SharePoint server, ensuring timely and accurate report delivery.

Built and maintained real-time and batch data pipelines using Apache Kafka, Apache Airflow, and AWS Lambda, reducing data latency from hours to seconds and increasing data processing efficiency. Led a successful database migration project from on-premises SQL Server to Amazon RDS, achieving a 40% reduction in infrastructure costs and significantly improving system scalability. Designed and developed scalable data pipelines using AWS Glue, facilitating ETL processes into Amazon S3 and Redshift, with rigorous testing to ensure data quality and accuracy. Designed and implemented data warehousing solutions using Amazon Redshift, optimizing query performance with techniques like indexing, partitioning, and distribution styles to support large-scale analytics and reporting Utilized SQL for comprehensive data validation and testing, identifying and resolving data inconsistencies and improving data integrity. Designed and implemented advanced data warehousing solutions using Snowflake and Amazon Redshift, optimizing query performance through indexing and partitioning techniques. Developed automated data transformation workflows in AWS Glue using PySpark and Scala, enhancing data processing speed and reliability. Created and managed real-time data visualization dashboards using Tableau and Power BI, providing actionable insights and improving decision-making for stakeholders. Deployed secure data processing and analytics systems on AWS using services like EMR, EC2, and S3, implementing VPCs and IAM roles to ensure data security and compliance. Conducted comprehensive API testing with Postman to ensure seamless data exchange between systems, enhancing system performance and reliability. Utilized Amazon Athena for serverless querying of data stored in Amazon S3, enabling ad-hoc data analysis and reducing the time required for data retrieval and reporting

Conducted comprehensive analyses of data to derive actionable insights, increasing actionable insights by 35%. Developed and managed ETL pipelines using Azure Data Factory (ADF) and SQL Server Integration Services (SSIS) to streamline data processing and reporting. Created and presented detailed reports and visualizations using Power BI and Excel, enhancing reporting capabilities for stakeholders. Executed complex SQL queries and scripts for data extraction, manipulation, and reporting, improving data-driven decision-making. Automated routine data processing tasks with Python and VBA, saving the team an average of 10 hours per week. Leveraged Data Factory to ingest data from diverse sources into Azure Data Lake, optimizing data integration processes. Worked with cross-functional teams to identify and seize business opportunities through data analytics and collaborative insights. Ensured data quality and integrity across various data sources and systems, including data warehouses and business reporting systems. Evaluated and improved performance of data processes and reporting systems, identifying areas for enhancement based on data analysis and user feedback. Stayed updated on industry trends and emerging technologies related to data analysis and warehousing, making recommendations for process improvements. Developed and documented detailed solution designs, ensuring compliance with data governance policies and regulatory requirements.

•Analyzed and transformed large datasets to identify trends, patterns, and anomalies to support business decision-making.•Developed and maintained ETL processes to extract, transform, and load data from various sources into data warehouses using tools such as SSIS (SQL Server Integration Services).•Designed and managed data pipelines to ensure data accuracy, consistency, and availability for analysis.•Created and maintained dashboards and reports using tools such as Tableau and Power BI to visualize supply chain performance metrics.•Collaborated with cross-functional teams to identify areas for process improvement within the supply chain.•Utilized SQL to develop complex queries, stored procedures, and functions for data extraction and manipulation.•Automated data collection and processing workflows using ETL tools and scripting languages to improve efficiency and reduce manual effort.•Facilitated data-driven decision-making by providing actionable recommendations based on data analysis.•Worked closely with IT and software development teams to define data requirements and support the implementation of new systems.