I configured Azure tools such as ADF, ADLS Gen 2, and Azure Logic App to meet ADF framework requirements, utilizing Azure Data Factory, Spark SQL, and Azure Synapse Analytics for ETL processes. I have hands-on experience with loading raw data into Azure Data Lake, enriching and curating it, and creating fact and dimension tables in Azure Synapse. I built and maintained ETL pipelines using Python and PySpark, optimizing data processing efficiency with techniques like parallel processing and data… Show more I configured Azure tools such as ADF, ADLS Gen 2, and Azure Logic App to meet ADF framework requirements, utilizing Azure Data Factory, Spark SQL, and Azure Synapse Analytics for ETL processes. I have hands-on experience with loading raw data into Azure Data Lake, enriching and curating it, and creating fact and dimension tables in Azure Synapse. I built and maintained ETL pipelines using Python and PySpark, optimizing data processing efficiency with techniques like parallel processing and data partitioning on Hadoop. I used Python for data cleaning and analysis, developed automation scripts to optimize infrastructure, and resolved PySpark issues during production loads.I utilized Spark and Spark-SQL to read parquet data and create tables in Azure Synapse, implementing and optimizing Spark jobs for large-scale data processing. I developed PySpark scripts for data ingestion and transformation, implemented Scala-based applications for batch and real-time processing, and managed Hadoop clusters for diverse data sources. I created Databricks notebooks with Python and SQL for various data models, designed and optimized SQL databases, and developed robust ETL pipelines using SQL. My experience includes managing NoSQL databases like MongoDB and Couchbase, developing data models in MySQL and PostgreSQL, and performing SQL query optimization.In addition, I implemented and optimized data pipelines in Snowflake for efficient data warehousing and analytics, created interactive dashboards and reports in Tableau and Power BI, and managed version control of ETL scripts using Git. I utilized Jira for project management and collaborated with cross-functional teams to gather requirements, design data models, and deploy analytics solutions using Snowflake, Tableau, and Power BI. My comprehensive experience in data engineering and analytics has enabled me to support data-driven decision-making and optimize data processing workflows. Show less

I have designed and implemented scalable data pipelines on AWS using AWS Glue and Amazon Redshift, resulting in a 40% reduction in data processing time. I developed and maintained data storage solutions with Amazon S3 and RDS, ensuring data integrity and availability. Automation of data workflows with AWS Lambda and CloudWatch improved accuracy and reduced manual intervention by 30%. My work included managing and optimizing AWS data infrastructure, including EC2 and VPC configurations, and… Show more I have designed and implemented scalable data pipelines on AWS using AWS Glue and Amazon Redshift, resulting in a 40% reduction in data processing time. I developed and maintained data storage solutions with Amazon S3 and RDS, ensuring data integrity and availability. Automation of data workflows with AWS Lambda and CloudWatch improved accuracy and reduced manual intervention by 30%. My work included managing and optimizing AWS data infrastructure, including EC2 and VPC configurations, and implementing end-to-end ETL pipelines in PySpark. I designed data mapping procedures for data extraction, analysis, and loading, and extensively worked on data flow creation and Jupyter notebooks for data processing.I orchestrated data workflows using Apache Airflow and Spark, scheduling and monitoring ETL jobs across distributed clusters. I developed Spark scripts using Python for efficient data processing and transformation, and scheduled Spark and Scala jobs using Airflow within a Hadoop Cluster. My tasks included data transformations and migration with Spark, PySpark, and Spark SQL, and using PySpark for machine learning model training and deployment. I designed Hadoop ecosystem components such as HDFS, Hive, and HBase, and optimized complex SQL queries, stored procedures, and triggers for data processing. Additionally, I developed NoSQL schemas on MongoDB for flexible storage solutions and maintained ETL pipelines using MySQL and PostgreSQL.I have experience designing and optimizing PostgreSQL queries and PL/pgSQL scripts, implementing PostgreSQL database schemas with partitioning and table inheritance, and working on complex SQL queries with DML and DCL operations. I designed scalable data warehousing solutions on Snowflake, created interactive dashboards in Tableau, and developed Power BI reports for data visualization. I managed version control for data engineering projects using Git and utilized JIRA for project management, issue tracking, and agile development. Show less

With extensive experience in migrating on-premises databases like SQL Server, MySQL, and PostgreSQL to Azure SQL Database, I specialize in managing schema migration and data transfer with minimal downtime. I have designed and optimized ETL workflows using Azure Data Factory, integrating data from diverse sources into Azure data lakes or Azure SQL Data Warehouse for scalability and reliability. Leveraging Azure Databricks, I execute Spark-based data processing tasks, optimizing performance… Show more With extensive experience in migrating on-premises databases like SQL Server, MySQL, and PostgreSQL to Azure SQL Database, I specialize in managing schema migration and data transfer with minimal downtime. I have designed and optimized ETL workflows using Azure Data Factory, integrating data from diverse sources into Azure data lakes or Azure SQL Data Warehouse for scalability and reliability. Leveraging Azure Databricks, I execute Spark-based data processing tasks, optimizing performance through caching, partitioning, and parallel execution. My expertise extends to data integration, ingestion, and ETL processes using Databricks, and I have developed on-demand data tables on Azure Blob Storage using Azure Functions and Databricks with Python and PySpark.I have resolved complex issues in Spark and Yarn resource management, including OOM errors, shuffle exceptions, heap space errors, and schema compatibility. Working on a Hortonworks-based Hadoop platform deployed on a 120-node cluster, I built a data lake using Spark, Hive, and NoSQL for data processing. My experience includes using Apache Spark 2.0 for real-time data processing, gathering requirements to develop data pipelines from various sources to Hadoop, and implementing data processing scripts in Spark using Python to optimize performance through efficient data transformations and parallel processing. Additionally, I have utilized PySpark for complex data transformations and migrations, ensuring data integrity and reliability, and managed Spark job scheduling and orchestration using Apache Airflow.In my roles, I have designed and optimized complex SQL queries for data extraction, transformation, and analysis, ensuring efficient data processing and retrieval. Show less

With extensive experience in leveraging Python and R for data manipulation, statistical analysis, and machine learning model development, I have consistently provided actionable insights and predictions to business stakeholders. I have designed and optimized complex SQL queries to extract and analyze large datasets from relational databases, ensuring data accuracy and reliability for reporting and analytics. My expertise extends to implementing and managing NoSQL databases like MongoDB and… Show more With extensive experience in leveraging Python and R for data manipulation, statistical analysis, and machine learning model development, I have consistently provided actionable insights and predictions to business stakeholders. I have designed and optimized complex SQL queries to extract and analyze large datasets from relational databases, ensuring data accuracy and reliability for reporting and analytics. My expertise extends to implementing and managing NoSQL databases like MongoDB and Cassandra, handling unstructured data, and facilitating scalable storage solutions. I have developed interactive dashboards in Power BI to visualize key metrics and trends, supporting data-driven decision-making across departments and senior management. Additionally, I have applied SAS for advanced statistical analysis and predictive modeling, uncovering patterns that contributed to strategic business initiatives.In my role, I have overseen data processing and analysis using Apache Hadoop ecosystem tools such as HDFS, Hive, and Spark, optimizing big data workflows for efficient data handling. I have employed Jupyter Notebook for exploratory data analysis (EDA) and prototyping machine learning models, documenting data insights and methodologies. My proficiency with Google Analytics has enabled me to analyze web traffic, user behavior, and campaign performance, providing insights to optimize digital marketing strategies and website performance. I have created detailed ER diagrams using tools like Rational Rose to model database structures and designed, deployed, and managed data infrastructure on AWS. Leveraging services such as EC2, S3, RDS, and Redshift, I have supported scalable data storage, processing, and analytics.Furthermore, I have built robust ETL pipelines using AWS Glue, leveraging Python and PySpark for data extraction, transformation, and loading into data warehouses or data lakes on AWS. Show less