Experienced in end-to-end data engineering and analytics solutions across Azure cloud platforms. Proficient in designing and implementing ETL pipelines utilizing PySpark, Spark SQL, and Azure Data Factory for data extraction, transformation, and loading. Skilled in optimizing performance through Spark Context and Spark-SQL while leveraging deep learning frameworks like TensorFlow and PyTorch for advanced analytics. Adept at Agile methodologies, with a track record of successful deployments and migrations, integrating Git with Jenkins for continuous integration. Strong background in data visualization using tools like Tableau, coupled with expertise in SQL, Hive, and MapReduce for efficient data processing.

Experienced in architecting and implementing end-to-end data pipelines and ETL/ELT processes on Google Cloud Platform (GCP) and Azure, leveraging technologies such as PySpark, Spark SQL, Informatica, Hadoop, MapReduce, and Hive. Proficient in migrating SQL databases to Azure Data Lake, Azure SQL Database, Azure Data Warehouse, and Azure Databricks using Azure Data Factory. Skilled in designing optimal data pipeline architectures on Microsoft Azure utilizing Azure Data Factory and Azure Databricks, and recreating application logic in Azure environments. Additionally, adept at data visualization and analysis using Python libraries like Matplotlib, NumPy, SciPy, and Pandas. Experience in continuous integration and continuous delivery (CI/CD) pipelines on AWS, as well as designing and optimizing data models and schemas for efficient querying and analysis.

• Used the Azure Key vault as a central location to store secrets and used Databricks notebooks and Azure Data Factory to refer to the secrets. Proficient in designing and implementing robust data pipelines on Google Cloud Platform (GCP), specializing in schema design in BigQuery and developing Spark/GCP Dataproc jobs for efficient data processing from various sources. Skilled in leveraging Cloud Dataflow and Apache Beam to create scalable data processing pipelines, while optimizing data storage solutions using services like BigQuery and Cloud Storage for enhanced reliability and performance. Experienced in deploying code to production using CI/CD pipelines in GitLab, and implementing real-time data streaming solutions with GCP's Pub/Sub and Dataflow. Additionally, adept at ensuring data governance and security measures, documenting requirements, integrating data validations, and developing Python and Scala scripts for data extraction and aggregation.

Proficient in developing and implementing MapReduce jobs using Hive and Pig, ranging from simple to complex, to address various business requirements. Skilled in implementing Avro and Parquet data formats for Apache Hive computations, optimizing data handling for customized business needs. Experienced in designing, deploying, and enhancing algorithms within Hadoop/HBase clusters to derive customer-defined metrics and unsupervised learning models. Adept at installing and configuring Hadoop ecosystem tools such as Hive, Pig, Sqoop, Flume, and Oozie. Additionally, experienced in data cleansing, enrichment, mapping, and validation processes to ensure the accuracy and efficiency of reported data. Proficient in Apache Pig scripting and developing MapReduce programs for data parsing, staging, and refinement in EDW environments.