With over 10 years of experience in both Data Engineering and Data Analysis, I have a strong background in designing, developing, and implementing data models for enterprise-level applications and systems.As an accomplished Data Engineer, my expertise lies in Python scripting and automation for Azure Databricks.I am experienced in ETL using Azure Databricks, including migrating on-premise Oracle processes to Azure Synapse Analytics and leveraging Matillion for Snowflake data pipeline development. Additionally, I am proficient in Spark scripting (PySpark), HDInsight, U-SQL, T-SQL, Spark SQL, Azure ADW, and Hive for seamless data loading and transformation.My successful implementation of PySpark on Azure Databricks for performance optimization, along with designing and maintaining scalable Hadoop data processing workflows, demonstrates my commitment to efficiency and excellence. I am skilled in MapReduce programming for raw data parsing, staging table population, and fine data storage in partitioned Hive tables. My proficiency extends to AWS Redshift, S3, Spectrum, and Athena for querying large datasets, creating Virtual Data Lakes, and optimizing Informatica workflows. Additionally, I am adept at Spark Streaming for real-time data processing from Kafka, with a focus on optimizing Hadoop cluster performance and fine-tuning configurations.I have extensive experience with relational SQL and NoSQL databases, including Oracle, Hive, Sqoop, HBase, and Cassandra. Moreover, I am accomplished in efficient data transformation within Snowflake, ensuring data integrity and alignment with business requirements.I have a proven ability to manage and version data engineering code and configurations in GitHub for collaboration, traceability, and rollback capabilities. My in-depth involvement in AWS Data Pipeline for configuring data loads from S3 to Redshift showcases my strong ETL expertise using AWS Glue. I have an extensive background in Data Warehousing, Business Intelligence, and ETL processes, including AWS Glue and Informatica. Furthermore, I have demonstrated the ability to enforce data security policies, access controls, and compliance standards within Snowflake, AWS, and other cloud environments. I have successfully implemented CI/CD pipelines in GitHub for automating testing and deployment of data engineering code changes. Lastly, I am accomplished in maintaining comprehensive documentation in Jira, capturing design decisions, dependencies, and project progress for effective project management.